How Chunking and Tokenization Help in Embedding Creation | Slides
SLIDE4 |
|
Tokenization is the process of breaking down text into smaller units, called tokens, which can be words, subwords, characters, or even sentences. It is a crucial step in Natural Language Processing (NLP) and machine learning as it helps models to better understand and process textual data. There are two main types of tokenization: 1. Word Tokenization: Splitting text into individual words (e.g., "I love NLP" becomes ["I", "love", "NLP"]). 2. Subword Tokenization: Breaking words into subword units, which is common in modern NLP models (e.g., "unbreakable" might be tokenized as ["un", "break", "able"]). Subword tokenization is used in models like GPT and BERT, allowing them to handle unknown words by breaking them into familiar subword units. It also helps reduce the overall vocabulary size. ChunkingChunking is the process of splitting large amounts of text into manageable "chunks" or pieces before processing. It’s often applied after tokenization, especially when dealing with longer texts that exceed the model’s token limit. In the context of NLP: - Tokenization breaks text into tokens. - Chunking then organizes tokens into coherent groups or smaller sequences for efficient processing. For instance, large documents are broken down into chunks of tokens to fit within the model's token limit (like 4096 tokens). Chunking ensures that each tokenized piece contains meaningful content and avoids breaking words or sentences inappropriately. How Tokenization and Chunking Work Together:
For example: - Input: "The quick brown fox jumps over the lazy dog." - Tokenization: ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]. - Chunking: If a model has a limit of 5 tokens per chunk, the tokens would be grouped like: - Chunk 1: ["The", "quick", "brown", "fox", "jumps"] - Chunk 2: ["over", "the", "lazy", "dog"] In sum, tokenization enables models to work with text as numerical data, while chunking helps organize this data into manageable segments for processing. |
Challenges-in-good-embeddings Chunking-and-tokenization Chunking Dimensionality-reduction-need Dimensionality-vs-model-perfo Embeddings-for-question-answer Ethical-implications-of-using Impact-of-embedding-dimension Open-ai-embeddings Role-of-embeddings-in-various