AI Fundamentals for Developers: Tokens, Vectors and Embeddings

If you're reading about AI, LLMs, or RAG, you'll hear terms like tokens, vectors, and embeddings all the time. They may sound complicated, but the basic idea is quite simple. Computers don't understand language the way humans do. Before an AI model can process text, it first converts it into numbers. The journey looks like this: Text ↓ Tokenizer ↓ Tokens ↓ Embeddings (Vectors) ↓ AI Model ↓ Output Enter fullscreen mode Exit fullscreen mode Or simply: Language → Small pieces → Numbers → Understanding 1. Tokens A token is a small piece of text that the model processes. For example: "I love cricket" Enter fullscreen mode Exit fullscreen mode may become: ["I", "love", "cricket"] Enter fullscreen mode Exit fullscreen mode But tokens are not always complete words. playing Enter fullscreen mode Exit fullscreen mode may become: ["play", "ing"] Enter fullscreen mode Exit fullscreen mode or unbelievable Enter fullscreen mode Exit fullscreen mode may become: ["un", "believ", "able"] Enter fullscreen mode Exit fullscreen mode So, words and tokens are not always the same thing. 2. Tokenizer A tokenizer is the tool that splits text into tokens. Example: Input: unbelievable Output: ["un", "believ", "able"] Enter fullscreen mode Exit fullscreen mode Different AI models use different tokenizers, so the same sentence may be split differently. You'll also notice that some tokens include spaces, such as " like" instead of "like". This helps the model better understand how text naturally appears. 3. Tokenization Tokenization is simply the process of splitting text into tokens. Text ↓ Tokenizer ↓ Tokens Enter fullscreen mode Exit fullscreen mode The tokenizer is the tool. Tokenization is the process. 4. How Does the Tokenizer Split Words? The tokenizer doesn't know every word beforehand. Instead, it learns common patterns from large amounts of text. You don't need to remember every method, but you'll often hear these names: BPE (Byte Pair Encoding) – merges common pieces of words. WordPiece – similar to BPE but chooses splits based on probability. For example: playing Enter fullscreen mode Exit fullscreen mode may become: play + ing Enter fullscreen mode Exit fullscreen mode Modern LLMs mostly use subword tokenization, sometimes combined with byte-level tokenization, because it handles new words, different languages, emojis, and symbols efficiently. 5. Tokens Become Numbers After tokenization, the computer still doesn't understand the text. Computers only work with numbers, so every token is converted into a vector. For example: cat Enter fullscreen mode Exit fullscreen mode might become: [0.2, -0.5, 1.3, ...] Enter fullscreen mode Exit fullscreen mode The numbers themselves are not important. They are simply how the computer represents text. 6. Vector vs Embedding These two terms are closely related, but they are not the same. A vector is simply a list of numbers. [0.2, -0.5, 1.3, 0.8] Enter fullscreen mode Exit fullscreen mode Those numbers could represent anything. An embedding is a vector that has been learned by an AI model to capture meaning. For example: cat → [0.2, -0.5, 1.3, ...] dog → [0.25, -0.45, 1.2, ...] car → [-1.8, 0.7, 2.4, ...] Enter fullscreen mode Exit fullscreen mode The important part isn't the numbers themselves. It's the distance between them. Since cat and dog have similar meanings, their embeddings are close together. Car is a different concept, so it is farther away. A simple way to remember it is: Vector = just numbers Embedding = numbers with meaning In other words: Every embedding is a vector, but not every vector is an embedding. 7. Token Embeddings Every token gets its own embedding. For example, king and queen have different embeddings, but they are close together because they are related. The model learns these relationships during training. 8. Text Embeddings Sometimes we don't want an embedding for each token. Instead, we create one embedding for an entire sentence, paragraph, or document. For example: I love cricket. Enter fullscreen mode Exit fullscreen mode becomes a single vector representing the meaning of the whole sentence. Text embeddings are commonly used for: Semantic search RAG Recommendations Finding similar documents For example, if someone searches: best sport in India the system can still find a document saying: Cricket is the most popular sport in India. Even though the wording is different, the meaning is similar. 9. Chunking Large documents are usually split into smaller pieces called chunks before creating embeddings. Document ↓ Chunks ↓ Embedding for each chunk Enter fullscreen mode Exit fullscreen mode Later, when someone asks a question, the system compares the question's embedding with all chunk embeddings and retrieves the most relevant chunks. This is one of the core ideas behind RAG. 10. Other Important Terms Vocabulary The vocabulary is the collection of tokens the tokenizer knows. Common vocabulary sizes are 30K, 50K, or 100K tokens. Special Tokens Models also use special control tokens such as: Start of text End of text Separator Padding These help the model understand the structure of the input. Context Window Every model has a maximum number of tokens it can process at once. This is called the context window. For example: 8K tokens 32K tokens 100K+ tokens If the input is longer than the context window, some text has to be removed or handled separately. Putting Everything Together The complete flow looks like this: Input Text ↓ Tokenizer ↓ Tokens ↓ Embeddings ↓ AI Model ↓ Output Tokens ↓ Readable Text Enter fullscreen mode Exit fullscreen mode A ChatGPT Example Suppose you ask ChatGPT: "What is the capital of India?" Behind the scenes, something like this happens: Your Question ↓ Tokenizer splits the text into tokens ↓ Each token is converted into an embedding (a vector that captures its meaning) ↓ Similar words and concepts are placed close together in the embedding space ↓ The model processes these embeddings to understand the context and relationships ↓ It predicts the most likely next tokens ↓ "Delhi is the capital of India." Enter fullscreen mode Exit fullscreen mode You only see the final answer, but internally the model never works directly with words. It works with tokens, embeddings, and mathematical operations on those embeddings to generate the response. What's Next? This blog covered the basics of how AI understands text using tokens and embeddings. These concepts are the foundation for almost everything you'll learn next. In the upcoming blogs, we'll keep building on these basics and cover topics like: RAG (Retrieval-Augmented Generation) explained in simple language How context works in AI and why it matters Different types of RAG, including Agentic RAG, Graph RAG, Corrective RAG, and more Vector Databases and how they store embeddings Semantic Search and how AI finds information by meaning instead of keywords AI Agents and Agentic AI—what they are and when to use them Each topic will be explained step by step, without assuming any prior AI knowledge. Stay tuned! This post was initially published on my blog. Check out the original source using the link below: