Lesson 8: Text processing & word embeddings

How does a computer understand what a word means? A computer only processes numbers, matrices and linear algebra — it cannot read letters or sentences directly.

This lesson walks through digitising text properly: breaking sentences into token codes (tokenization), getting past the out-of-vocabulary trap, and on to representing meaning as vectors in a high-dimensional space (word embeddings). We also cover the mathematics of cosine similarity and build a synonym-finding application, solving semantic similarity with PyTorch.

8.1 From writing to numbers: tokenization

Before feeding text into a machine learning model, the raw text must go through a mandatory preprocessing step called tokenization. This is the process of cutting a long character string into smaller units of information, called tokens (single characters, syllables, words or phrases).

There are three main tokenizer designs:

• Character-level tokenization: splits text down to individual letters (a, b, c...).
  Advantage: an extremely compact vocabulary (a few hundred characters), and it never fears an unknown word.
  Drawback: the computer has to process very long sequences, and neurons struggle to learn a word's overall meaning because the information is shattered at the letter level.
• Word-level tokenization: splits on whitespace into whole words.
  Advantage: preserves each word's original meaning well.
  Drawback: vocabulary explosion (up to millions of words). In particular, on meeting a word never seen during training, the model hits the out-of-vocabulary (OOV) problem and must replace that word with a placeholder [UNK] (unknown), losing the sentence's information entirely.
• Subword-level tokenization: the gold standard for today's large language models such as GPT and BERT, using algorithms like BPE (byte pair encoding) or WordPiece.
  How it works: keeps common words whole, but cuts rare or compound words into meaningful prefix/suffix fragments (the word "embeddings" might become "embed" and "##dings"). This keeps the vocabulary at an optimal size (roughly 30,000–50,000 tokens) while still representing any new word by assembling fragments, eliminating OOV entirely.

8.2 The semantic map: word embeddings

Once the tokenizer has turned a sentence into a list of integer IDs: $$\text{"I love AI"} \to [102, 540, 891]$$ how do we carry that information into a neural network?

The simplest approach is one-hot encoding: create a vector as long as the entire vocabulary $V$, put a 1 at the word's ID position and 0 everywhere else. But this has two fatal weaknesses:

1. An extremely wasteful sparse matrix: with a vocabulary of $50{,}000$ words, every single word becomes a $50{,}000$-dimensional vector of almost all zeros.
2. Total loss of semantic relationship: any two one-hot vectors $v_a$ and $v_b$ are always orthogonal in vector space, so their dot product is zero: $$v_a \cdot v_b = 0$$ Which means the computer treats "cat" and "dog" as entirely unrelated — exactly as unrelated as "cat" and "table".

Because it is trained simultaneously on an enormous body of text, the embedding matrix automatically shifts each word's coordinates so that: words appearing in similar contexts end up near each other in the high-dimensional space. That is the premise behind the classic semantic-similarity arithmetic: $$\text{Embedding("King")} - \text{Embedding("Man")} + \text{Embedding("Woman")} \approx \text{Embedding("Queen")}$$

8.3 Measuring semantic distance: cosine similarity

With words represented as vectors in a high-dimensional space, how do we measure how semantically similar they are?

We cannot use ordinary geometric distance (Euclidean, L2) for a direct comparison, because Euclidean distance is sensitive to a vector's length (its norm). In sentence-comparison tasks a long sentence can produce aggregate vectors of very large magnitude, throwing Euclidean distance far off compared with a short sentence even when both share the same topic.

Instead we measure the angle between two vectors, with the formula for cosine similarity:

8.4 Hands-on project: a semantic space built by hand

This lesson's project takes a keyword, turns it into an embedding vector, then computes cosine similarity against the whole vocabulary to find the closest words in meaning — including the word-algebra operation king − man + woman.

# word_similarity.py
# Lesson 8: Text processing & word embeddings
# Practical AI Engineer series
#
# Run it with:  python word_similarity.py
# Requires:     pip install torch
#
# IMPORTANT — THE VECTORS BELOW ARE HAND-WRITTEN, NOT TRAINED.
# This is not Word2Vec. Word2Vec is a training algorithm that reads billions of
# sentences and discovers these numbers on its own. Here the 9 vectors are typed
# out by hand, with each of the 4 dimensions given a meaning by a human, so that
# you can SEE why the geometry works. A real 300-dimensional trained embedding is
# opaque: nobody can say what dimension 174 means.
#
# The cost of that clarity is that the numbers come out artificially clean — see
# the note on the 1.0000 score at the bottom of this file.

import torch
import torch.nn as nn

vocab = {
    'king': 0,
    'queen': 1,
    'man': 2,
    'woman': 3,
    'computer': 4,
    'programming': 5,
    'artificial_intelligence': 6,
    'coffee': 7,
    'tea': 8,
}
inverse_vocab = {v: k for k, v in vocab.items()}

# Dimension 0: royalty · 1: gender (positive male, negative female)
# Dimension 2: technology · 3: drinks
embedding_weights = torch.tensor(
    [
        [1.0, 0.9, 0.0, 0.0],  # king
        [1.0, -0.9, 0.0, 0.0],  # queen
        [0.0, 1.0, 0.0, 0.0],  # man
        [0.0, -1.0, 0.0, 0.0],  # woman
        [0.0, 0.0, 1.0, 0.0],  # computer
        [0.0, 0.0, 0.9, 0.0],  # programming
        [0.0, 0.1, 1.0, 0.0],  # artificial_intelligence
        [0.0, 0.0, 0.0, 1.0],  # coffee
        [0.0, 0.0, 0.0, 0.9],  # tea
    ],
    dtype=torch.float32,
)

vocab_size, embedding_dim = embedding_weights.shape
embed = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim)
# Fixed weights, never trained further.
embed.weight = nn.Parameter(embedding_weights, requires_grad=False)


def cosine_similarity(vector_a, matrix_b):
    """Cosine of the angle between one vector (1,d) and every row of (V,d)."""
    dot_product = torch.sum(vector_a * matrix_b, dim=1)
    norm_a = torch.norm(vector_a, p=2, dim=1)
    norm_b = torch.norm(matrix_b, p=2, dim=1)
    # +1e-8 guards against dividing by zero for an all-zero vector.
    return dot_product / (norm_a * norm_b + 1e-8)


def find_most_similar(target_word, top_n=3):
    if target_word not in vocab:
        print(f"'{target_word}' is not in the vocabulary.")
        return

    target_vector = embed(torch.tensor([vocab[target_word]]))
    scores = cosine_similarity(target_vector, embed.weight)
    top_scores, top_indices = torch.topk(scores, k=len(vocab))

    print(f"--- closest words to '{target_word}':")
    shown = 0
    for score, idx in zip(top_scores.tolist(), top_indices.tolist()):
        word = inverse_vocab[idx]
        if word == target_word:
            continue  # a word is always its own closest match; skip it
        print(f'  {shown + 1}. {word:<24} cosine {score:.4f}')
        shown += 1
        if shown >= top_n:
            break
    print()


def analogy(a, b, c, top_n=2):
    """Solve 'a is to b as c is to ?' — the classic king - man + woman."""
    vec = embed(torch.tensor([vocab[a]])) - embed(torch.tensor([vocab[b]])) + embed(torch.tensor([vocab[c]]))
    scores = cosine_similarity(vec, embed.weight)

    print(f'=== word analogy: {a} - {b} + {c}')
    shown = 0
    for score, idx in zip(*[t.tolist() for t in torch.topk(scores, k=len(vocab))]):
        word = inverse_vocab[idx]
        # Exclude the three input words. With hand-made vectors this barely
        # matters, but with REAL trained embeddings the input word almost always
        # ranks first, and forgetting to exclude it is the classic mistake that
        # makes analogy code look broken.
        if word in (a, b, c):
            continue
        print(f'  {shown + 1}. {word:<24} cosine {score:.4f}')
        shown += 1
        if shown >= top_n:
            break
    print()


if __name__ == '__main__':
    find_most_similar('coffee', top_n=2)
    find_most_similar('computer', top_n=2)
    analogy('king', 'man', 'woman')

    # Why 'tea' scores exactly 1.0000 against 'coffee': their vectors are
    # [0,0,0,1] and [0,0,0,0.9] — exactly parallel, differing only in length,
    # and cosine ignores length. A real trained embedding never gives exactly
    # 1.0 for two different words. This is the artificial cleanliness that comes
    # with hand-writing the numbers.
    print('note: coffee and tea are exactly parallel by construction, hence 1.0000.')

Running it produces:

--- closest words to 'coffee':
  1. tea                      cosine 1.0000
  2. man                      cosine 0.0000

--- closest words to 'computer':
  1. programming              cosine 1.0000
  2. artificial_intelligence  cosine 0.9950

=== word analogy: king - man + woman
  1. queen                    cosine 0.9950
  2. computer                 cosine 0.0000

note: coffee and tea are exactly parallel by construction, hence 1.0000.

The analogy works: king − man + woman gives queen at 0.995. But the other three numbers are the part worth learning from, because they expose the limits of building this by hand.

How to run this project on your own machine

1. Install: pip install torch. No data download needed.
2. Download word_similarity.py at the end of the lesson, or retype the code above.
3. Run it: python3 word_similarity.py. There is no randomness, so the result is always identical.
4. Then try three things, each teaching something:
   • Add a new word to vocab and a matching vector row — say 'prince': 9 with [1.0, 0.9, 0.0, 0.0]. It will score 1.0 against "king", because you just typed the same direction. That is the most direct way to feel that "meaning" here is decided by you.
   • Change "queen"'s gender dimension from -0.9 to +0.9 and rerun. The analogy stops producing "queen" — proof that the result comes from the geometry, not from the name.
   • Remove the if word in (a, b, c): continue line in analogy(). It barely changes the answer here, but remember this behaviour when you use real embeddings in Lesson 14.

Lesson summary & bridge to what's next

Bridge to the next lesson: individual word vectors cannot express the order of a sentence. To handle long language sequences over time we need recurrent networks and the attention mechanism, in Lesson 9.