Lesson 9: Recurrent networks (RNN) and the rise of attention

An image is data on a fixed, unchanging grid. Natural language and audio signals are completely different: they are sequential data, of continuously varying length and with a strict temporal order. Swap two words in a sentence and the meaning can reverse or vanish. To handle sequences we need the recurrent neural network (RNN). This lesson goes into the mathematics of RNNs, why they suffer vanishing gradients, how gated memories like LSTM and GRU fix it, and the single biggest stepping stone towards modern AI: the attention mechanism.

9.1 How a recurrent neural network remembers a sequence

An ordinary MLP or CNN passes data in one direction only, from input through hidden layers to output (feedforward). They have no notion of time or memory: each sample is processed independently, with no relationship to the one before.

An RNN fixes this by introducing a recurrent loop inside the neuron. At each time step $t$ the network takes the current input $x_t$ together with the hidden state from the previous step $h_{t-1}$ (think of it as the network's memory of the past) to compute a new hidden state $h_t$:

But training an RNN on long sequences — a paragraph of more than 50 words, say — runs into a serious mathematical problem: the vanishing gradient.

To compute derivatives for the early time steps and update their weights, the BPTT algorithm (backpropagation through time) has to multiply by the weight matrix $W_{hh}^T$ once per time step. If the eigenvalues of $W_{hh}$ are less than 1, multiplying a number smaller than 1 by itself $N$ times drives the derivative exponentially towards zero: $$\lim_{N \to \infty} (0.9)^N = 0$$ The consequence is that the network loses the ability to learn long-term dependencies, remembering only the words near the end of the sentence.

9.2 A gated filter for information: LSTM & GRU

To solve vanishing gradients properly, Hochreiter and Schmidhuber proposed the LSTM (long short-term memory) architecture. Rather than carrying a single plain hidden state $h_t$, an LSTM adds a parallel information channel called the cell state ($c_t$) — think of it as an information motorway running the length of the sequence — controlled by gates built from the sigmoid activation $\sigma$:

• Forget gate ($f_t$): decides how much of the old information to erase from the cell state: $$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$$
• Input gate ($i_t$): decides how much new information from the input gets written into the cell state: $$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$$ $$\tilde{c}_t = \tanh(W_c [h_{t-1}, x_t] + b_c)$$
• Updating the cell state ($c_t$): multiply the old information by the forget gate, then add the new: $$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$
• Output gate ($o_t$): decides which part of the cell state becomes the next hidden state $h_t$: $$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$$ $$h_t = o_t \odot \tanh(c_t)$$

Because the cell state is updated by linear addition ($+$) rather than repeated matrix multiplication, the derivative can flow straight back into the past without vanishing, which is what lets an LSTM learn very long-range context.

GRU (gated recurrent unit) is a slimmed-down LSTM. It merges the cell state and hidden state into one and uses only two gates: an update gate and a reset gate. A GRU computes faster and uses less memory while matching LSTM performance on many tasks.

9.3 The idea of attention

In a traditional sequence-to-sequence translation model (an LSTM encoder-decoder), the encoder has to compress the entire long input sentence into a single final hidden state — the context vector — and hand that to the decoder to produce the translation.

That is like forcing someone to memorise a whole book and then write the translation without ever being allowed to turn back to an earlier page. When the input sentence is long, the context vector is overloaded and translation quality degrades badly.

Attention exists to remove that bottleneck. Its philosophy is remarkably simple: let the decoder look back at all of the encoder's hidden states, at every time step, and work out for itself which word in the source matters most for the word about to be produced:

9.4 Hands-on project: an LSTM sentiment classifier — and its limits

This lesson's project trains an LSTM to classify service reviews as positive or negative. It runs, it gives convincing-looking results — and the most valuable part is where it fails.

# sentiment_lstm.py
# Lesson 9: Recurrent networks (RNN) and the rise of attention
# Practical AI Engineer series
#
# Run it with:  python sentiment_lstm.py
# Requires:     pip install torch
#
# READ THIS BEFORE BELIEVING THE OUTPUT.
# Six training sentences is far too few to learn sentiment. What the model can do
# with six sentences is memorise which specific WORDS go with which label — and
# that is exactly what it does. The script therefore ends with a test on entirely
# unseen words, where it scores about 50/50: a coin flip. That contrast is the
# point of the project, not an accident.
#
# The Vietnamese review text stays Vietnamese: it is the DATA being classified,
# and it is what makes the word-memorisation effect visible.

import torch
import torch.nn as nn
import torch.optim as optim

# Seed everything, so the numbers printed in the lesson can actually be compared.
torch.manual_seed(42)

# 1. A tiny sample dataset of Vietnamese service reviews.
dataset = [
    ("dịch vụ xuất sắc nhân viên thân thiện", 1), # 1: Tích cực
    ("đồ ăn ngon phục vụ rất nhanh", 1),
    ("sản phẩm tuyệt vời đóng gói cẩn thận", 1),
    ("quá tệ đồ ăn nguội lạnh phục vụ kém", 0),  # 0: Tiêu cực
    ("giao hàng chậm trễ chất lượng tồi tệ", 0),
    ("thái độ nhân viên rất lồi lõm không mua lại", 0)
]

# 2. A crude word-level tokenizer.
words = set()
for text, _ in dataset:
    words.update(text.split())

# sorted() matters: a Python set iterates in an order that changes between runs
# (string hashing is randomised), which would give the words different IDs every
# run and make the output impossible to reproduce.
vocab = {word: idx + 2 for idx, word in enumerate(sorted(words))}  # 0: padding, 1: OOV
vocab["[PAD]"] = 0
vocab["[UNK]"] = 1
inverse_vocab = {v: k for k, v in vocab.items()}

# Turn text into a fixed-length sequence of integers, padding the short ones.
def text_to_sequence(text, max_len=8):
    tokens = text.split()
    seq = []
    for token in tokens:
        seq.append(vocab.get(token, 1))
    # Padding hoặc Truncate
    if len(seq) < max_len:
        seq += [0] * (max_len - len(seq))
    else:
        seq = seq[:max_len]
    return seq

# Build the input tensors.
x_data = torch.tensor([text_to_sequence(text) for text, _ in dataset], dtype=torch.long)
y_data = torch.tensor([label for _, label in dataset], dtype=torch.float32).unsqueeze(1)

# 3. The SentimentLSTM architecture.
class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim):
        super(SentimentLSTM, self).__init__()
        
        # padding_idx=0 tells the layer to keep the padding vector at zero and never train it.
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
        
        # The recurrent layer.
        # batch_first=True gives the input shape (batch, sequence length, features).
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)
        
        # Binary classifier head; Sigmoid so BCELoss can read it.
        self.classifier = nn.Linear(hidden_dim, 1)
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, x):
        # x shape: (Batch, Sequence Length)
        embedded = self.embedding(x) # shape: (Batch, Sequence Length, Embedding Dim)
        
        # Run the sequence through the LSTM.
        # out: the hidden state at EVERY time step
        # (hn, cn): the hidden state and cell state at the LAST time step only
        out, (hn, cn) = self.lstm(embedded)
        
        # Take the final hidden state as a summary of the whole sentence. This single
        # vector is the bottleneck that attention (section 9.3) exists to remove.
        last_hidden = hn[-1] # shape: (Batch, Hidden Dim)
        
        logits = self.classifier(last_hidden)
        predictions = self.sigmoid(logits)
        return predictions

if __name__ == "__main__":
    print("=== Preparing the training data ===")
    print(f"vocabulary size: {len(vocab)}")
    print(f"input tensor shape: {x_data.shape}\n")
    
    # Build the model.
    model = SentimentLSTM(vocab_size=len(vocab), embedding_dim=16, hidden_dim=8)
    
    criterion = nn.BCELoss()  # binary cross-entropy, for a two-class problem
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    
    epochs = 40
    print("=== Training the SentimentLSTM ===")
    for epoch in range(1, epochs + 1):
        model.train()
        optimizer.zero_grad()
        
        predictions = model(x_data)
        loss = criterion(predictions, y_data)
        
        loss.backward()
        optimizer.step()
        
        if epoch % 10 == 0:
            # Training accuracy only — there is no held-out set here, by design.
            binary_predictions = (predictions >= 0.5).float()
            accuracy = (binary_predictions == y_data).sum().item() / len(y_data) * 100
            print(f"Epoch {epoch:02d}/{epochs} | Loss: {loss.item():.4f} | Accuracy: {accuracy:.1f}%")
            
    print("\n=== Inference on words the model HAS seen ===")
    model.eval()

    def predict(comment):
        seq = torch.tensor([text_to_sequence(comment)], dtype=torch.long)
        with torch.no_grad():
            p = model(seq).item()
        unknown = [w for w in comment.split() if w not in vocab]
        label = "positive" if p >= 0.5 else "negative"
        print(f"  \"{comment}\"")
        print(f"    {p * 100:6.2f}% positive -> {label:8} | {len(unknown)}/{len(comment.split())} words unknown")
        return p

    # Every word in these two sentences already appears in the training data.
    predict("đồ ăn ngon phục vụ nhanh tuyệt vời")
    predict("phục vụ quá tệ chất lượng tồi")

    print("\n=== The honest test: words the model has NEVER seen ===")
    # Same sentiment, completely different vocabulary. Every token becomes [UNK],
    # so the model has nothing memorised to fall back on.
    p_good = predict("bánh mì thơm giòn lịch sự")
    p_bad = predict("nhà hàng bẩn thỉu hôi hám")

    print("\n=== What that means ===")
    good_label = "positive" if p_good >= 0.5 else "negative"
    bad_label = "positive" if p_bad >= 0.5 else "negative"
    if good_label == bad_label:
        print(f"  Both sentences came out {good_label}, even though one praises and one")
        print("  complains. Every one of their words is [UNK], so the model has nothing")
        print("  memorised to go on and simply collapses to one side.")
    print("  With 6 training sentences the model memorised which WORDS carry which")
    print("  label — it learned nothing about sentiment itself. Swap the words and the")
    print("  knowledge is gone. That is the real lesson of this project.")
    print("  Real sentiment analysis needs thousands of examples, or embeddings")
    print("  pretrained on a large corpus. Lesson 14 uses the pretrained route.")

Running it produces:

=== Preparing the training data ===
vocabulary size: 43
input tensor shape: torch.Size([6, 8])

=== Training the SentimentLSTM ===
Epoch 10/40 | Loss: 0.5675 | Accuracy: 100.0%
Epoch 40/40 | Loss: 0.0536 | Accuracy: 100.0%

=== Inference on words the model HAS seen ===
  "đồ ăn ngon phục vụ nhanh tuyệt vời"
     78.68% positive -> positive | 0/8 words unknown
  "phục vụ quá tệ chất lượng tồi"
     17.18% positive -> negative | 0/7 words unknown

=== The honest test: words the model has NEVER seen ===
  "bánh mì thơm giòn lịch sự"
     36.33% positive -> negative | 6/6 words unknown
  "nhà hàng bẩn thỉu hôi hám"
     31.46% positive -> negative | 5/6 words unknown

=== What that means ===
  Both sentences came out negative, even though one praises and one
  complains. Every one of their words is [UNK], so the model has nothing
  memorised to go on and simply collapses to one side.

Read the last four numbers side by side, because they are the whole lesson. On words it has seen: 78.68% and 17.18% — cleanly separated, looking like a working sentiment classifier. On words it has never seen: 36.33% and 31.46% — both classified negative, even though one praises and one complains.

The model learned nothing about sentiment. It learned that token 12 goes with label 1. When every token becomes [UNK] it has nothing to look up, so it collapses to one side. The 100% training accuracy is memorisation of 6 sentences — the same signature as Lesson 7, only better hidden here, because the two test sentences reuse the old words.

How to run this project on your own machine

1. Install: pip install torch.
2. Download sentiment_lstm.py at the end of the lesson, or retype the code above.
3. Run it: python3 sentiment_lstm.py. You will get exactly the numbers above, thanks to torch.manual_seed(42) and one more detail: the vocabulary is sorted() before IDs are assigned. Without the sort, a Python set iterates in a different order on every run (string hashing is randomised), the word IDs change with it, and the result never matches the previous run.
4. Then try two things:
   • Add 6 more sentences to dataset — write your own, covering both sides. Rerun and see whether the "entirely new words" test improves. You will get a feel for just how much data it takes before this starts to mean anything.
   • Change nn.LSTM to nn.RNN, leaving every other parameter alone. On 8-token sentences there is barely any difference — exactly as sections 9.1 and 9.2 said: vanishing gradients only become a problem on long sequences.

Lesson summary & bridge to what's next

Bridge to the next lesson: attention solves the bottleneck, but it is still bolted onto a sequential RNN. Lesson 10 removes the recurrence entirely and keeps only attention — the Transformer architecture.