For a RAG system to work at scale, crudely cutting text every N characters is nowhere near enough. The quality of what you retrieve depends directly on two things: the chunking strategy, and the indexing capability of the vector database.
Lesson 14 ended on a finding: how you cut text decides whether the answer is right. This lesson turns that observation into measurements. We compare 3 chunking strategies across two different kinds of text to see which wins where, implement an approximate search graph in the HNSW family and measure what it really saves and what it costs, and finally build hybrid search with RRF (Reciprocal Rank Fusion) over a document set where each retriever alone fails somewhere.
ollama pull bge-m3); without it the program still runs and says exactly which parts it skipped. A note on runtime: section 15.2 builds graphs over 5,000 and then 20,000 vectors in pure Python, so it takes a minute or two. That is the price of seeing the algorithm instead of calling a library.
Knowledge you need: Lesson 14 for the whole RAG pipeline, cosine similarity and TF-IDF — this lesson upgrades exactly step 2 (chunking) and step 4 (retrieval) of that pipeline.
15.1 Chunking strategies
The chunking strategy you pick directly determines your recall rate and how diluted the context becomes. Chunks too large and the LLM drowns in irrelevant text; chunks too small and the supporting context is severed.
The three most common strategies today:
- Fixed-size chunking: cut mechanically every $N$ characters or words. Extremely fast, but it routinely slices sentences in half and cuts through important figures or proper nouns — exactly the failure Lesson 14 measured.
-
Recursive character chunking: split using a priority-ordered list of separators: double
newline
\n\n(paragraph), single newline\n(line), full stop.(sentence), comma,, and finally whitespace. It tries to keep sentences whole within the chunk size limit. - Semantic chunking: the more advanced option. Scan the text sentence by sentence, compute an embedding vector for each, and measure similarity between consecutive sentences. A new chunk boundary is created wherever that similarity drops sharply below a preset threshold.
Measured: which strategy wins, and on what kind of text?
The project runs all three strategies over two documents with opposite properties. The first is a tidy NDA where every clause is its own fully punctuated paragraph:
=== Tidy document (NDA, clean punctuation) ===
query: Mức phạt tiền vi phạm rò rỉ dữ liệu là bao nhiêu?
fixed-size (120 chars) 6 chunks best 0.4108
-> 't vi phạm hành chính đối với trường hợp rò rỉ dữ liệu khách hàng'
recursive (punctuation) 6 chunks best 0.4529
-> '[Điều 2] Mức phạt vi phạm hành chính đối với trường hợp rò rỉ dữ'
semantic (word overlap) 6 chunks best 0.4529
-> '[Điều 2] Mức phạt vi phạm hành chính đối với trường hợp rò rỉ dữ'
semantic (real embeddings) 6 chunks best 0.4529
-> '[Điều 2] Mức phạt vi phạm hành chính đối với trường hợp rò rỉ dữ'
The three "smart" strategies produce identical results ($0.4529$), and all beat
fixed-size ($0.4108$ — its chunk begins with 't vi phạm', meaning the words "Mức phạt" were
sliced in half). This does not mean semantic chunking is useless: in this NDA the sentence
boundaries coincide with the semantic boundaries, so splitting on punctuation already is splitting on
meaning. The expensive method's advantage does not show up because there is nothing left for it to
improve.
The second document is a meeting transcript — no full stops, no capitals, typed in a hurry. This kind of data is extremely common inside real companies:
=== When does the strategy actually matter? ===
tidy : fixed-size 6 chunks / best 0.4108 recursive 6 chunks / best 0.4529
messy: fixed-size 4 chunks / best 0.5869 recursive 1 chunks / best 0.3553The result inverts completely. On the transcript the punctuation-based splitter finds no full stop at all and so merges the whole document into exactly 1 chunk, dropping the match to $0.3553$ — losing to the dumbest strategy in the file, fixed-size at $0.5869$. The lesson is not "which strategy is best" but this: every clever chunking strategy rests on an assumption about the text's structure, and when that assumption is wrong they fail worse than the naive approach. Look at your actual data before choosing.
15.2 Inside a vector database: HNSW indexing
If the corpus holds 1 million chunks, computing cosine similarity linearly (exact search) between the question and every chunk costs $O(N)$. So vector databases (ChromaDB, Pinecone, Qdrant) use approximate nearest neighbour (ANN) search, typically with the layered graph algorithm HNSW (Hierarchical Navigable Small World).
HNSW is built on a multi-layer graph, much like a skip list:
- The top layer (sparse): holds very few, widely scattered vector nodes. A search takes large jumps here to locate the general neighbourhood of the query vector quickly.
- The lower layers (progressively denser): node density increases. The algorithm descends to refine its path, taking shorter hops to home in on the true nearest neighbours.
The important word is approximate: the algorithm follows the graph and therefore only
ever inspects a small part of the corpus, which means it can miss the genuinely nearest vector.
The parameter controlling that trade-off is usually called ef (the size of the candidate
queue during search): larger ef misses less but computes more.
=== What an approximate index really costs and saves ===
brute force always computes N distances and is always exact.
N = 5000 vectors, 64 dimensions, 50 clusters
ef recall@1 distances vs brute
16 76% 468 10.7x
64 99% 821 6.1x
256 100% 3037 1.6x
N = 20000 vectors, 64 dimensions, 50 clusters
ef recall@1 distances vs brute
16 29% 613 32.6x
64 66% 1448 13.8x
256 100% 2990 6.7xThis table says three things the $O(\log N)$ formula does not:
- There is no free lunch. At $N = 5000$, reaching $100\%$ recall costs $3037$ distance computations — only $1.6$ times faster than a full scan. All of the "many times faster" comes from accepting misses: $ef = 16$ gives a $10.7$ times speedup but recall falls to $76\%$.
- The advantage grows with scale. This is the part that matters. Four times the data (5,000 → 20,000) leaves the work at $100\%$ recall almost unchanged ($3037 \to 2990$), so the saving jumps from $1.6$ to $6.7$ times. Extrapolate to millions of vectors and that gap becomes the hundreds of times that actually justifies ANN.
- The parameters must scale with the data. At the same $ef = 64$: recall is $99\%$ at $N = 5000$ but falls to $66\%$ at $N = 20000$. A configuration that tested fine against 5,000 documents will silently miss a third of its results once the corpus grows — with no error raised anywhere.
15.3 Hybrid search: combining meaning and keywords
Semantic search with dense embeddings is excellent at capturing the gist of a question but routinely fails when the user searches for an exact proper noun, product code or error code. The reason is direct: an embedding model is trained to place things that are close in meaning close together, and two error codes differing by one digit are about as close in meaning as anything can be.
The answer is hybrid search — running two independent retrievers in parallel:
- Dense retrieval: use an embedding model and measure the cosine angle.
- Sparse retrieval: use BM25 (an evolution of Lesson 14's TF-IDF that adds frequency saturation and document-length normalisation).
- $M$ is the set of retrieval methods (dense and sparse).
- $r_m(d)$ is the position of document $d$ in method $m$'s result list (1-indexed).
- $k$ is a smoothing constant, conventionally $60$, which limits the influence of very low-ranked documents.
Measured: where does each retriever fail?
The project builds a small technical-support corpus containing two nearly identical documents differing by
exactly one digit in the fault code: E-1042 for the NX-200 model and E-1024
for the NX-300. Then it runs two queries through all three rankers:
=== Hybrid search: where each retriever alone fails ===
exact fault code : E-1024
bm25 [ok ] doc1=1.777 doc0=0.649 doc2=0.000
dense [ok ] doc1=0.488 doc0=0.421 doc2=0.313
rrf [ok ] doc1=0.033 doc0=0.032 doc2=0.032
paraphrased symptom: máy của tôi phát ra tiếng ồn lớn ở phần làm mát
bm25 [MISS] doc2=1.232 doc0=0.649 doc1=0.649
dense [ok ] doc0=0.616 doc1=0.611 doc2=0.497
rrf [ok ] doc0=0.033 doc2=0.032 doc1=0.032For the exact fault code, both find it — but the gap to the runner-up is what to look at: BM25 gives $1.777$ against $0.649$, decisively. The embedding gives $0.488$ against $0.421$, a margin of only $0.067$. Two documents differing by a single digit are almost indistinguishable in semantic space. Grow the corpus slightly and that ordering will flip.
For the paraphrased query ("my machine is making a loud noise in the cooling part" — using none of the document's own words like "fan" or "rattling"), BM25 picks the wrong document: it grabs the dust-filter cleaning guide purely because they share the word "machine". The embedding understands the intent and picks the fan-fault document.
In both cases RRF picks correctly — not because it is cleverer, but because it only needs one of the two retrievers to rank the right document first. That is the value of hybrid search: when two blind spots do not overlap, fusing them covers both.
15.4 Lesson 15 project: three measurements in one file
The file has exactly three parts matching the three sections above: the chunking comparison over two kinds of document; a hand-written navigable small-world graph with a distance-computation counter; and BM25 + dense + RRF over the support corpus. All standard library, apart from the embedding calls to Ollama.
"""Lesson 15 project: chunking strategies, ANN indexing and hybrid search.
Run: python3 chunking_evaluation.py
Optional: Ollama with an embedding model (`ollama pull bge-m3`) for parts 1c
and 3. Everything else is standard library only.
Three questions, each answered with a measurement rather than a claim:
1. Do the three chunking strategies actually differ? On what kind of text?
2. How much does an approximate index really save, and what does it cost?
3. When does keyword search beat semantic search, and vice versa?
"""
import heapq
import json
import math
import random
import re
import urllib.error
import urllib.request
OLLAMA = "http://localhost:11434"
EMBED_PREFERRED = ["bge-m3", "nomic-embed-text", "mxbai-embed-large"]
# A tidy document: every clause is its own well-punctuated paragraph.
NDA_DOCUMENT = """
Hợp đồng bảo mật thông tin (NDA) của JS-Tools quy định rõ:
[Điều 1] Mọi tài liệu thiết kế hệ thống và mã nguồn dự án đều được phân loại là Mật.
Nhân viên không được chia sẻ thông tin này ra ngoài dưới bất kỳ hình thức nào.
[Điều 2] Mức phạt vi phạm hành chính đối với trường hợp rò rỉ dữ liệu khách hàng lên tới 500,000,000 VND.
Hành vi vi phạm nghiêm trọng có thể dẫn đến việc chấm dứt hợp đồng lao động lập tức mà không bồi thường.
[Điều 3] Thời hạn hiệu lực của thỏa thuận bảo mật kéo dài 5 năm kể từ ngày chấm dứt hợp đồng làm việc tại công ty.
Mọi tranh chấp sẽ được giải quyết tại Tòa án Nhân dân Thành phố Hồ Chí Minh.
"""
# A messy document: a meeting transcript with no reliable sentence boundaries.
# This is the case where punctuation-based splitting has nothing to work with.
TRANSCRIPT_DOCUMENT = """
an ok vay minh chot lai phan deploy nhe ban build xong thi day len staging
truoc da dung day thang len prod nua nhe lan truoc bi roll back met lam
binh ukm ma cai server staging no het dung luong roi day
an vay thi don log di
binh ok de toi don
an chuyen khac nhe ve cai bao gia cho khach hang ben Q
binh cai do ben sales bao la ho muon giam 15 phan tram
an giam nhieu the a thoi de toi hop voi sep tuan sau roi quyet
"""
# ---------------------------------------------------------------------------
# Part 0 - shared helpers
# ---------------------------------------------------------------------------
def tokenize(text):
return re.findall(r"\b\w+\b", text.lower())
def bag_cosine(text1, text2):
"""Cosine similarity over raw word counts. No semantics, just overlap."""
counts1, counts2 = {}, {}
for token in tokenize(text1):
counts1[token] = counts1.get(token, 0) + 1
for token in tokenize(text2):
counts2[token] = counts2.get(token, 0) + 1
vocab = set(counts1) | set(counts2)
dot = sum(counts1.get(w, 0) * counts2.get(w, 0) for w in vocab)
norm1 = math.sqrt(sum(v * v for v in counts1.values()))
norm2 = math.sqrt(sum(v * v for v in counts2.values()))
if norm1 == 0 or norm2 == 0:
return 0.0
return dot / (norm1 * norm2)
def vector_cosine(v1, v2):
dot = sum(a * b for a, b in zip(v1, v2))
norm1 = math.sqrt(sum(a * a for a in v1))
norm2 = math.sqrt(sum(b * b for b in v2))
return 0.0 if norm1 == 0 or norm2 == 0 else dot / (norm1 * norm2)
def ollama_models():
try:
with urllib.request.urlopen(f"{OLLAMA}/api/tags", timeout=5) as response:
return [m["name"] for m in json.loads(response.read())["models"]]
except urllib.error.URLError:
return []
def pick_embed_model():
names = ollama_models()
for wanted in EMBED_PREFERRED:
for name in names:
if name == wanted or name.startswith(wanted + ":"):
return name
return None
def embed(text, model):
body = json.dumps({"model": model, "prompt": text}).encode("utf-8")
request = urllib.request.Request(
f"{OLLAMA}/api/embeddings", data=body,
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
return json.loads(response.read())["embedding"]
# ---------------------------------------------------------------------------
# Part 1 - the three chunking strategies
# ---------------------------------------------------------------------------
def fixed_size_chunk(text, chunk_size=120):
"""Cut every chunk_size characters, regardless of what is there."""
return [text[i:i + chunk_size].strip()
for i in range(0, len(text), chunk_size)]
def recursive_character_chunk(text, chunk_size=120):
"""Split on punctuation first, then pack sentences up to chunk_size."""
sentences = [s for s in re.split(r"(?<=[.\n])\s+", text.strip()) if s]
chunks, current = [], ""
for sentence in sentences:
if current and len(current) + len(sentence) > chunk_size:
chunks.append(current.strip())
current = sentence
else:
current += " " + sentence
if current.strip():
chunks.append(current.strip())
return chunks
def semantic_chunk(text, threshold=0.20, similarity=bag_cosine):
"""Start a new chunk wherever consecutive sentences stop being similar.
`similarity` is injectable so the same function can run on word overlap
(no dependencies) or on real embeddings (part 1c).
"""
sentences = [s.strip() for s in re.split(r"(?<=[.\n])\s+", text.strip())
if s.strip()]
if not sentences:
return []
chunks, current = [], sentences[0]
for index in range(1, len(sentences)):
if similarity(sentences[index - 1], sentences[index]) < threshold:
chunks.append(current)
current = sentences[index]
else:
current += " " + sentences[index]
chunks.append(current)
return chunks
def best_match(query, chunks, score=bag_cosine):
scored = [(score(query, chunk), index) for index, chunk in enumerate(chunks)]
return max(scored)
def compare_chunkers(label, document, query, embed_model=None):
"""Run all strategies over one document and report what each retrieves."""
print(f"=== {label} ===")
print(f" query: {query}")
strategies = [
("fixed-size (120 chars)", fixed_size_chunk(document, 120)),
("recursive (punctuation)", recursive_character_chunk(document, 120)),
("semantic (word overlap)", semantic_chunk(document, 0.20)),
]
if embed_model:
cache = {}
def embed_similarity(a, b):
for text in (a, b):
if text not in cache:
cache[text] = embed(text, embed_model)
return vector_cosine(cache[a], cache[b])
strategies.append(("semantic (real embeddings)",
semantic_chunk(document, 0.55, embed_similarity)))
results = {}
for name, chunks in strategies:
score, index = best_match(query, chunks)
results[name] = (len(chunks), score, chunks[index])
print(f" {name:<28} {len(chunks):2} chunks best {score:.4f}")
print(f" {'':<28} -> {chunks[index][:64]!r}")
print()
return results
# ---------------------------------------------------------------------------
# Part 2 - what an approximate index actually buys
# ---------------------------------------------------------------------------
def squared_distance(a, b):
return sum((x - y) ** 2 for x, y in zip(a, b))
def build_graph(vectors, neighbours=16, sample=200):
"""A navigable small-world graph: every node linked to near neighbours.
Real HNSW stacks several of these graphs in layers and uses a smarter
neighbour-selection heuristic. This single layer is enough to show where
the saving comes from, and where it stops.
"""
graph = {i: set() for i in range(len(vectors))}
for i in range(len(vectors)):
candidates = random.sample(range(len(vectors)),
min(len(vectors), sample))
nearest = sorted((squared_distance(vectors[i], vectors[j]), j)
for j in candidates if j != i)[:neighbours]
for _, j in nearest:
graph[i].add(j)
graph[j].add(i)
return graph
def graph_search(vectors, graph, query, ef=64, entry=0):
"""Greedy best-first walk. Returns (best index, distance computations)."""
calls = [0]
def distance_to(i):
calls[0] += 1
return squared_distance(query, vectors[i])
visited = {entry}
candidates = [(distance_to(entry), entry)]
best = [(-candidates[0][0], entry)]
while candidates:
current_distance, current = heapq.heappop(candidates)
if -best[0][0] < current_distance and len(best) >= ef:
break # everything left in the queue is worse than what we hold
for neighbour in graph[current]:
if neighbour in visited:
continue
visited.add(neighbour)
neighbour_distance = distance_to(neighbour)
if len(best) < ef or neighbour_distance < -best[0][0]:
heapq.heappush(candidates, (neighbour_distance, neighbour))
heapq.heappush(best, (-neighbour_distance, neighbour))
if len(best) > ef:
heapq.heappop(best)
return sorted((-d, i) for d, i in best)[0][1], calls[0]
def make_clustered_vectors(count, dimension=64, clusters=50, spread=0.25):
"""Real embeddings sit in clusters, not spread evenly. Mimic that."""
centres = [[random.gauss(0, 1) for _ in range(dimension)]
for _ in range(clusters)]
vectors = [[x + random.gauss(0, spread) for x in centres[i % clusters]]
for i in range(count)]
return centres, vectors
def measure_index(count, ef_values, trials=100):
"""Recall and distance computations against an exact brute-force scan."""
random.seed(42)
centres, vectors = make_clustered_vectors(count)
graph = build_graph(vectors)
rows = []
for ef in ef_values:
hits, total_calls = 0, 0
for _ in range(trials):
centre = centres[random.randrange(len(centres))]
query = [x + random.gauss(0, 0.25) for x in centre]
exact = min(range(count),
key=lambda i: squared_distance(query, vectors[i]))
found, calls = graph_search(vectors, graph, query, ef=ef)
hits += found == exact
total_calls += calls
rows.append((ef, hits / trials, total_calls / trials,
count / (total_calls / trials)))
return rows
def report_index():
print("=== What an approximate index really costs and saves ===")
print(" brute force always computes N distances and is always exact.\n")
for count in (5000, 20000):
print(f" N = {count} vectors, 64 dimensions, 50 clusters")
print(f" {'ef':>5} {'recall@1':>9} {'distances':>10} {'vs brute':>9}")
for ef, recall, calls, speedup in measure_index(count, (16, 64, 256)):
print(f" {ef:>5} {recall:>8.0%} {calls:>10.0f} {speedup:>8.1f}x")
print()
# ---------------------------------------------------------------------------
# Part 3 - hybrid search: BM25, dense, and RRF fusion
# ---------------------------------------------------------------------------
# Two documents that differ by a single digit in the fault code, plus two
# unrelated ones. This is the shape of a real support knowledge base.
SUPPORT_DOCS = [
"Sự cố mã E-1042: quạt tản nhiệt của model NX-200 kêu to bất thường. "
"Thay quạt theo quy trình bảo hành.",
"Sự cố mã E-1024: quạt tản nhiệt của model NX-300 kêu to bất thường. "
"Thay quạt theo quy trình bảo hành.",
"Hướng dẫn vệ sinh bộ lọc bụi định kỳ ba tháng một lần cho toàn bộ dòng "
"máy NX.",
"Chính sách hoàn tiền áp dụng trong vòng 30 ngày kể từ ngày mua hàng.",
]
def bm25_scores(query, documents, k1=1.5, b=0.75):
"""Classic BM25: TF saturation plus length normalisation."""
tokenised = [tokenize(d) for d in documents]
average_length = sum(len(d) for d in tokenised) / len(tokenised)
scores = []
for document in tokenised:
score = 0.0
for term in tokenize(query):
frequency = document.count(term)
if frequency == 0:
continue
containing = sum(1 for d in tokenised if term in d)
idf = math.log(1 + (len(tokenised) - containing + 0.5)
/ (containing + 0.5))
norm = 1 - b + b * len(document) / average_length
score += idf * frequency * (k1 + 1) / (frequency + k1 * norm)
scores.append(score)
return scores
def ranks_from_scores(scores):
"""Position of each document, 1 = best."""
order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
ranks = [0] * len(scores)
for position, index in enumerate(order, start=1):
ranks[index] = position
return ranks
def reciprocal_rank_fusion(rank_lists, k=60):
"""Combine rankings without ever comparing incomparable score scales."""
return [sum(1 / (k + ranks[i]) for ranks in rank_lists)
for i in range(len(rank_lists[0]))]
def report_hybrid(embed_model):
print("=== Hybrid search: where each retriever alone fails ===")
if embed_model is None:
print(" skipped - no embedding model installed"
" (`ollama pull bge-m3`)\n")
return
document_vectors = [embed(d, embed_model) for d in SUPPORT_DOCS]
queries = [
("exact fault code ", "E-1024", 1),
("paraphrased symptom", "máy của tôi phát ra tiếng ồn lớn ở phần làm mát", 0),
]
for label, query, expected in queries:
sparse = bm25_scores(query, SUPPORT_DOCS)
query_vector = embed(query, embed_model)
dense = [vector_cosine(query_vector, v) for v in document_vectors]
fused = reciprocal_rank_fusion(
[ranks_from_scores(sparse), ranks_from_scores(dense)])
print(f" {label}: {query}")
for name, scores in (("bm25 ", sparse), ("dense", dense),
("rrf ", fused)):
ordered = sorted(range(len(scores)), key=lambda i: scores[i],
reverse=True)
mark = "ok " if ordered[0] == expected else "MISS"
top_scores = " ".join(f"doc{i}={scores[i]:.3f}"
for i in ordered[:3])
print(f" {name} [{mark}] {top_scores}")
print()
print(" On the exact code both find it, but look at the gap to the")
print(" runner-up: BM25 is decisive, while the embedding barely separates")
print(" E-1024 from E-1042 - two documents one digit apart.")
print(" On the paraphrase BM25 picks the wrong document; RRF follows the"
" retriever")
print(" that was right, without ever comparing the two score scales")
print(" directly. RRF numbers are always small and close together: with")
print(" k=60, rank 1 scores 1/61 and rank 2 scores 1/62. Only the order")
print(" matters, never the magnitude.\n")
def main():
embed_model = pick_embed_model()
if embed_model:
print(f"Embedding model in use: {embed_model}\n")
else:
print("No embedding model found; parts 1c and 3 will be skipped.\n")
tidy = compare_chunkers("Tidy document (NDA, clean punctuation)",
NDA_DOCUMENT,
"Mức phạt tiền vi phạm rò rỉ dữ liệu là bao nhiêu?",
embed_model)
messy = compare_chunkers("Messy document (meeting transcript, no periods)",
TRANSCRIPT_DOCUMENT,
"khach hang ben Q muon giam gia bao nhieu phan tram?",
embed_model)
# On tidy text, punctuation already sits on the semantic boundaries, so the
# two strategies agree. That is a property of the text, not of the method.
print("=== When does the strategy actually matter? ===")
for label, results in (("tidy ", tidy), ("messy", messy)):
fixed = results["fixed-size (120 chars)"]
recursive = results["recursive (punctuation)"]
print(f" {label}: fixed-size {fixed[0]} chunks / best {fixed[1]:.4f}"
f" recursive {recursive[0]} chunks / best {recursive[1]:.4f}")
print(" On tidy text the punctuation already sits on the semantic")
print(" boundaries, so every smart strategy agrees and beats fixed-size.")
print(" On the transcript there is no sentence punctuation at all, so the")
print(" recursive splitter collapses to a single chunk and loses to the")
print(" dumbest strategy in the file.\n")
report_index()
report_hybrid(embed_model)
if __name__ == "__main__":
main()How to run this project on your machine
-
Run
python3 chunking_evaluation.py. No libraries needed. For the real-embedding semantic chunking and the hybrid search sections, start Ollama withollama pull bge-m3. -
Allow a minute or two at the indexing section — it builds graphs over 25,000 vectors in pure Python. The
recall and distance-computation figures will match this lesson exactly thanks to
random.seed(42). -
Then try breaking it three ways:
-
In
make_clustered_vectors, changeclusters=50to1— the data no longer forms clusters and becomes uniformly spread. Recall collapses at the sameef: approximate search is viable because real data has cluster structure, not because the graph performs magic. -
In
build_graph, dropneighbours=16to4. The graph gets sparser, distance computations fall but recall falls faster — this is the $M$ parameter every vector database makes you choose at index-creation time, and which you cannot change afterwards. -
In
reciprocal_rank_fusion, changek=60to1. RRF scores spread out enormously and rank 1 dominates completely, making the result close to "just take whichever retriever was most confident" — losing the fusion that a large $k$ provides.
-
In
Lesson summary & what comes next
- Achieved: measuring that a "smart" chunking strategy only wins when the text has the structure it assumes — on a meeting transcript, punctuation splitting collapsed to 1 chunk and lost to plain fixed-size cutting.
- Achieved: implementing an approximate search graph and measuring the real trade-off curve: $100\%$ recall is only $1.6$ times faster at $N=5000$, but $6.7$ times at $N=20000$ — the advantage lives in the scaling, not in a fixed number.
-
Achieved: seeing the same
ef=64give $99\%$ recall at 5,000 vectors but only $66\%$ at 20,000 — a silent degradation that raises no error. - Achieved: building hybrid search with RRF, and measuring each side's blind spot: the embedding separates two one-digit-apart fault codes by only $0.067$, while BM25 picks entirely the wrong document when the question shares none of the document's words.
Bridge to the next lesson: this lesson improved how you store and how you search. But when the user's question is vague to begin with, or when the right document sits at rank 5 rather than rank 1, no index can save you. Lesson 16 addresses exactly those two: rewriting the query (query rewriting) and reordering the results with a cross-encoder (reranking).
Download the practice code for this lesson
The Python file chunking_evaluation.py — the chunking comparison, the navigable small-world
graph with its distance counter, and BM25 + dense + RRF hybrid search (run
python3 chunking_evaluation.py):
📖 Further reading
- Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs — the original HNSW paper (Malkov & Yashunin, 2016)
- Reciprocal Rank Fusion Outperforms Single Retrieval Methods and Hybrid Search Combination — the paper introducing RRF (Cormack et al., 2009)
- LangChain Document Transformers — a detailed guide to production text-splitting strategies (LangChain Documentation)
Comments