Half of My Hybrid Search Was Silently Off

2026-07-09T00:00:00+00:00
TL;DR — I pointed my own retrieval engine at my own blog and it ranked the wrong post first. Not dirty data, not bad embeddings: one word of Rust. Search terms were joined with a space, which SQLite's full-text search reads as AND, so every natural-language question silently zeroed out the keyword half of "hybrid" search and left it running on embeddings alone. Swapping the space for OR</code> moved the right answer from #3 to #1.
Takeaway: retrieval is an engineering surface with its own silent failures — so don't reach for it until your context genuinely won't fit in the model's window. Cache-and-dump (paste it all in, let caching pay for it) is the boring default that wins until it physically can't. </blockquote>
I asked my own search engine for the one post I've written about saving tokens with a Rust CLI. I've written exactly one. It came back third.
❯ ctx search "the post about saving tokens with a rust cli" --mode hybrid --explain
Search: mode=hybrid, alpha=0.60, candidates: 0 keyword + 80 vector

1. [0.60] filesystem:blog / rust-memory-cheatsheet.md
    scoring: keyword=0.000  semantic=1.000  → hybrid=0.600
2. [0.56] filesystem:blog / nes1.md
    scoring: keyword=0.000  semantic=0.927  → hybrid=0.556
3. [0.45] filesystem:blog / rtk-token-killer.md
    scoring: keyword=0.000  semantic=0.745  → hybrid=0.447
</code></pre>
Ahead of the right answer: a Rust memory</em> cheatsheet, and an emulator for the NES</em>. The
post it should have handed me — "How I Cut My Claude Code Token Usage by 90% with
RTK"</a>, tagged tokens</code>, rust</code>, cli</code>, transparently about saving
tokens with a Rust CLI — lost to two posts whose main qualification is that they also say
"Rust" a lot.</p>
Nothing here is dirty. The corpus is 40 blog posts I wrote and proofread. The right document
is in the index. The query is a plain English question. And Context
Harness</a> — the retrieval engine I built and have been
writing this series about</a> — confidently ranked a
cheatsheet about smart pointers above it.</p>
Last post</a> I promised this one would be "what
retrieval actually buys you, and the specific moments it's slop in, slop out." I want to
correct that framing before I use it, because "slop in, slop out" is garbage-in-garbage-out,
and GIGO is the most-written and least-interesting take in all of RAG: clean your chunks, tune
your overlap, and so on. My chunks are clean. What bit me was something else, and — I'll spoil
the ending — it turned out to be one word of my own code. So this is two posts in one: the bug
and the fix, which is the honest shape of building anything; and then the larger question the
bug throws into relief, which is whether you should be reaching for retrieval at all.</p>
The take I'm not writing</h2>
The failure above is not dirty data. It's structural, and it hides inside the word everyone
reaches for when they want retrieval to sound trustworthy: hybrid.</strong></p>
The pitch for hybrid search is genuinely good, which is why everyone repeats it. Keyword search
— BM25</a> over an inverted index — nails exact terms
and whiffs on paraphrase: it'll find "BM25" and miss "that ranking function with term-frequency
saturation." Semantic search — cosine similarity over embeddings — is the mirror image: catches
the paraphrase, misses the exact string. So you run both and blend the scores. Keyword covers
semantic's blind spot; semantic covers keyword's. A safety net with two independent ropes.</p>
Context Harness does exactly this, on purpose. Keyword candidates come from SQLite's FTS5
bm25()</code>. Vector candidates come from brute-force cosine
similarity</a>
over every chunk — no approximate index, just the dot products. Each side is min-max
normalized per
query</a>,
then blended:</p>
hybrid = (1 - alpha) * keyword_norm + alpha * semantic_norm
</code></pre>
with alpha = 0.6</code>, leaning slightly toward semantics. That's the whole scoring model. Two
ropes, one knot. So why did a post that's about</em> saving tokens with a Rust CLI lose to a
memory cheatsheet?</p>
Read the first line of the output</h2>
Search: mode=hybrid, alpha=0.60, candidates: 0 keyword + 80 vector
</code></pre>
Zero keyword candidates.</strong> One rope wasn't there. "Hybrid" quietly degraded to pure semantic
search, and pure semantics decided "rust cli tokens" was closest to a cheatsheet about Rust
memory. I only know one rope was missing because I passed --explain</code>; without it I'd have
gotten a clean-looking ranked list and no hint that half the machinery had switched off.</p>
Why zero? Watch it flip on a single word:</p>
❯ ctx search "tokens rust cli" --mode keyword
1. [1.00] filesystem:blog / rtk-token-killer.md

❯ ctx search "saving tokens rust cli" --mode keyword
(no results)
</code></pre>
tokens rust cli</code> finds the right post instantly, perfect score. Add saving</code> — a word that
isn't in that post — and the entire result set collapses to nothing. That is the behavior of an
AND</em>: every term required, one miss and you get zero. And a natural-language question is a pile
of words your target document mostly doesn't contain (the</code>, post</code>, about</code>, saving</code>, with</code>,
a</code>), so under AND the keyword rope goes slack on exactly the queries a human or a model
actually types. It only holds for keyword-salad input that nobody writes at a chat box.</p>
I didn't design an AND. So I opened the file.</p>
One word</h2>
The query text becomes an FTS5 MATCH</code> string in one small function
(sqlite_store.rs</code></a>):</p>
fn fts_query_from_user_text(query: &str) -> String {
    query
        .split(|c: char| !(c.is_alphanumeric() || c == '_'))
        .filter(|term| !term.is_empty())
        .collect::<Vec<_>>()
        .join(" ")          // ← the bug
}
</code></pre>
It splits the query into words and joins them with a space. And in FTS5, a space between terms
is</em> an implicit AND. So the post about saving tokens with a rust cli</code> becomes a query that
means every one of those words must appear in a single document</em> — which nothing does. The
keyword arm wasn't broken. It was doing precisely what I told it to, and what I told it was
wrong.</p>
You can watch the fix work before changing any Rust, straight against the FTS index — space
(AND) versus OR</code>:</p>
-- implicit AND (space-joined): the current behavior
sqlite> SELECT bm25(chunks_fts), title FROM chunks_fts ...
        WHERE chunks_fts MATCH 'the post about saving tokens with a rust cli' ...;
-- (no rows)

-- OR-joined
sqlite> ... WHERE chunks_fts MATCH 'the OR post OR about OR saving OR tokens OR ... OR cli' ...;
-11.43  rtk-token-killer.md
 -9.01  context-harness-rag-rust.md
 -7.65  rtk-token-killer.md
</code></pre>
OR lets a document match on any</em> term, and BM25 does the rest: common words like the</code> and
with</code> appear everywhere, so their inverse-document-frequency weight is near zero and they
contribute almost nothing; rare, load-bearing words like tokens</code> and rust</code> and cli</code> drive the
score. The right post wins on the terms that matter. So the fix is one word:</p>
-        .join(" ")
+        .join(" OR ")
</code></pre>
The after</h2>
Rebuild ctx</code>, re-run the exact query that started this — same corpus, same index, nothing
re-embedded, only the query construction changed:</p>
❯ ctx search "the post about saving tokens with a rust cli" --mode hybrid --explain
Search: mode=hybrid, alpha=0.60, candidates: 80 keyword + 80 vector

1. [0.85] filesystem:blog / rtk-token-killer.md
    scoring: keyword=1.000  semantic=0.745  → hybrid=0.847
2. [0.69] filesystem:blog / rust-memory-cheatsheet.md
    scoring: keyword=0.224  semantic=1.000  → hybrid=0.690
3. [0.67] filesystem:blog / nes1.md
    scoring: keyword=0.364  semantic=0.878  → hybrid=0.673
</code></pre>
candidates: 80 keyword + 80 vector</code> — both ropes taut. The token-killer post goes from third to
first; the cheatsheet and the emulator, which only ever had the semantic rope, drop below it.</p>



  
  The same query, before and after the one-word change. Left: `join(' ')` — 0 keyword candidates, the right post buried at #3. Right: `join(' OR ')` — 80 keyword candidates, the right post first.</figcaption>
</figure>
That's the entire fix, and it's the whole reason to instrument retrieval in the first place: the
--explain</code> line is what turned "the results feel off" into "one arm reports zero candidates,"
which is a bug you can find in a file instead of a vibe you argue about.</p>
It wasn't a one-off, either. Earlier I'd asked a four-repo index — 446 documents, roughly 700,000
tokens — how its own</em> hybrid scoring normalizes, and the document literally titled
hybrid-scoring-with-min-max-normalization</code> came in second</strong>, beaten by fifteen thousandths of
a point by a vaguer doc that caught the one stray keyword hit. After the fix:</p>
❯ ctx search "how does hybrid scoring normalize keyword and semantic scores" --mode hybrid --explain
1. [1.00] filesystem:ch / 0005-hybrid-scoring-with-min-max-normalization.md
    scoring: keyword=1.000  semantic=1.000  → hybrid=1.000
</code></pre>
First, perfect score. The engine can now find the spec that describes the engine. And the query
that already worked — a keyword-rich set up remote build machines for nix</code>, which was pulling
its right answer at a clean 1.000</code> before — still sits at 1.000</code> after, with the cross-repo
runbook it surfaced from a different</em> repo actually scoring higher than before. On the three
queries that defined the problem, the right answer is now first, and the control didn't regress.</p>
I'm being careful with that sentence, because OR is not free — it broadens the candidate pool,
so somewhere it has to cost me. So I ran a bigger check: fifteen queries by hand across both
corpora, each with the answer I expected written down first. Twelve came back identical, two
improved, and one regressed — how do I run a database in the browser</em> used to put my
database-in-the-browser post</a> first, and now ranks a
closely related SurrealDB-caching post there instead, with the on-the-nose post bumped to
second. OR pulled in a neighbor and the blend preferred it: a real, if small, tax, landing
exactly where I'd have guessed — a keyword-shaped query where AND's precision was quietly doing
work. Better than three cherry-picked wins. Still fifteen queries I chose, graded against
answers I decided were right, which is not a benchmark — it's marking my own homework, and it's
the next post's problem. For now the honest claim is narrow and true: a one-word bug was
zeroing out half of hybrid search on every natural-language query, and now it isn't.</p>
The bug is the argument</h2>
Here's the part that outlasts the fix. A single word — a space where an OR</code> belonged — silently
disabled one of the two retrieval strategies I built the whole engine around, on exactly the
inputs it would see in real use, and it produced confident, plausible, cited</em> wrong answers the
entire time. No crash. No error. A ranked list with scores, every one of them a lie of omission.
I'd have kept trusting it if I hadn't gone looking.</p>
That is the true cost of retrieval, and it's why the interesting question isn't "how do I make
RAG better," it's "should I be running RAG here at all." Because look at what the whole episode
was for</em>: those 40 blog posts are about 60,000 tokens. They fit in a context window with room
to spare. The correct thing to do with a corpus that small is not to retrieve over it — it's to
paste every post into the prompt, let prompt
caching</a> make the repeat
cost trivial, and let the model read all of it. Do that and this bug cannot happen to you</em>,
because there's no ranking step to get wrong. There's no #1 to incorrectly promote, because there
is no list. I introduced an entire class of silent failure to solve a problem — "which twelve
chunks?" — that, at this scale, I did not have.</p>
Cache-and-dump is the monolith of context: unglamorous, and correct right up until it
physically won't fit in the window.</strong> Retrieval is the service you extract when the monolith
overflows — and, exactly like extracting a microservice, the moment you do it you sign up for an
operational surface the monolith never had. Query construction. FTS semantics. Score
normalization. A one-word bug that halves your quality without telling you. You don't take that
on because it's elegant. You take it on because you were forced to. And the reasons that force
you are specific — not "it's cheaper" (with caching, dumping is cheap; the invoice going up
isn't the work getting better</a>), but capability:</p>

Scale past the window</strong> — a corpus you genuinely can't fit, like those four repos.</li>
Relevance you can't predict</strong> — the one document out of hundreds you didn't know you'd need
until the question existed.</li>
A corpus that changes</strong> — so you don't re-shovel and re-cache everything on every edit.</li>
Attention rot</strong> — even when it all fits, more context is not free
accuracy</a>;
a model handed 700K tokens attends worse than one handed the right 5K.</li>
</ol>
That cross-repo runbook the fixed engine surfaced — a doc from one repository answering a
question I'd only documented in another, neither aware the other exists — is retrieval earning
its keep, and it earned it because reason #1 was real: 700K tokens is not going in a window. On
the blog, none of the four apply, so retrieval had no job, and inventing one for it is how I
ended up ranking a smart-pointer cheatsheet above the post I asked for.</p>
This is the same lesson as last post</a>, one layer down.
There I'd chopped a solo project into 78 documents and six agents because decomposition felt</em>
like rigor, and called it microservices-on-myself. Retrieval on a corpus that fits in the window
is the identical mistake: paying the coordination cost of a distributed system to solve a problem
a monolith already had handled. Extract the service when you're forced. Not before.</p>
Distrust the retriever — enough to instrument it</h2>
There's a discipline I lean on hard at work; I'll keep it generic, but it travels: treat the
model as an advisor to be verified, not an authority to be trusted</em>. You don't take its
confident output at face value — you build the system so the output gets checked. Almost nobody
extends that same suspicion to the retriever</em>. Retrieved context gets a pass. It feels like
ground truth: it came from your own documents, it has a similarity score, it's the thing you
point to when someone asks whether the model made it up. But rust-memory-cheatsheet</code> came back
at 0.60</code> for a question about token savings, and that number was never a measure of whether it
was right</em> — it was a measure of embedding proximity with one of two ropes cut, which is a very
different thing wearing the same costume. The model downstream can't tell the difference. It
receives the top chunk as grounding and does what it does with grounding: builds a confident,
well-cited answer on top of it. On a bad ranking, retrieval doesn't remove the confabulation
risk — it supplies</em> the confabulation with a footnote.</p>
So distrust it. But the useful form of distrust isn't a shrug about how retrieval is
treacherous; it's --explain</code>. Instrument the thing so that "this feels wrong" becomes "the
keyword arm returned zero candidates," which is a bug with a location and a fix. Suspicion that
ends in humility is a pose. Suspicion that ends in a diff is engineering.</p>
Where this goes</h2>
Context Harness got one bug better this afternoon, in public, and I'll take it — but the honest
scope of what I proved is: on three queries that used to fail, the right answer is now first. The
reranker I keep deferring</a>
would catch a different, harder class of miss; OR-joining is not the last word on query
construction; and I still can't tell you, with a number, whether the engine is good</em> — only that
it's better on the cases I looked at, which is exactly how you fool yourself.</p>
Which is the next problem, and a harder one than a one-word bug: I've spent this whole post saying
"first" and "better" and "wrong" as if I can measure any of it. I mostly can't — not the way a
benchmark would — and the gap between what I can measure and what actually matters is where the
next post lives.</p>
— Parker Jones</p>
Parker Jones Dev Blog - retrieval

Half of My Hybrid Search Was Silently Off

The after</h2> Rebuild ctx</code>, re-run the exact query that started this — same corpus, same index, nothing re-embedded, only the query construction changed:</p>
