My unpublished benchmark post said 10x. The honest number is 1.2x to 1.8x.

There's a post in my drafts folder called "How foxstash got 10x faster (and beat hnswlib in pure Rust)". I wrote it a few versions ago, about Foxstash, my Rust vector search library. It never went out. Reading it this week, I'm glad, because almost every number in it compares two different things.

What the draft got wrong

It compared speed at different recall. Approximate search trades accuracy for speed, and recall is the accuracy: how many of the true 10 nearest neighbours a search actually finds. The draft had Foxstash at 61% recall and hnswlib at 40%, with each library on a different search setting. That isn't a race. It's two libraries doing two different jobs, and the QPS column can't tell you which one is faster. Nobody runs retrieval at 61% recall either, which should have been the first clue.

It counted threads differently. The headline number came from Foxstash's batch API, which spreads queries across every core. The table didn't say what the other libraries were given. Further down, the draft's table for the then-current version had the answer: Foxstash on one thread did 1,322 queries per second against hnswlib's 4,004. On the one comparison that was close to fair, it was 3x slower.

So the post said "2x faster than the C++ baseline" while its own data said "3x slower on one thread, at a recall nobody uses". I'd have published a headline my own table contradicted.

The rules that replaced it

The benchmark write-up in the repo now opens with four traps, "every one of which has caught this project":

1. Compare at matched recall, not matched settings. Each competitor's speed is read off its own recall curve at exactly Foxstash's recall.
2. Count the threads. hnswlib's Python query call uses every core by default. An earlier version of the write-up timed that against a single-threaded Rust loop and reported Foxstash as "about 11x slower". The same mistake, pointed the other way.
3. Never run two benchmarks at once. Running the Python harness while a Rust build held all 16 cores halved hnswlib's apparent speed, 5,478 queries per second instead of 10,850.
4. Verify the corpus is the corpus. The folder named sift1m held 10,000 vectors. A benchmark against it would have produced a perfectly plausible number for an index 100 times smaller than its label. The loader now checks the shape against a manifest and won't run otherwise.

And a fifth, found later: tune the competition as hard as yourself. The harness carried a comment saying to sweep the graph-degree setting on every library, "not just ours", because tuning your own while pinning theirs manufactures a win. The code under that comment swept hnswlib's and faiss's settings and pinned Foxstash's. It happened to be rigged against us, and when the sweep finally ran, the published numbers held. But a rule written in a comment and broken in the code right beneath it is a comment, not a rule.

The numbers that survive

Single-threaded, at matched recall, on real datasets, against each competitor's best setting:

These are our own benchmarks, not third-party results, and the costs are published beside them:

• Full precision is unremarkable. Foxstash's uncompressed mode is 0.88x hnswlib on SIFT1M, and on GIST1M it lands between hnswlib and faiss. The wins come from compressed storage, not from the graph or the distance code.
• faiss builds much faster at high dimension: 408 seconds against our 1,141 on GIST1M.
• No single mode wins everywhere. SQ8 wins at 384 dimensions and below, RaBitQ at 768 and above. At 128 dimensions RaBitQ is about 12x slower than SQ8; at 960, SQ8 is no faster than full precision.

Where the speed comes from is the opposite of the old draft's story. Search is limited by memory latency, not arithmetic: a distance computation costs about one trip to RAM. SQ8 keeps 8-bit codes in the block the search reads, with the full vectors in a separate array touched only for the final rescoring. Same graph, same number of distance computations per query, 87.1 nanoseconds each down to 67.3. Fewer bytes moved, not more cores.

A headline under 2x is a lot less fun to write than 10x. It's also the only one of the two I can defend line by line.

What I'd tell you if you were buying this work

When a vendor shows you a speed number for anything that approximates (search, compression, recommendations, an AI model), ask four things: at what accuracy, on how many threads, on what data, and was the competitor tuned as hard as they tuned themselves. If any answer is "the defaults" or "we didn't check", the number is marketing. And ask to see the costs. A benchmark with no losses in it wasn't run honestly.