# My unpublished benchmark post said 10x. The honest number is 1.2x to 1.8x.

| | The old draft | The benchmark I'd sign today |
|---|---|---|
| Headline | "10x faster", "2x faster than hnswlib" | **1.20x to 1.79x** hnswlib, **1.30x to 1.43x** faiss |
| Recall | 61% for Foxstash, 40% for hnswlib | **the same** for all three, 98% to 99.5% |
| Threads | Foxstash's batch API on every core | **one thread**, every library |
| Data | 100,000 vectors at 128 dimensions | **SIFT1M and GIST1M**, a million real vectors each, shape checked |

There's a post in my drafts folder called "How foxstash got 10x faster (and beat hnswlib in pure
Rust)". I wrote it a few versions ago, about Foxstash, my Rust vector search library. It never went
out. Reading it this week, I'm glad, because almost every number in it compares two different
things.

## What the draft got wrong

**It compared speed at different recall.** Approximate search trades accuracy for speed, and recall
is the accuracy: how many of the true 10 nearest neighbours a search actually finds. The draft had
Foxstash at 61% recall and hnswlib at 40%, with each library on a different search setting. That
isn't a race. It's two libraries doing two different jobs, and the QPS column can't tell you which
one is faster. Nobody runs retrieval at 61% recall either, which should have been the first clue.

**It counted threads differently.** The headline number came from Foxstash's batch API, which spreads
queries across every core. The table didn't say what the other libraries were given. Further
down, the draft's table for the then-current version had the answer: Foxstash on one thread did 1,322 queries per
second against hnswlib's 4,004. On the one comparison that was close to fair, it was 3x *slower*.

So the post said "2x faster than the C++ baseline" while its own data said "3x slower on one thread,
at a recall nobody uses". I'd have published a headline my own table contradicted.

## The rules that replaced it

The benchmark write-up in the repo now opens with four traps, "every one of which has caught this
project":

1. **Compare at matched recall, not matched settings.** Each competitor's speed is read off its own
   recall curve at exactly Foxstash's recall.
2. **Count the threads.** hnswlib's Python query call uses every core by default. An earlier version of
   the write-up timed that against a single-threaded Rust loop and reported Foxstash as "about 11x
   slower". The same mistake, pointed the other way.
3. **Never run two benchmarks at once.** Running the Python harness while a Rust build held all 16
   cores halved hnswlib's apparent speed, 5,478 queries per second instead of 10,850.
4. **Verify the corpus is the corpus.** The folder named `sift1m` held 10,000 vectors. A benchmark
   against it would have produced a perfectly plausible number for an index 100 times smaller than
   its label. The loader now checks the shape against a manifest and won't run otherwise.

And a fifth, found later: **tune the competition as hard as yourself.** The harness carried a
comment saying to sweep the graph-degree setting on every library, "not just ours", because tuning
your own while pinning theirs manufactures a win. The code under that comment swept hnswlib's and
faiss's settings and pinned Foxstash's. It happened to be rigged *against* us, and when the sweep
finally ran, the published numbers held. But a rule written in a comment and broken in the code
right beneath it is a comment, not a rule.

## The numbers that survive

Single-threaded, at matched recall, on real datasets, against each competitor's best setting:

| Dataset | Foxstash mode | vs hnswlib | vs faiss | Recall@10 |
|---|---|---|---|---|
| SIFT1M (1M × 128-d) | SQ8 | **1.20x** | **1.30x** | 99.5% |
| GIST1M (1M × 960-d) | RaBitQ | **1.79x** | **1.43x** | 98.3% |

These are our own benchmarks, not third-party results, and the costs are published beside them:

- **Full precision is unremarkable.** Foxstash's uncompressed mode is 0.88x hnswlib on SIFT1M, and
  on GIST1M it lands between hnswlib and faiss. The wins come from compressed storage, not from
  the graph or the distance code.
- **faiss builds much faster at high dimension:** 408 seconds against our 1,141 on GIST1M.
- **No single mode wins everywhere.** SQ8 wins at 384 dimensions and below, RaBitQ at 768 and above.
  At 128 dimensions RaBitQ is about 12x slower than SQ8; at 960, SQ8 is no faster than full precision.

Where the speed comes from is the opposite of the old draft's story. Search is limited by memory
latency, not arithmetic: a distance computation costs about one trip to RAM. SQ8 keeps 8-bit codes
in the block the search reads, with the full vectors in a separate array touched only for the final
rescoring. Same graph, same number of distance computations per query, 87.1 nanoseconds each down
to 67.3. Fewer bytes moved, not more cores.

A headline under 2x is a lot less fun to write than 10x. It's also the only one of the two I can defend
line by line.

## What I'd tell you if you were buying this work

When a vendor shows you a speed number for anything that approximates (search, compression,
recommendations, an AI model), ask four things: at what accuracy, on how many threads, on what data,
and was the competitor tuned as hard as they tuned themselves. If any answer is "the defaults" or
"we didn't check", the number is marketing. And ask to see the costs. A benchmark with no losses in
it wasn't run honestly.