The delete that didn't delete

Foxstash shipped three releases this week: 0.7 on Tuesday, 0.8 on Wednesday, 0.9 on Saturday. Foxstash is my Rust retrieval library, the part of a search or AI system that finds the right passages, running inside your application with no server.

A changelog would tell you what's new. This is the other list: four things that were wrong, and the specific moment each one got caught. Three of the four came to light because of the same reviewer, and it wasn't me.

Some context first, because it matters to every story. I maintain Foxstash, and I'm also building an internal meeting recorder that uses it to search transcripts. Claude Code sessions help me with both, one per project, on separate machines. This week those sessions messaged each other directly, worked each problem through between them, and brought it to my attention with the evidence attached. So the library has a customer who reads every change, asks pointed questions, and has to live with the answers. That turned out to be the most valuable reviewer of the week.

1. The 80-second shortcut

0.8 added encryption at rest: you supply a cipher and a key, and Foxstash encrypts the write-ahead log, the checkpoints and the index snapshots. Foxstash never holds the key.

The index snapshot was the awkward one. It holds the vectors, so it has to be covered, and wiring encryption into the index format is real work. The shortcut was obvious: when encryption is on, don't save the snapshot at all, and rebuild the index from the encrypted checkpoint every time the store opens. No plaintext vectors on disk. Much less code.

I asked for a measurement before anyone committed to it. Rebuilding at 768 dimensions took 23 seconds at 45,000 documents, 42 seconds at 90,000, and 72 seconds at 180,000. Loading the snapshot took about one second. That's 77 to 125 times slower, and at the meeting recorder's scale of about 200,000 documents it puts an 80-second wait in front of a desktop app every time it opens. So the snapshot gets encrypted, not dropped.

Then the design went to the meeting recorder's session with one question: will this cost you too much? It came back with a design review instead. The first draft authenticated each encrypted file but didn't bind it to anything, so an old snapshot, or one copied from a different store, would have opened cleanly. That's a rollback attack, and nobody had thought about it. Now every file's authenticated data names its role, the store it belongs to, and its generation. A wrong key, a tampered file or a rolled-back file is a hard error. It never quietly falls back to a rebuild, because a slow success that hides tampering is worse than a loud failure.

2. The free lunch that wasn't

EmbeddingGemma 2 lets you keep only the first 512, 256 or 128 of its 768 dimensions. Smaller vectors, smaller index. The question was what that costs in search quality.

The first benchmark said nothing. Recall at 99.9%, at every size, in every storage mode. It got written up as a free storage win and recommended.

The meeting recorder's session read the table and pointed out the flaw. Each truncated run had been scored against exact search at the same truncated size. So the 128-dimension index was being asked "do you find what a perfect 128-dimension search would find?" and the answer was yes. But the real question is whether it finds what the full 768-dimension embedding would have found, and the yardstick couldn't measure that, because it had shrunk along with the thing it was measuring.

Scored against the full embedding: 91.7% at 512 dimensions, 81.9% at 256, 67.1% at 128. A third of the true neighbours gone at the smallest size. That's a real trade of quality for memory, not a free lunch, and it's how the numbers appear on the Foxstash page today.

3. The delete that didn't delete

This is the one that gave the week its name.

The meeting recorder needs a retention feature: when a meeting is deleted, its text and its vectors have to leave the disk. Its session asked a question with an assumption folded into it: "I've written my test assuming delete, then compact, and the content is gone from every file. Is that right?"

It wasn't. Delete in Foxstash is a soft delete: it writes a "removed" record and hides the document from search. The original text is still in the log. If the document was checkpointed, it's in the checkpoint. And it's in the index snapshot, which stores the passage text alongside the vector.

Compact is the cleanup. It rebuilds from the documents that are still live, writes a clean checkpoint and snapshot, and prunes old ones. But it keeps the previous checkpoint as a fallback, because the setting that controls that defaults to 2. So after one compact, the checkpoint from before the delete is still on disk, with the deleted meeting inside it, in plain text.

The part worth sitting with is that the test was green. The test inserted a document, deleted it and compacted, without ever making a checkpoint in between. So the document only ever lived in the log, and one compact always clears the log. The test passed because it skipped the one step that causes the failure. In real use, a meeting that's been around for a while has been checkpointed.

0.9 adds purge. It deletes, then compacts as many times as the retention setting requires, and returns a report proving that nothing older than the purge survives. If anything does, that's an error, never a partial success. It comes with a test that proves the old way does leave the data behind, so the guarantee isn't vacuous: a test that can't fail proves nothing.

And one honest limit, written into the docs and the website: purge deletes files. It doesn't overwrite the disk blocks underneath them, which is normal for any database. If deleted data must be unrecoverable, pair purge with encryption at rest and rotate the key. I won't claim more than that.

4. The position that wasn't an identity

A collection that reranks its results was holding every full-precision vector twice: once in the index and once in the document store. Holding it once cut the index's footprint from 74.3 MiB to 15.8 MiB at 20,000 documents of 768 dimensions. Results are identical.

The near-miss was in the obvious implementation: when the index needs a vector, ask the document store for document number N, where N is the index's node number. It reads naturally. It's also wrong, because the index reorders its nodes when it builds, for speed. Node 5 isn't the fifth document. Nothing had ever depended on the two matching, which is exactly why it was easy to assume they did.

It wouldn't have crashed. It would have reranked every result against some other document's vector and returned slightly worse answers, the kind a quick test waves straight through. A design review of the actual code caught it before any was written. The fix maps each node to its document from the record the builder already keeps of where each node came from, and a test checks that searching through the shared vectors gives exactly the same results and scores as the old way.

Also this week

0.9 is a breaking release. It removes TurboQuant, a storage mode that had been deprecated since 0.7 because TurboRabit beats it at every bit budget we measured. Removing a public option is the kind of decision a maintainer shouldn't infer, so the work stopped and asked first, with the size of the change in hand: 161 references across five crates. After the removal the core test count dropped from 291 to 284, and the seven missing tests were accounted for one by one instead of being waved off as "still green".

The Foxstash page on our site is updated for 0.9, and every claim on it traces to the code.

If you're buying software

Ask a vendor what "delete" means in their system, and ask to see the test. Most deletes hide data rather than erase it, and that's often fine, as long as everyone knows which one they're getting. Then ask who reviews their work besides them. The best review I got this week came from the one user who'd have to live with the result. If a vendor can't name anyone like that, the first one is probably you.

Built while the rest are sleeping. Narcoleptic Fox, Orlando FL.