Evaluating Lumberroom
Our Embedding Benchmark Needed a Better Test
October 9, 2026
EmbeddingGemma 2 retrieved the right session in the top five for 98% of our first LongMemEval-S questions. The existing bge-base model reached 96.2%.
That looked like a useful improvement. Before using it to justify an embedding change in Lumberroom, I needed to understand what the score represented.
The first answer was narrower than the headline: each question searched only its own roughly forty-eight sessions. A larger-pool test then exposed problems in our harness. Repairing those problems produced a much less decisive comparison, with an input-window difference still unresolved.
Retrieval was the part we measured
LongMemEval evaluates long-term conversational memory. Our harness used its data to test retrieval through Lumberroom's actual write and search routes: write a session as a document, search with the question, and check whether a gold session appears in the returned results.
Recall at five, or R@5, asks whether the right session makes the first five results. Ranking measures such as NDCG and reciprocal rank also care about where relevant sessions appear.
We did not generate an answer from those sessions or ask a judge whether it was correct. These are retrieval results, not the benchmark's answer-accuracy score.
Both models used the same remote embedding adapter and the engine's normal hybrid search path against a scratch server and PostgreSQL database. The benchmark configuration also relaxed the credential tripwire and write-size cap so it could store whole sessions. Running through the product's routes did not make every setting production-equivalent.
The candidate pool changed the question
The first mode gave each of the 500 questions its own session pool. That is a useful controlled comparison. It also keeps unrelated sessions from other questions out of the ranking.
I asked for a corpus-wide mode that wrote the shared collection first, then ran every search against it.
The first attempt did something else: it wrote one question's sessions and searched immediately, then continued to the next question. Early questions faced a small pool; late questions faced a much larger one. Its scoring could also reject another question's copy of a relevant session because that copy had a different identity in the harness.
Both runs eventually hit a timeout without a completed report. Their partial recall figures were not usable model results.
Writing once needed clearer accounting
The corrected corpus-wide path writes each unique session ID before the first search and credits matching session identities across questions.
Review then found another distinction. The dataset's 23,867 session references contained 19,195 unique session IDs. Those IDs rendered to 18,464 distinct texts. The server could collapse repeated text into an existing row, so an accepted write was not necessarily a new database row.
Those counts answer different questions. I could not describe the pool size by copying the accepted-write counter.
The checked-in comparison and run reports record the corrected accounting. They also retain a caveat in the bge corpus-wide run: seven session writes were refused during a connection failure. None was a gold session, so the reported metric was unaffected, but the run did not meet the harness's zero-refusal rule.
The larger pool weakened the recall claim
For the recorded comparison, R@5 looked like this:
| Retrieval mode | bge-base | EmbeddingGemma 2 |
|---|---|---|
| Each question's own pool | 96.2% | 98.0% |
| Shared corpus | 35.0% | 37.0% |
These rows describe different tasks. The lower shared-corpus scores do not mean retrieval became worse in production.
LongMemEval's questions simulate different users. Pooling their histories makes a question about one person's preferences compete with similar facts from hundreds of other personas. That is a demanding distraction test, rather than a forecast for one person's memory store.
The uncertainty mattered as much as the point values. In the scoped comparison, the paired 95% interval for the R@5 improvement excluded zero. In the shared corpus it ran from -2.2 to +6.2 percentage points. That result did not establish a clear recall advantage. Of the shared-corpus metrics reported with paired intervals, only NDCG@10's interval excluded zero.
The model still had a longer view
There was a remaining confound in both modes. bge had a 512-token input window; Gemma had 8,192. A median session was about 2,240 Gemma tokens, so the models often received different portions of the same conversation.
The comparison measured the configurations I was considering operating. It could not attribute the gain to the embedding model alone.
A Gemma run constrained to the same token window would help separate those effects. That control had not been run. A character cap would not substitute for it: token density varies within and between conversations.
The outcome was a more useful evaluation record: completed run reports, clearer session accounting, separate retrieval modes, and an explicit missing control. Before carrying a score into the next model decision, I want the candidate pool and the input each model saw beside it.