Running Lumberroom Cloud
Running EmbeddingGemma 2 on Graviton
November 20, 2026
We wanted Lumberroom Cloud to retrieve the right memory from longer conversations and move relevant results higher in search. EmbeddingGemma 2 gave us a longer input window and better ranking in our evaluation. The question was what it would cost to run.
We serve the quantized text model with llama.cpp on one embedding server, on a host separate from the application machines. Moving the embedder from AMD x86 to Graviton lowered estimated monthly compute costs by about 26%. A short query took about 160 milliseconds to embed on the new host. Longer inputs took more time, even after we fixed a CPU backend choice that had made the first Arm run nearly six times as slow as the old host on a 500-token input.
Why we chose EmbeddingGemma 2 for Cloud
Lumberroom uses embeddings to find memories by meaning. The open-source distribution bundles quantized BGE Base English v1.5, which gives a self-hosted installation a local embedder out of the box. BGE accepts up to 512 tokens. Our EmbeddingGemma 2 evaluation used an 8,192-token window, enough to fit the median session of about 2,240 Gemma tokens. Both models produce 768-dimensional vectors. Gemma also supports multilingual text and code. BGE model card · EmbeddingGemma 2 model card
We compared the two on LongMemEval-S. When each question searched its own session pool, Gemma raised recall at five from 96.2% to 98.0% and NDCG at ten, which rewards ranking relevant results higher, from 88.8 to 93.8 on a 100-point scale. When we pooled the benchmark sessions, the ranking gain narrowed to 2.6 points, with a paired 95% interval of +0.3 to +5.0 points. The recall gain in that harder test was uncertain. Its interval ran from −2.2 to +6.2 percentage points. Our benchmark write-up
The models often saw different amounts of the same conversation. We chose Gemma because the configuration we could deploy gave us a longer view of the input and better ranking in the evaluation. An equal-window comparison would tell us how much of that ranking gain came from the model itself. For the open-source default, BGE remains a bundled local option.
Getting llama.cpp to run well on Graviton
The first Graviton3 run took 13.8 seconds to embed roughly 500 tokens. The old AMD host took 2.44 seconds with the same quantized Gemma model. On Graviton4, llama.cpp still took 10.1 seconds with the backend it selected automatically.
The arm64 container shipped several GGML CPU backends. We tested them and found that keeping only the armv8.2_2 library available brought the 500-token time down from 10.1 to 3.53 seconds on the same Graviton4 instance. On the Graviton3 instance, it fell from 13.8 to 4.25 seconds. We pinned the tested backend for that container image. The timing sweep identified which library worked best for our model and image. It did not isolate the instructions responsible for the gain.
| Rehearsal configuration | Embed threads | ~20 tokens | ~100 tokens | ~500 tokens | ~1,000 tokens |
|---|---|---|---|---|---|
m6a.xlarge, AMD baseline |
2 | 0.17 s | 0.39 s | 2.44 s | 5.09 s |
r7g.large, automatic backend |
2 | 0.18 s | 1.17 s | 13.8 s | 38.2 s |
r7g.large, pinned backend |
2 | n/a | 0.71 s | 4.25 s | n/a |
r8g.large, automatic backend |
2 | 0.16 s | 0.99 s | 10.1 s | 26.6 s |
r8g.large, pinned backend |
2 | 0.16 s | 0.56 s | 3.53 s | 8.19 s |
m8g.xlarge, pinned backend |
4 | 0.08 s | 0.31 s | 1.90 s | 4.22 s |
These are median embedding times from the October 31 rehearsal at approximate input lengths. The short-query column measures the conversion from text to a vector. A complete search also includes database retrieval, ranking and request handling. The four-thread m8g.xlarge measures the benefit of resizing the whole configuration, including the extra threads.
Sizing the embedding host
We initially priced the Graviton3 r7g.large at about $74 a month and the m7g.xlarge at about $104, including disks, public IPv4 and estimated snapshots. After the rehearsal, we chose the two-vCPU Graviton4 r8g.large. Its estimated compute bill was about $60 a month, down from $81 on the AMD host. Including the other infrastructure costs put the estimates at about $79 and $100 respectively. These were sizing estimates, and snapshot costs vary with data changes. AWS R8g specifications
The smaller Graviton kept short-query embedding close to the AMD baseline. The 500-token call took about a second longer. Spending about $112 a month in total on the larger Graviton4 option would have cut that call to 1.90 seconds, about 22% below the AMD baseline, and brought the short query to 80 milliseconds.
We started with the smaller size because the short-query timing held up and the longer-call tradeoff was acceptable. The larger instance stays available if time spent waiting for longer embeds makes the extra compute worthwhile.
What we had to migrate
A new embedding model needs a new set of stored vectors. Even at the same width, BGE and Gemma place text in different vector spaces. We wrote both vectors for new memories while filling a second vector slot in the background, then switched each tenant once its replacement vectors were ready. Keeping the old slot during the transition gave us a rollback path. Lumberroom’s model migration guide
Scores also needed recalibration. Gemma assigned higher cosine similarity to some pairs of memories that human reviewers had judged should stay separate. We raised the automatic dedupe threshold from 0.97 with BGE to 0.995 with Gemma. A stricter cutoff can leave more duplicates in the store, a tradeoff we accepted to reduce the chance of merging distinct memories.
The move from x86 to Arm raised a narrower version of the same problem. We kept Gemma’s weights fixed, but the two runtimes still produced slightly different scores. The largest cosine shift in rehearsal was 0.0044 against our target of 0.001. Some pairs chosen to sit near the dedupe cutoff crossed it. We used the same two-slot migration to regenerate stored vectors on Arm and give queries and stored memories a consistent numerical baseline. The backfill took longer than expected, and we checked the serving path with live search spot checks after the switch.
EmbeddingGemma 2 gave us a way to use more of a long conversation during retrieval. The Graviton move lowered the cost of serving it while keeping short-query embedding close to the old baseline. We know where that saving comes with a latency cost, and how much of that time we can recover with a resize.