LongMemEval-S
Published LongMemEval benchmark; 500 questions, with 470 scored after excluding 30 abstentions. SHA-256 d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442.
Read the headline
What 459 of 470 means.
LongMemEval-S contains 500 questions over synthetic chat histories. Thirty abstention questions are excluded from this aggregate, leaving 470 scored questions. Each scored question names the session or sessions containing its evidence.
Firekeep received the question, returned its first ten retrieval results, and earned one hit when any of those results came from a labeled evidence session. It earned 459 hits and missed 11. The exact ratio is 97.65957446808511%, displayed as 97.7%.
459questions hit
÷
470questions scored
=
97.7%Recall@10
This measures retrieval. It asks whether the labeled evidence reached the context window. It does not score the final answer a language model might generate from that evidence.
Complete result
The headline and the shape behind it.
Recall answers whether any labeled evidence appeared. Coverage asks how much of all labeled evidence appeared. Mean reciprocal rank rewards finding the first relevant result early. NDCG rewards relevant results throughout the top ten, with more weight near the top.
97.6596%Evidence Recall@10
92.8014%Evidence Coverage@10
91.7354%Mean reciprocal rank
76.8179%NDCG@10
| Question type | Hits | Scored | Recall@10 |
|---|---|---|---|
| Knowledge update | 72 | 72 | 100% |
| Multi-session | 121 | 121 | 100% |
| Single-session assistant | 56 | 56 | 100% |
| Single-session preference | 29 | 30 | 96.6667% |
| Single-session user | 62 | 64 | 96.875% |
| Temporal reasoning | 119 | 127 | 93.7008% |
Frozen protocol
Exactly what ran.
The run began with empty, isolated Qdrant, Neo4j, and Redis stores. Authentication was disabled to match the benchmark's single-workspace posture. The fixture's sessions were serialized inside each namespace, with up to eight namespaces in flight.
Commit-pinned
mxbai-embed-large
mxbai-embed-large:latest, digest 468836162de7f81e041c43663fedbbba921dcea9b9fefea135685a39b2d83dd8, through Ollama 0.32.4 on an NVIDIA GeForce RTX 5080.
Raw, top ten
format=raw, top_k=10, token_budget=10000, with eight recall workers across independent namespaces.
Ingestion covered 23,867 fixture session rows and 124,366
/memory/learn calls, including 13 repeated session keys. HTTP 200 responses with
status=partial were rejected and retried rather than counted as completed.
Acceptance gates
The run had to clear every gate before scoring.
- Fresh-store count was zero before ingest.
- 124,366 expected source calls completed; zero ingest errors.
- 124,263 Qdrant points matched 124,263 distinct graph memory IDs.
- 500 recall records covered 500 unique question IDs.
- Zero recall errors, missing questions, or ledger-gap questions.
- Zero queue or dead-letter items after ingest and recall.
- Qdrant payload hash was identical before and after recall.
- Selected lifecycle and scorer state was identical before and after recall.
Accepted artifact hashes
- Recall JSONL
- 6797da4989244a946f1aaacf7e79a23ed4030a797e9b4b23f7e4bc3af2a2685f
- Score JSON
- 5a1fee799c6f9104d30a1e33da42da26ca0c5396769305badca735a1059cc2f0
- Qdrant payload
- 911a5c91963f8211f01a0824ce0e3b3d7cd4aba15f0449ec25b996d70b39a942
- Lifecycle state
- 9df0b18b1422f7999413ca88df9488224fa88197e397ee9c217a15d4502c10bf
Dated comparison
The same questions, paired with the August 3 result.
The 470 scored question IDs were paired against the previously published run. The current result adds 74 net hits, a change of 15.7447 percentage points. The comparison describes the measured difference; it does not assign that difference to a single cause.
385 → 459questions with labeled evidence in the top ten
| Outcome | Questions |
|---|---|
| Evidence found in both runs | 384 |
| Evidence missed in both runs | 10 |
| Previous miss → current hit | 75 |
| Previous hit → current miss | 1 |
The exact two-sided McNemar p-value is 2.0381730293306602e-21.
Reproduce and inspect
Follow the same frozen path.
- Check out the benchmarked Firekeep build.Use commit
1d4405a256e922a1494f251bb8ef78cd81f9d382. - Verify both input digests.Match the LongMemEval-S dataset and embedding-model hashes published above.
- Start empty isolated stores.Require zero Qdrant, Neo4j, and Redis benchmark state before ingest.
- Execute every fixture call.Preserve ordering and repeated rows, and retry partial responses; require 124,366 successes.
- Clear the store and queue gates.Verify unique counts, queue depth, and dead-letter state before recall.
- Run the frozen retrieval row.Use raw output,
top_k=10, andtoken_budget=10000across all 500 questions. - Score the 470 non-abstention questions.Reject the run on any recall, ledger, integrity, or immutability failure.
Result dataCurrent machine-readable resultMetrics, slices, gates, hashes, and missed question IDs.
MethodFull accepted methodologyFrozen inputs, execution design, acceptance gates, and reproduction outline.
HistoryPrevious dated resultThe earlier artifact remains available rather than being overwritten.
Interpretation
Read the result at the right altitude.
These boundaries make the number more useful: they say exactly where it applies.
- Retrieval, not final-answer accuracy.The result measures whether labeled evidence reached the first ten results.
- One corrected deterministic run.It is not a multi-run confidence interval.
- GPU embedding runtime.Sampled CPU comparisons showed small floating-point score drift while preserving all sampled top-ten memberships and orders.
- Synthetic chat histories.LongMemEval-S differs from Firekeep's normal distilled production memories with provenance.