# Firekeep LongMemEval-S benchmark: 97.7% Evidence Recall@10

Firekeep retrieved at least one labeled evidence-bearing session in the first
10 results for **459 of 470 scored LongMemEval-S questions**.

[Machine-readable result](https://firekeep.ai/benchmarks/longmemeval-s-20260828.json) ·
[Full methodology](https://firekeep.ai/benchmarks/METHODOLOGY.md) ·
[HTML report](https://firekeep.ai/longmemeval-benchmark.html)

Accepted August 28, 2026 by the Firekeep team.

## What 459 of 470 means

LongMemEval-S contains 500 questions over synthetic chat histories. Thirty
abstention questions are excluded from this aggregate, leaving 470 scored
questions. Each scored question names the session or sessions containing its
evidence.

Firekeep returned its first 10 retrieval results and earned one hit when any
result came from a labeled evidence session. It earned 459 hits and missed 11:

`459 / 470 = 97.65957446808511%`, displayed as **97.7% Evidence Recall@10**.

This measures retrieval: whether labeled evidence reached the context window.
It does not score the final answer a language model might generate from that
evidence.

## Complete result

| Metric | Result |
| --- | ---: |
| Evidence Recall@10 | **97.6596%** |
| Evidence Coverage@10 | **92.8014%** |
| Mean reciprocal rank | **91.7354%** |
| NDCG@10 | **76.8179%** |

Recall answers whether any labeled evidence appeared. Coverage asks how much
of all labeled evidence appeared. Mean reciprocal rank rewards finding the
first relevant result early. NDCG rewards relevant results throughout the top
10, with more weight near the top.

### Evidence Recall@10 by question type

| Question type | Hits | Scored | Recall@10 |
| --- | ---: | ---: | ---: |
| Knowledge update | 72 | 72 | 100% |
| Multi-session | 121 | 121 | 100% |
| Single-session assistant | 56 | 56 | 100% |
| Single-session preference | 29 | 30 | 96.6667% |
| Single-session user | 62 | 64 | 96.875% |
| Temporal reasoning | 119 | 127 | 93.7008% |

## Frozen build and protocol

- **Dataset:** [LongMemEval-S](https://arxiv.org/abs/2410.10813), 500 questions, SHA-256
  `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`.
- **Scored population:** 470 non-abstention questions; 30 abstentions excluded.
- **Firekeep:** build
  [`1d4405a256e922a1494f251bb8ef78cd81f9d382`](https://github.com/kapella-hub/FirekeepHQ/commit/1d4405a256e922a1494f251bb8ef78cd81f9d382).
- **Embeddings:** `mxbai-embed-large:latest`, digest
  `468836162de7f81e041c43663fedbbba921dcea9b9fefea135685a39b2d83dd8`.
- **Runtime:** Ollama 0.32.4 on an NVIDIA GeForce RTX 5080.
- **Recall:** `format=raw`, `top_k=10`, `token_budget=10000`, eight workers.

The run began with empty, isolated Qdrant, Neo4j, and Redis stores.
Authentication was disabled to match the benchmark's single-workspace posture.
The fixture contained 23,867 session rows and produced 124,366
`/memory/learn` calls, including 13 repeated session keys. HTTP 200 responses
carrying `status=partial` were rejected and retried.

## Acceptance gates

- Fresh-store count before ingest: zero.
- Source calls: 124,366 expected, 124,366 completed, zero errors.
- Final unique memories: 124,263 Qdrant points and 124,263 distinct graph IDs.
- Recall: 500 records, 500 unique question IDs, zero errors.
- Missing questions and ledger-gap questions: zero.
- Queue and dead-letter items after ingest and recall: zero.
- Qdrant payload and selected lifecycle/scorer state were unchanged by recall.

Accepted hashes:

| Artifact | SHA-256 |
| --- | --- |
| Recall JSONL | `6797da4989244a946f1aaacf7e79a23ed4030a797e9b4b23f7e4bc3af2a2685f` |
| Score JSON | `5a1fee799c6f9104d30a1e33da42da26ca0c5396769305badca735a1059cc2f0` |
| Qdrant payload | `911a5c91963f8211f01a0824ce0e3b3d7cd4aba15f0449ec25b996d70b39a942` |
| Lifecycle state | `9df0b18b1422f7999413ca88df9488224fa88197e397ee9c217a15d4502c10bf` |

## Paired comparison with August 3

The same 470 question IDs were paired against the previously published result.
The previous run found evidence for 385 questions; the current run found it for
459. That is 74 net additional hits, a change of 15.7447 percentage points.

| Outcome | Questions |
| --- | ---: |
| Evidence found in both runs | 384 |
| Evidence missed in both runs | 10 |
| Previous miss → current hit | 75 |
| Previous hit → current miss | 1 |

The exact two-sided McNemar p-value is `2.0381730293306602e-21`. This
comparison describes the measured difference; it does not assign that
difference to a single cause.

## Reproduction outline

1. Check out Firekeep build
   `1d4405a256e922a1494f251bb8ef78cd81f9d382`.
2. Verify the dataset and embedding-model digests above.
3. Start empty isolated stores and verify every initial count is zero.
4. Execute all 124,366 fixture calls with the frozen ordering, repeated-row,
   and partial-response rules.
5. Verify unique memory counts, queue state, and dead-letter state.
6. Run raw retrieval with `top_k=10` and `token_budget=10000` for all 500
   questions.
7. Score the 470 non-abstention questions and reject the run on any recall,
   ledger, integrity, or immutability failure.

## Limits

1. The result measures evidence retrieval, not generated-answer accuracy.
2. It is one corrected deterministic run, not a multi-run confidence interval.
3. The accepted run used a GPU embedding runtime. Sampled CPU comparisons
   showed small floating-point score drift while preserving all sampled
   top-10 memberships and orders.
4. LongMemEval-S uses synthetic chat histories. Firekeep normally stores
   distilled memories with provenance, so production data has a different
   shape.

## Evidence files

- [Current machine-readable result](https://firekeep.ai/benchmarks/longmemeval-s-20260828.json)
- [Accepted methodology](https://firekeep.ai/benchmarks/METHODOLOGY.md)
- [Previous dated result](https://firekeep.ai/benchmarks/longmemeval-s-20260803.json)
- [Exact Firekeep build](https://github.com/kapella-hub/FirekeepHQ/commit/1d4405a256e922a1494f251bb8ef78cd81f9d382)