# We measured whether our agents actually follow their instructions

Every team running AI coding agents ships them an instruction file — recall
before you answer, record what you learn, declare risky actions before taking
them. We had never seen anyone measure whether those instructions are obeyed.
So we measured ours, on our own production deployment, and published the
numbers. Some of them are embarrassing. One of the first things the
measurement caught was us.

## Why we started measuring

Four separate times, a Firekeep capability existed, worked, and was never used
— until one paragraph landed in the rendered instruction block with a
concrete, observable trigger. A clarification board that tool descriptions
alone never fired. Recall triggers that sat idle while storage and retrieval
worked perfectly. A feedback signal the whole ranking layer depended on,
living only in a docstring. A calibration system with no predictions in it,
because nothing told agents to state their confidence.

The pattern was too consistent to ignore: the binding constraint on an agent
system is not the tools — it is the instruction layer. And that layer is
written by hand, on judgment, and never measured. If a paragraph of
instructions is what makes or breaks a capability, then the paragraph deserves
what every other production system gets: monitoring.

## The first table

Firekeep records every session as a structured replay trace and computes
quality metrics per session from that record. Compliance is then a
deterministic predicate over those stored metrics — no model in the loop.
The founding measurement, computed 2026-08-11 on our own live deployment,
over the 32 sessions in the evaluation window:

| Instruction (as rendered to agents) | Predicate | Compliance |
|---|---|---|
| "Recall before you answer" | memory_read_count > 0 | 18/32 — 56% |
| "Write as you go" (record learnings) | memory_write_count > 0 | 15/32 — 47% |
| Recalled knowledge visibly used | recall_used_rate > 0 | 8/32 — 25% |
| Working state kept current | context_snapshot_count > 0 | 20/32 — 62% |
| "Declare consequential actions" | brier_score is not None | 0/32 — 0% |
| Outcome-bearing events ≥ 2 | outcome_event_count >= 2 | 10/32 — 31% |

Read it plainly: barely half of sessions recalled team memory before
answering. A quarter visibly used what they recalled. And the instruction to
declare consequential actions — shipped the day before with a fresh client
release — had reached zero sessions, which is not disobedience but a rollout
fact, and the difference matters. This table now lives on the product
dashboard, recomputed live, with the predicates frozen so every future number
stays comparable to this baseline.

## What a number like this can and cannot claim

A compliance rate measures **behavior** — whether sessions did the instructed
thing. It does not measure whether doing it helped. Our outcome signal is
still too thin to support quality claims, so any statement like "recalling
first improved results" would currently be fiction, and the dashboard says so
on the same surface that shows the rates.

The way out of that limitation is recursive, and it is why the 0% row matters
most: the instructions most worth enforcing first are the ones that create
outcome signal. Feedback compliance populates per-memory usefulness. Declared
predictions populate calibration scores. Completion discipline populates real
success and failure. Every compliance win makes the quality question more
answerable.

## The measurement caught its own authors first

Within a day of shipping the table, an external review filed six findings
against it. We fixed four the same evening. Then, while building the next
round, we discovered that one of the review's "confirmed" findings was itself
wrong — the reviewer had reasoned from call sites without checking a category
gate between the emitter and the scorer, and our fix had dutifully relabeled a
row to say the opposite of what the code does. The row was measuring genuine
agent discipline all along; our correction had overclaimed in the other
direction. Both the finding and the counter-finding are recorded, dated, in
the design spec.

The second catch was worse. The "declare consequential actions" experiment was
believed to reach agents through two delivery channels: the rendered
instruction file and the MCP handshake. Tracing the receiving end showed the
second channel never existed — the local gateway discards backend handshake
text and serves its own, which did not carry the instruction. Our test suite
asserted the server sends the text; nothing ever asserted an agent receives
it. The instruction layer had a silent delivery failure, and no measurement
could have caught it, because exposure was not being measured.

If your agent instructions have never been measured, this is the shape of what
you don't know: not just "are they obeyed" but "did they even arrive."

## A rate is only as honest as its denominator

That delivery failure set the design for round two. A compliance rate over
"all sessions" conflates three populations: sessions that received the
instruction and complied, sessions that received it and didn't, and sessions
the instruction never reached. Counting that third group against the
instruction blames the text for the postal service.

So sessions now carry **exposure receipts**. The rendered instruction block is
stamped with a content hash; the client re-hashes what is actually on disk at
session start — not what the installer intended, what is there — and reports
it, with the runtime and client version, as attribution labels on the session
record. The compliance table splits every rate three ways: exposed, not
exposed, and unknown — and everything unverifiable is unknown, never counted
against anyone. Nothing backfills, and the dashboard discloses it.

## The pre-registered experiment, and its first movement

The 0% row doubles as a natural experiment, pre-registered in the design spec:
the instruction shipped with a recorded 0/32 baseline, so the brier_score row
moving off zero — or not moving — is the first Living Instructions
measurement. It moved: as of this writing the row reads 1/35. One session
proves nothing about effect size; the honest statement is that the row left
zero after the instruction arrived and not before. Fittingly, that first
compliant session reads unknown in the exposure column — it ran on a client
from before the receipts shipped.

## Now Firekeep grades the work — not just the call that ended it

A completed session used to count as a success by default: the completion call
returned, so the session "succeeded." That is the RPC working, not the work —
and the layer that ranks recall by what actually helped was quietly reading a
constant that said everything worked, always. We rebuilt that from the ground
up.

A session's grade now names the task. On completion an agent reports whether the
work **succeeded, failed, or partially landed**; the grade is bound to the
session's verified owner, written once, and can never be overwritten or erased.
An ungraded completion records nothing — never a default success. "We finished"
and "it worked" are finally, structurally, different events, and everything
downstream reads the real one.

That grade is the newest row on the compliance table above, delivered — like
every instruction that ever changed behavior here — through the one channel that
actually reaches agents: the completion tool's own description. It is a
pre-registered experiment, shipped with its baseline recorded, so the row's
movement is evidence rather than a story told after the fact. The early numbers
are early, and we publish them early. That is the whole method.

## The grade ships with the check that keeps it honest

Self-grading has one failure mode, and it is the one that matters: an agent that
reports success on work that failed. An optimistic grading signal is worse than
none — it is the confident wrong answer this entire system exists to refuse. So
the grade never ships alone. For every session that claims success, a detector
cross-examines the claim against evidence the agent did not author — recorded
tool failures in the same trace, a session abandoned rather than completed — and
surfaces how often success was asserted over a contradiction. It is
visibility-only by design: it judges the grader, gates nothing, and reports *not
enough data yet* below a disclosed sample rather than a flattering zero. A system
that grades itself and refuses to audit that grade is not measuring; it is
marketing. This one audits.

The same principle runs one layer deeper. Memories, and now reusable Skills,
carry an efficacy score built only from sessions with a real graded outcome.
Unproven entries sit at neutral; a penalty with no fresh evidence decays back to
neutral instead of compounding; no single agent can bury a memory the team
shares. The ranking earns its weight from evidence and gives it back in
evidence's absence — it never ratchets, and it never invents a verdict it cannot
support.

## Where this goes, and what we won't let it do

Low-compliance instructions become rewrite jobs drafted by the fleet's own
agents; every draft goes through human approval — instruction text is a
prompt-injection surface, so no exceptions — and approved variants are A/B
validated across real sessions once volume supports it. The system never
rewrites its own instructions autonomously, in any round. The models never
change; the interface to them does, under measurement.

The compliance table ships with Firekeep — the Autopilot tab computes it live
from your own sessions, with your own numbers, which will not be ours.

- Product: https://firekeep.ai/
- Install: https://firekeep.ai/docs.html
- HTML version of this page: https://firekeep.ai/instruction-compliance.html