Reports and history
Reports
Cache files are unique per scenario. The cache directory is derived
from the feature file's path, not just its name, so
features/admin/login.saffron and features/shop/login.saffron no longer
write to the same files. Flat layouts keep the paths they already have. A
run also refuses to start when two selected scenarios would still map to
one file (two names that slug alike), because silently overwriting a
recording is the worst outcome available.
saffron status --json is the machine-readable overview behind the
IDE panels: every feature file with its scenarios (header line, tags,
outline rows, cached / proposal pending / last status), the tag list
with counts, pending proposals with narrative and verification, the
last run's totals, the run history, vocabulary health (recorded and
unrecorded counts, divergent steps, and duplicate wordings proven by
identical recordings, never by string similarity), the effective
config, and quality: the suite seen from its history (size, each
scenario's latest result, recent failures with their streak, flaky
scenarios that pass and fail by turns, repeated heals, scenarios never
run or left out of the last run, the slowest, pass rate and duration
trends, results per tag). One implementation in the runner, so the JetBrains
plugin, the VS Code extension and any dashboard agree.
Subscription or API key. Saffron runs on whatever the Claude Agent SDK
finds: ANTHROPIC_API_KEY if set, otherwise the Claude Code login on the
machine. The two are reported differently. On an API key the dollar
figure is what was billed. On a subscription no money changes hands per
run, so the summary line and the report headline the 5-hour window
before and after the run (for example 12% → 19%, the same figure
Claude Code's /usage shows) and label the dollars as the
API-equivalent: what the same run costs in CI with a key. The
weekly window is deliberately not shown. Runs also record which
models answered, so cost comparisons across runs are like for like.
The reading comes from an SDK method marked experimental; if it changes
or is unavailable, the run is unaffected and the report falls back to
tokens plus the API-equivalent figure. Codex, Antigravity and Cursor
report tokens but no price or plan window, so their runs show tokens and
"cost not reported by the provider".
Every run writes .saffron/reports/latest.html and latest.json. The HTML
report is a dark, dashboard-style page: KPI strip (pass rate, passed /
adapted / failed, AI calls, in+out tokens, cache traffic, cost),
per-scenario cards with a step timeline, assert/drift chips, agent
narratives, adaptation callouts, verified/unverified proposal pills, and
before/after feature-file diffs. Cache traffic matters: on real agent
sessions, prompt-cache reads and writes are ~95%+ of the true token volume
and cost: Saffron captures both (cacheReadTokens,
cacheCreationTokens) per scenario and in totals, so the visible
"in+out tokens" number is never mistaken for the bill. Cached scenarios
display a green 0 tokens pill. The JSON twin powers CI assertions and
tooling.
Execution replay. With "trace": "retain-on-failure" (or "on"),
a scenario keeps an execution trace, and saffron trace opens it as a
replay: the scenario's own Given, When and Then on the left, each step
opening onto the actions replayed and the calls behind them; the DOM at
the selected action in the middle, rendered as a live page, with a
filmstrip of the run; the failure, retries, call log, network, and what
the agent changed on the right. Saffron's own data sits beside
Playwright's: the retries a step absorbed, the tabs it opened, switched to
or closed onto, the response each waits for / expects a response
check matched (method, path, status), and what the agent cost for the
scenario (calls, tokens, cost, models; providers report it per session,
so a replayed step is simply free). The failed step is selected on arrival.
Every traced scenario of the run opens in one replay, with a scenario
picker. Agent browser calls are included, proof replay sessions are selectable, and
recorded target matches can be highlighted. Only a heal marks steps ✦
healed; a first recording reads "recorded by the agent", a proof replay or
a replay of a pending proposal "ran the proposed recording", and a pending
proposal that failed before the run fell back is shown as its own earlier
attempt. The selected step shows the attempted/proposed actions; the
Saffron panel accepts or rejects the exact pending proposal with
staleness checks (a stale one can still be rejected) and an
acknowledgement for an unverified recording.
{env:VAR} values, including one a data file holds, are masked back to
their tokens before the trace is kept: the event log, DOM snapshots,
stylesheets and every request and response body, URL, header and
WebSocket message, in the spellings the value takes there (JSON escapes,
percent-encoding, HTML entities, CSS escapes); URL schemes and hosts are
kept so the replay can load the page's resources, while a secret
elsewhere in a URL, credentials included, is masked. Values shorter than three characters,
binary encodings and the filmstrip's pixels cannot be masked.
The trace is Playwright's format, rendered by the snapshot engine of the
Playwright Saffron runs with; the replay around it is Saffron's.
Failure screenshots. When a step fails, Saffron photographs the tab it
failed in, at that moment, before anything else touches the page. A red
scenario shows it under the failure ("the page when it failed"), which
usually answers the question on sight: the heading says Catalog where the
step expected Products. A scenario the agent healed keeps the picture
too, labelled as the page when the cached step stopped working, so whoever
reviews the proposal sees what the agent saw. If a scenario is still red
after the agent ran, the picture is taken again at the end, because the
page has moved on. Images are embedded in the HTML report, so it stays one
file you can mail or open inside an IDE, and latest.json carries the
project-relative paths under evidence for tools. screenshot in the
config chooses viewport (default), full-page or off. Green scenarios
carry none, and each run replaces the last one's files.
Run history and trends
Every run appends one trimmed line to .saffron/history.jsonl (committing
it is recommended: it's small, append-only, and keeps trends alive across
clones). From it, each report carries deltas vs the previous run and
20-run sparklines on the pass-rate and cost tiles, plus three
attention panels when they have something to say: chronic scenarios
(2+ heals or 3+ reds in the last 10 runs: re-record instead of paying for
more heals), step divergence (the Level-2 evidence, trended per run),
and recurring failure themes (the most frequent red-run errors across
the last 20 runs: the data source for vocabulary roadmap decisions). The
CLI prints the deltas and chronic warnings after each run.
saffron dashboard shows the suite rather than one run, from the same
history and in the report's design: how many tests, the share passing and
the scenarios failing (each scenario's latest result) and for how many
runs in a row, flaky scenarios (passed and failed by turns over the last
20 runs), scenarios the agent healed again and again, never run and left
out of the last run, the slowest, pass rate and execution time per run,
and results per tag. It writes .saffron/reports/dashboard.html and opens
it; --stdout prints the page, which both editors show as their
Dashboard. The numbers are quality in saffron status --json.
The economics, in one table
Numbers from a real 5-scenario booking-site suite:
| AI recording run | Cached replay | |
|---|---|---|
| Duration | ~10 min | ~7 s |
| AI calls | 133 | 0 |
| Tokens | 25,093 | 0 |
| Cost | $6.49 | $0.00 |