Reports and history
Reports
Cache files are unique per scenario. The cache directory is derived
from the feature file's path, not just its name, so
features/admin/login.saffron and features/shop/login.saffron no longer
write to the same files. Flat layouts keep the paths they already have. A
run also refuses to start when two selected scenarios would still map to
one file (two names that slug alike), because silently overwriting a
recording is the worst outcome available.
saffron status --json is the machine-readable overview behind the
IDE panels: every feature file with its scenarios (header line, tags,
outline rows, cached / proposal pending / last status), the tag list
with counts, pending proposals with narrative and verification, the
last run's totals, the run history, vocabulary health (recorded and
unrecorded counts, divergent steps, and duplicate wordings proven by
identical recordings, never by string similarity) and the effective
config. One implementation in the runner, so the JetBrains
plugin, the VS Code extension and any dashboard agree.
Subscription or API key. Saffron runs on whatever the Claude Agent SDK
finds: ANTHROPIC_API_KEY if set, otherwise the Claude Code login on the
machine. The two are reported differently. On an API key the dollar
figure is what was billed. On a subscription no money changes hands per
run, so the summary line and the report headline the 5-hour window
before and after the run (for example 12% → 19%, the same figure
Claude Code's /usage shows) and label the dollars as the
API-equivalent: what the same run costs in CI with a key. The
weekly window is deliberately not shown. Runs also record which
models answered, so cost comparisons across runs are like for like.
The reading comes from an SDK method marked experimental; if it changes
or is unavailable, the run is unaffected and the report falls back to
tokens plus the API-equivalent figure.
Every run writes .saffron/reports/latest.html and latest.json. The HTML
report is a dark, dashboard-style page: KPI strip (pass rate, passed /
adapted / failed, AI calls, in+out tokens, cache traffic, cost),
per-scenario cards with a step timeline, assert/drift chips, agent
narratives, adaptation callouts, verified/unverified proposal pills, and
before/after .feature diffs. Cache traffic matters: on real agent
sessions, prompt-cache reads and writes are ~95%+ of the true token volume
and cost: Saffron captures both (cacheReadTokens,
cacheCreationTokens) per scenario and in totals, so the visible
"in+out tokens" number is never mistaken for the bill. Cached scenarios
display a green 0 tokens pill. The JSON twin powers CI assertions and
tooling.
Run history and trends
Every run appends one trimmed line to .saffron/history.jsonl (committing
it is recommended: it's small, append-only, and keeps trends alive across
clones). From it, each report carries deltas vs the previous run and
20-run sparklines on the pass-rate and cost tiles, plus three
attention panels when they have something to say: chronic scenarios
(2+ heals or 3+ reds in the last 10 runs: re-record instead of paying for
more heals), step divergence (the Level-2 evidence, trended per run),
and recurring failure themes (the most frequent red-run errors across
the last 20 runs: the data source for vocabulary roadmap decisions). The
CLI prints the deltas and chronic warnings after each run.
The economics, in one table
Numbers from a real 5-scenario booking-site suite:
| AI recording run | Cached replay | |
|---|---|---|
| Duration | ~10 min | ~7 s |
| AI calls | 133 | 0 |
| Tokens | 25,093 | 0 |
| Cost | $6.49 | $0.00 |