Reference

Roadmap and milestones

Single source of truth for what's planned, what's gated on what, and what "done" means for each item. History and rationale live in the design log; shipped behavior in Features & Architecture. Last updated 2026-09-19.

Where we are

All four engineered loops are closed: recording is verified (zero-AI proof replay + bounded refinement before any proposal is filed), the inner loop is tuned (deliberate snapshots + batching: −60% AI calls, −44% cost), Level-1 step reuse is scope-complete (island seeding, quoted-value generalization, data-table seeding, new-scenario cost scales with novel steps only), and the outer loop ships trend memory: run history, deltas and sparklines, chronic-scenario detection, and the divergence/failure-theme evidence streams that gate M3 and feed the action vocabulary. The Gherkin data vocabulary is complete (M2): key/value and record tables, doc strings, Examples params, all parameterized, all edit-free at replay, all honestly stale on structural change. Step sets shipped with the .saffron dialect (M9): flows written once, invoked anywhere, expanded at parse time.

Shipped	Date
MVP: record → zero-token replay → runtime heal → reviewed accept	Jul 2
Locator heal propagation (`accept --propagate`)	Jul 3
`--strict`, npm packaging, docs site	Jul 3–4
Dynamic-content vocabulary (capture/differs/matches/attribute, `{date±N}`, tabs)	Jul 3
Verified proposals (proof replay + refine pass)	Jul 4
Cost composition capture (cache read/write tokens)	Jul 4
Inner-loop optimization (snapshot-mode none, batching)	Jul 4
Level-1 step reuse: index, islands, quoted-value, data tables	Jul 5
M1. run history, trend deltas & sparklines, chronic-scenario detection, failure themes	Jul 5
M4. assertion policy v2: opt-in adaptable-mid, final assertion block always strict	Jul 5
M2. record tables (`<table:row:col>`), doc strings (`<docstring>`), cross-step params	Jul 5
M9. step sets & the `.saffron` dialect (`StepSet:` define / `StepSet` invoke, parse-time expansion)	Jul 6
M10. IDE integration: `saffron steps`/`--snippets`, `saffron mcp`, VS Code extension (sibling repo)	Jul 6
M8. duplicate-wording rename proposals (behavior-proven) + `saffron author` + `saffron lsp` (JetBrains/Neovim)	Jul 6
M5 core: cross-browser replay (firefox/webkit), parallel workers, `healModel`	Jul 7
M6. PUBLIC RELEASE: `saffron-ai@0.1.0` on npm, free for any use, closed source	Jul 7
M12 Tier 1: `{env:VAR}` secrets: masked caches/reports/history, preflight, `.env`	Jul 7
M2 leftover: dialogs, file uploads, drag-and-drop, iframes (replay on all 3 engines)	Jul 7
Recording ergonomics: live dialog/drag/iframe recording (passive-listener fix, frame-fallback asserts)	Jul 19
Network vocabulary (waitForResponse / expectResponse): wait on or assert backend API calls (URL/method/status/body matchers)	Jul 19
Agent skill: `skills/saffron/SKILL.md` ships in the package; `saffron init` installs it, registers `saffron mcp` per host, adds the AGENTS.md/CLAUDE.md block	Sep 13
Multi-tag `--filter` (OR) + scenario tags in report JSON / history for test-management keying	Sep 13
JetBrains plugin (saffron-jetbrains-plugin): `.saffron` file type + bundled grammar + LSP4IJ auto-wiring; built, tested, Marketplace publish pending	Sep 13
Wrong-recording recovery: `saffron run --rerecord`, recorder keeps typed values (`pressSequentially`), empty-fill guard stamps UNVERIFIED, `accept --all` skips UNVERIFIED	Sep 13
Retries visible: per-step `retried ×N` chip and terminal note (retries stay per action)	Sep 13
M13: guided first run, `saffron init --examples` installs the Saucedemo suite with config and .env.example	Sep 13

The road ahead, at a glance

Saffron roadmap: shipped work and the milestones ahead

The diagram groups shipped work by what it lets you do; the milestone numbers below are the build order it came from. The vocabulary track (M9 step sets fed M8 dedupe, which fed M10 IDE support) is shipped, and so is the public release (M6): saffron-ai is on npm. The runtime track (M5) is independent, M3 stays a gated decision rather than a scheduled milestone, and the SaaS layer (M7) is the long-term horizon. Full scope and rationale for each milestone below.

M1. Outer loop: run history & trends

Status: SHIPPED 2026-07-05

.saffron/history.jsonl (one trimmed line per run, commit recommended); report deltas vs previous run + 20-run pass-rate/cost sparklines; chronic detection (2+ heals or 3+ reds in 10 runs, warned in CLI and report); per-run divergence recorded (the M3 evidence stream); recurring failure themes aggregated from red-run errors (the M2 data source).

M2. Step & data vocabulary completion

Status: SHIPPED 2026-07-05 (record tables + doc strings)

Multi-row record tables (N columns + header row, a list of records): per-cell parameterization: recorded values become <table:row:col> references resolved from the scenario's own table at replay. Cell-value edits replay at zero tokens; row-count or header changes alter the key set and are honestly stale (no fragile per-row iteration detection, the agent records the actual actions, the cells parameterize).
Doc strings (""" blocks): parsed, rendered to the agent, and recorded as <docstring> references: content edits replay free; literal recordings stay protected by exact-content staleness.
Cross-step references: unambiguous table/docstring params are scenario-wide (like Examples params), so a later assertion on the note's text follows content edits too. Validated live on the demo guestbook: record $0.53, then full value+content rewrite replayed GREEN at 0 tokens, row addition went honestly stale.
Leftover SHIPPED 2026-07-07: the four known action-vocabulary candidates: handleDialog (armed before its trigger), uploadFile (input or armed file chooser, portable relative paths), dragTo (two semantic targets), and iframe-scoped targets (target.frame). Replay proven by 12 real-browser unit tests plus the four widget scenarios in the example suite, green on all three engines. Recording ergonomics fixed 2026-07-19: the dialog auto-dismiss pathology was Saffron's own bug, not the MCP's, the orchestrator's launching Playwright client dismisses any dialog on a page with no local listener, destroying it before the agent's CDP client could present it. Agent sessions now run with passive dialog listeners armed (withDialogsVisible), and all four capabilities record live: dialog $0.66 (was $5.33 under the workaround), drag $0.18, iframe $0.24, upload $0.30, every proposal proof-verified. Frame-less assertions also gained a replay fallback that searches child frames, so a recording that misses an iframe boundary still verifies.

M3. Level 2: shared step library

Status: gated (decision, not schedule)

Gate: Level-1 divergence data. The step index already reports how often identical step text carries different recordings. If real suites show divergence is rare (the user's Cucumber experience predicts this, authors parameterize instead of forking), Level 2 is safe; if it's common, Level 1

propagation remains the right architecture and this milestone is dropped, not postponed.

Scope if greenlit:

Shared step recordings as the source of truth; scenario caches hold references.
Heal ping-pong detection: a shared step healed in opposite directions within N runs is flagged polymorphic → split the step text or fork it.
Per-scenario overrides (shadowing) as the escape hatch.
Affected-scenario re-verification: updating a shared step queues zero-AI proof replays for every referencing scenario.
saffron show <scenario>: flatten references into the effective action list for review.

M4. Assertion policy v2

Status: SHIPPED 2026-07-05

assertionPolicy: "strict" | "adaptable-mid" (config or --assertion-policy, default strict). Under adaptable-mid, mid-scenario checkpoint assertions may be adapted by the agent, with an adaptation note, suggested .feature edit, proof replay, and the accept gate, while the final assertion block (the trailing run of assertion steps, the scenario's verdict) stays strict under every policy: never merged from agent recordings, always re-verified as written, never refined around. Live-validated both ways on a broken mid checkpoint: strict → red, adaptable-mid → yellow with the checkpoint adapted and verified.

M5. Runtime & providers

Status: core SHIPPED 2026-07-07 (3 of 5) · remaining items demand-driven

Shipped:

Cross-browser replay: --browser firefox|webkit (config browser). Replay is plain Playwright, so the whole committed suite runs green on all three engines (validated on the example suite). Recording/healing stay Chromium (the agent attaches over CDP): under another browser, a scenario needing the agent fails honestly with "record with --browser chromium, then replay cross-browser" guidance.
Parallel workers: --workers N (config workers). Zero-AI replays run concurrently, one browser context per worker; anything needing the agent is re-run sequentially with full agent support, so healing keeps its exact-browser-state semantics (cost: one extra ~100ms replay for those scenarios). ~2× on the tiny example suite; scales with suite size.
healModel: --heal-model (config healModel), a cheaper model for heal sessions and their refine passes; recordings keep the main model. Plumbed per-run through the provider (ctx.model).

Remaining (picked up when demand appears):

LLM-less MCP replay: an alternative replay engine that drives the Playwright MCP server programmatically (no AI) for environments that standardize on MCP.
Multi-provider agents: an OpenAI-backed AgentProvider beside the Claude one: the interface boundary already exists.

M6. Public release (free, closed-source)

Status: SHIPPED 2026-07-07, saffron-ai@0.1.0 on npm (reshaped from the original open-source plan, see the design log's "Distribution" entry). Published as saffron-ai, one brand across npm, the site, and the public repo; verified from the public registry on a clean install. The bare saffron npm name is squatted by a deprecated package: a name dispute remains an option later.

Free to use, including commercially, under the Saffron Free Use License v1.0 (replaces Apache-2.0: relicensed cleanly while nothing had been distributed). Users configure ANTHROPIC_API_KEY or a Claude Code login; replay-only CI needs no credentials.
Closed-source packaging done: single bundled+minified CLI (esbuild, no sourcemaps), npm tarball ships dist-pkg + README + LICENSE only; standalone install smoke-tested (--version, steps, run --no-agent all green from the tarball).
First-run preflight done: missing-credentials warning with setup instructions instead of an SDK stack trace.
Feedback channel: GitHub issues on the public site repository (saffron-ai), linked from the site.
Published 2026-07-07; the site quickstart (npm i -D saffron-ai) is now literally true.

M7. Open-core SaaS layer

Status: long-term · after public release traction

Hosted layer on top of the free CLI: team dashboard, cache-review UI with visual diffs (where a Level-2 registry would also earn its place), scheduled cloud runs, run-history analytics across projects. The CLI must never require it.

M8. Step vocabulary & authoring

Status: SHIPPED 2026-07-06 (reader-friendly guide with diagrams: Step Sets & Vocabulary)

Shipped: duplicate-wording rename proposals, after any successful recording, a step whose action list is IDENTICAL (actionsEqual) to an indexed step with different text gets a rename feature-edit appended to the proposal (adaptation note explains; accept --with-feature-edit converges the wording; empty no-op action lists never match; divergent canonicals are skipped). saffron author <prose-file>, drafts a .saffron file from plain paragraphs with the vocabulary injected as required wording; live-validated: authored file reused the vocabulary verbatim, invoked the existing StepSet, and would record almost fully seeded ($0.12 to author). Original scope notes below.

Kills the Cucumber pain of the same step existing under two wordings ("I navigate to the home page" vs "I visit the home page"). In Saffron the pain has a price tag: exact step text is the cache identity, so a second wording is an index miss that re-records at token cost. The step index derived from committed caches is the registry; this milestone surfaces it (no second, hand-maintained registry, those rot):

saffron steps: SHIPPED early with M10 Phase A; list/search the vocabulary (files ∪ caches) with status badges, usage counts, --json export and --snippets; --json export for tooling.
Duplicate detection with rename proposals (mechanical, zero AI): after a recording, a new step whose actions are identical (actionsEqual) to an indexed step with different text is behavior-proven duplicate wording: Saffron files a .feature rename-edit proposal to the canonical wording via the existing feature-edit machinery. Each duplicate costs one recording, then the vocabulary converges. (Cucumber structurally cannot do this, it never observes that two wordings produce identical behavior.)
saffron author <prose-file>: the agent drafts a .feature from plain-paragraph requirements, with the step index injected as the required vocabulary (prefer existing wordings exactly). Synergy: authored files are maximally seedable, so their first recording is mostly zero-AI islands. Interaction: duplicate detection sharpens the Level-1 divergence data that gates M3. IDE-side prevention (completion + near-duplicate diagnostics while typing) is M10, built on this milestone's index plumbing.

M9. Step sets & the `.saffron` dialect

Status: SHIPPED 2026-07-06 (reader-friendly guide with diagrams: Step Sets & Vocabulary; shipped behavior also in Features)

Sets are project-wide (registry scans every .saffron file; library-file convention features/shared.steps.saffron for application-wide flows). Live-validated on the demo guestbook: two .saffron scenarios sharing a step set recorded with 4/5 steps seeded ($0.61 total incl. verification), replayed at 0 tokens; a table-value edit inside the set stayed green; a reworded set step made BOTH invoking scenarios honestly stale; heal feature-edits route to the set definition line.

Kills the repeated-step-group pain (the same 5–6 checkout-prelude steps pasted into every scenario of a file). A step set is a named, reusable sequence of steps, defined with a first-class StepSet: keyword in a .saffron file: a superset dialect of Gherkin (the TypeScript playbook: .ts = .js + keywords, honest new extension, tooling maps back cleanly):

Feature: Checkout Flow

StepSet: Complete guest information
    Given I am on guest information page
    When I enter guest name "Chathuranga Jayasinghe"
    And I click continue button
    Then I am not on guest information page

Scenario: Verify checkout happy path
    Given I am on calendar page
    ...
    And I click choose button
    StepSet Complete guest information
    Then I am on checkout page
    ...

Syntax (locked 2026-07-06): definition is StepSet: <name> (block keyword, with colon, like Scenario:); invocation is StepSet <name> (step position, no colon, like Given). One spelling, symmetric. Sets are a .saffron-exclusive feature: plain .feature files stay fully supported for everything else, and adopting sets in an existing file is a pure rename to .saffron with zero content edits (superset, the TypeScript playbook).

The set shape convention (from the worked example): first step is a guard ("Given I am on guest information page"), last step is an exit assertion ("Then I am not on guest information page"), every invocation carries its own precondition and success check.
Assertions inside sets get position-dependent strictness, for free: after expansion the set's Thens are ordinary assertions of the calling scenario: strict by default, adaptable only under adaptable-mid, and when a set is invoked as the last thing in a scenario its trailing Thens join the final assertion block (strict under every policy, forever). No new rules; expansion does it.
Parse-time expansion, exactly like Background merging: sub-steps keep their exact text identity, so caching, Level-1 seeding, heal propagation, and staleness all work with zero engine changes. Editing a set makes its scenarios honestly stale, but re-recording is almost fully seeded, so convergence is cheap.
Implementation: a pre-processor rewrites StepSet: lines to Scenario: same-line (line numbers preserved 1:1) and feeds the standard @cucumber/gherkin parser, no fork. The real cost is the edit-back path: featureEdits/heal proposals need step provenance ({file, line, fromSet}) because an expanded step's source line lives in the set definition, possibly in another file, and editing it fixes every caller at once (they go stale, re-record seeded).
Guard rails: set names are project-wide unique (same-file definitions resolved first), which also catches the colon-typo at an invocation site (an accidental StepSet: block duplicates an existing name → parse error with a "remove the colon to invoke" hint); invoking an unknown set is a parse error; an And directly after an invocation inherits kind from the set's last step: docs recommend an explicit keyword on the step after an invocation.
v1 rules: sets may contain tables, doc strings, and <param> placeholders (resolved from the calling scenario's Examples row); no nested sets; sets never run standalone but their steps join the seeding vocabulary; reports group expanded steps under the invocation with a "from StepSet" badge.
Docs ship the one-line VS Code files.associations and .gitattributes snippets that give .saffron highlighting and GitHub rendering on day one.

M10. IDE integration: `.saffron` language support & step completion

Status: SHIPPED 2026-07-06 · marketplace publish rides with M6 · reader-friendly guide: Step Sets & Vocabulary

Shipped surface: saffron steps [search] [--json] [--snippets] (the vocabulary command: Phase A also delivered the core of M8's saffron steps), saffron mcp (stdio server: search_steps, list_step_sets), saffron lsp (the same experience for editors beyond VS Code: JetBrains via the free LSP4IJ plugin on Community editions or the built-in LSP API on Ultimate, Neovim via lspconfig; completion with badges, StepSet go-to-definition, hover, diagnostics, served by the real parser), and the Saffron VS Code extension living in the sibling repository ../saffron-vscode (zero-dependency: .saffron grammar, badge-carrying completion, StepSet completion, go-to-definition, hover, unknown/duplicate-set and near-duplicate wording diagnostics). The extension is published on the VS Code Marketplace (ChathurangaJayasinghe.saffron-vscode, 2026-08-15).

While typing a .saffron (or .feature) file, the editor suggests the project's existing steps, so duplicate wording is prevented at the keyboard, before it ever costs a recording.

Two framing decisions:

The completion vocabulary is the union of parsed files + the cache index, not caches alone: files are the source of wording (a teammate's unrecorded steps must complete too); caches add status. Every suggestion carries a badge no Cucumber tool can offer, ✓ recorded (replays at zero tokens), ⚠ divergent, ○ unrecorded, authors literally see which steps are already paid for.
Completion alone is half the cure: the author who confidently types a new wording never opens the popup. The layer that kills the duplication pain is a near-duplicate diagnostic (squiggle: "similar to 'I navigate to the home page', recorded, used in 4 scenarios" + rename quick-fix). IDE prevention before recording; M8's behavior-proven detection after: same pain attacked from both sides.

Phased by cost:

Phase A: zero-extension stopgap (can ship with M9): saffron steps --json (machine-readable vocabulary export) and saffron steps --snippets writing .vscode/saffron.code-snippets, VS Code completes snippets natively, regenerate after each run. Crude matching, no badges, works immediately.
Phase B: AI authoring surface: saffron mcp, a small MCP server exposing search-steps / list-step-sets over the live index. In 2026 "someone manually writing a file" is often a Copilot/Claude session, assistants can't read completion popups, they need the vocabulary as context. Same vocabulary humans get, served to machines (and the interactive twin of saffron author).
Phase C: the real language tooling: a Language Server (LSP, so JetBrains/Neovim reuse it) + thin VS Code client: TextMate grammar with first-class StepSet: highlighting; step completion with the status badges; StepSet-name completion after StepSet ; go-to-definition on invocations (what makes .saffron feel like a real language); hover showing a step's recorded actions, usage count, and last-recorded date; diagnostics, unknown StepSet name, duplicate set definitions, near-duplicate step wording.

Ruled out: piggybacking the official Cucumber VS Code extension, it completes from step definitions (glue-code regexes), which Saffron deliberately doesn't have; it has nothing to index here.

M13: Guided first run from `saffron init`

Status: SHIPPED 2026-09-13

saffron init offers to install a small example suite so a new user's first saffron run has something real to record and replay:

Prompt: "Install example tests? (y/N)". Non-interactive runs (CI, --yes/--no-examples) never block.
The examples target the public https://www.saucedemo.com demo shop (login, cart, checkout), written the way the runner rewards and in .saffron files only: short scenarios, {env:VAR} for the credentials, one Scenario Outline, StepSets in a library file invoked at the start of scenarios and in the middle, tags for --filter.
saffron.config.json is written with baseURL https://www.saucedemo.com, retries: 1, strict: false, and a .env.example documenting SAUCE_PASSWORD.
The examples ship inside the npm package next to the skill (examples/saucedemo/) so they stay versioned with the runner; init copies them into features/ only when the folder is empty.
Docs: a "Your first run" page that walks the user through record, review, accept and the zero-token replay using exactly these files.

M14. Run from the IDE

Status: SHIPPED 2026-09-16 (JetBrains plugin 0.2.0, VS Code extension 0.2.0; side panel in 0.2.1), with one follow-up open.

Shipped: a Saffron run configuration type in JetBrains IDEs (run with files, folders, tags, replay-only, headed, re-record and extra arguments; report; accept), right-click → Run on .saffron files and folders in both IDE families, the Saffron tool window in JetBrains IDEs (feature files with scenario counts, search, Run Selected / Run All, Open Report, Accept All Proposals, last-run totals, pending proposals), and the matching Saffron side panel in VS Code (feature files with scenarios as children, checkboxes and Run selected, replay-only and headed toggles, last-run totals, pending proposals). Both panels grew into dashboards on top of saffron status --json (2026-09-17): a Proposals list to tick and accept or reject with the narrative and proof-replay verdict, Tags to tick and run, vocabulary health (divergent steps, duplicate wordings proven by identical recordings) with the effective config, and the run report embedded in the IDE (JCEF tab in JetBrains, webview editor tab in VS Code) instead of re-drawn charts. VS Code runs commands in a dedicated terminal from the workspace folder; JetBrains runs them through the Run tool window and keeps the configuration in the run widget.

Open follow-up, gutter run icons per Scenario (the Cucumber-plugin feel), needs two things first:

Single-scenario runs in the CLI: a features/login.saffron:12 form (file plus line, Cucumber-style) so the icon has something to call. Today saffron run selects by file, folder and tag only.
Scenario structure in the editor: the JetBrains plugin colours .saffron files through a TextMate grammar, so there is no PSI to hang a line marker on. The practical route is the language server publishing a code lens ("Run scenario") above each Scenario line, which both LSP4IJ and VS Code render.

Also under consideration: --exclude <tags> for "everything except".

M15. Multi-tab and multi-session scenarios

Multi-tab: SHIPPED 2026-09-18. Plain prose ("switch back to the first tab", "close the preview tab", "open a new tab at the admin page"); the agent's tab operations are cached as switchTab, closeTab and newTab (0-based, creation order) and replay in order. Clicks that open tabs are still followed automatically; an explicit switch is not undone by the follow.

Multi-session (two browsers in one scenario): PLANNED. "User A sends a message, user B sees it" needs a session concept in the step language, something like As "buyer" I open the cart mapping to named browser contexts, carried through recording (one agent, several contexts), cache (each action tagged with its session), replay (one context per session, separate storage states) and the report. Cross-domain scenarios need nothing: absolute URLs outside baseURL are cached as written.

M16. Failure evidence: screenshots and video

Status: PLANNED. A red scenario shows its error text today and nothing of what the page looked like. Planned: a screenshot of the active page at the moment an action or assertion fails, embedded in the report; optional video per scenario (video: off | on | retain-on-failure); a predictable, git-ignored .saffron/artifacts/ layout that saffron prune keeps small; and artifact paths in latest.json so both IDE panels can open them. Pixels can show secrets that text masking cannot reach, and the docs will say so.

M17. Execution replay: time-travel debugging

Status: PLANNED. Not a video: a replay of a run you can step through, with the page inspectable at every action and the network alongside, navigated by your Given, When and Then.

Parallel and cross-browser: what is left

Both are shipped in their core form (M5): --workers N replays concurrently, and --browser chromium | firefox | webkit replays on any engine. Planned on top: sharding across CI machines (--shard 2/4), parallel recording and healing under a concurrency cap, per-worker timing in the report; branded channels (chrome, msedge), a browser matrix in one run with per-browser results, and device and viewport profiles.

M12. Secrets: `{env:VAR}` tokens

Status: Tier 1 SHIPPED 2026-07-07 (Tier 2, the agent-blind MCP proxy, remains the follow-up). Live-validated end to end: an agent recording that TYPED the real credentials produced artifacts containing only tokens: grep for the secret across proposal, cache, reports and history found zero hits; replay resolved from env at 0 tokens; a missing var failed fast by name; .env filled gaps with real env winning.

Credentials must never live in feature files, and in Saffron that contract has to hold across SEVEN surfaces: feature files, committed caches, proposals, the agent conversation, reports, history.jsonl, and terminal errors.

Tier 1 (scoped):

{env:VAR} tokens, deliberately the {date+N} execute-time family, NOT <param>: they resolve lazily in executeAction, so every replay path (runs, proof replays, seeded islands, assertion guards) is covered by one resolver, and the seeding machinery is untouched.
Caches/proposals store only the token: recording masks referenced env VALUES back to tokens (the deparameterize mechanism).
Output hygiene: narratives, adaptations, error strings, reports, and history get a masking pass over the referenced values only, the environment is never enumerated.
Preflight: missing env vars fail fast, by name, before the run.
.env auto-load from the project root (git-ignored; real env always wins), so local dev and CI behave identically.
Honest limitation, documented: during RECORDING the agent must type the real value, so it transits the AI provider, replays never do. Recommendation: record with rotatable staging credentials.

Tier 2 (follow-up): agent-blind secrets, an MCP proxy wrapped around the Playwright MCP server: the model sees and types only the token; the proxy substitutes the real value into the browser and masks the code-echo on the way back. The model never holds the secret. Residual (inherent): a secret the page DISPLAYS appears in snapshots.