Reference

Roadmap and milestones

Single source of truth for what's planned, what's gated on what, and what "done" means for each item. History and rationale live in the design log; shipped behavior in Features & Architecture. Last updated 2026-10-30.

Where we are

All four engineered loops are closed: recording is verified (zero-AI proof replay + bounded refinement before any proposal is filed), the inner loop is tuned (deliberate snapshots + batching: −60% AI calls, −44% cost), Level-1 step reuse is scope-complete (island seeding, quoted-value generalization, data-table seeding, new-scenario cost scales with novel steps only), and the outer loop ships trend memory: run history, deltas and sparklines, chronic-scenario detection, and the divergence/failure-theme evidence streams that gate M3 and feed the action vocabulary. The Gherkin data vocabulary is complete (M2): key/value and record tables, doc strings, Examples params, all parameterized, all edit-free at replay, all honestly stale on structural change. Step sets shipped with the .saffron dialect (M9): flows written once, invoked anywhere, expanded at parse time. The agent runs on any of four providers (M5): Claude, Codex, Google AI through Antigravity, or Cursor, each through the vendor's own client on the machine's login, or on an API key for a CI machine with no login (ANTHROPIC_API_KEY, CODEX_API_KEY, GEMINI_API_KEY, CURSOR_API_KEY). Codex and Antigravity are verified live, Cursor not yet. A heal made in CI is reviewed on your machine (M20): saffron export packs it, saffron import checks it against your checkout, the CI runs views of both IDEs import a run with a click, and CI can open a draft pull request with the heals instead. saffron dashboard shows the suite's quality from its run history.

Shipped	Date
MVP: record → zero-token replay → runtime heal → reviewed accept	Jul 2
Locator heal propagation (`accept --propagate`)	Jul 3
`--strict`, npm packaging, docs site	Jul 3–4
Dynamic-content vocabulary (capture/differs/matches/attribute, `{date±N}`, tabs)	Jul 3
Verified proposals (proof replay + refine pass)	Jul 4
Cost composition capture (cache read/write tokens)	Jul 4
Inner-loop optimization (snapshot-mode none, batching)	Jul 4
Level-1 step reuse: index, islands, quoted-value, data tables	Jul 5
M1. run history, trend deltas & sparklines, chronic-scenario detection, failure themes	Jul 5
M4. assertion policy v2: opt-in adaptable-mid, final assertion block always strict	Jul 5
M2. record tables (`<table:row:col>`), doc strings (`<docstring>`), cross-step params	Jul 5
M9. step sets & the `.saffron` dialect (`StepSet:` define / `StepSet` invoke, parse-time expansion)	Jul 6
M10. IDE integration: `saffron steps`/`--snippets`, `saffron mcp`, VS Code extension (sibling repo)	Jul 6
M8. duplicate-wording rename proposals (behavior-proven) + `saffron author` + `saffron lsp` (JetBrains/Neovim)	Jul 6
M5 core: cross-browser replay (firefox/webkit), parallel workers, `healModel`	Jul 7
M6. PUBLIC RELEASE: `saffron-ai@0.1.0` on npm, free for any use, closed source	Jul 7
M12 Tier 1: `{env:VAR}` secrets: masked caches/reports/history, preflight, `.env`	Jul 7
M2 leftover: dialogs, file uploads, drag-and-drop, iframes (replay on all 3 engines)	Jul 7
Recording ergonomics: live dialog/drag/iframe recording (passive-listener fix, frame-fallback asserts)	Jul 19
Network vocabulary (waitForResponse / expectResponse): wait on or assert backend API calls (URL/method/status/body matchers)	Jul 19
Agent skill: `skills/saffron/SKILL.md` ships in the package; `saffron init` installs it, registers `saffron mcp` per host, adds the AGENTS.md/CLAUDE.md block	Sep 13
Multi-tag `--filter` (OR) + scenario tags in report JSON / history for test-management keying	Sep 13
JetBrains plugin (saffron-jetbrains-plugin): `.saffron` file type + bundled grammar + LSP4IJ auto-wiring; on the JetBrains Marketplace	Sep 13
Wrong-recording recovery: `saffron run --rerecord`, recorder keeps typed values (`pressSequentially`), empty-fill guard stamps UNVERIFIED, `accept --all` skips UNVERIFIED	Sep 13
Retries visible: per-step `retried ×N` chip and terminal note (retries stay per action)	Sep 13
Failure screenshots in the report and the IDE panels; `saffron prune`; `saffron diff`; configuration reference (0.7.x)	Sep 19–21
M19. external test data: `data/` folder, `{data:...}`, list checks, Examples from a file, `{unique:...}`, data per environment (0.8.0)	Sep 21
Google Chrome and Microsoft Edge (`--browser chrome\|msedge`), recording included; `setup` and `teardown` commands around a run (0.8.1)	Sep 21
Sharding across CI machines (`--shard 2/4`) with `saffron report --merge`: one report and one history line, equal to an unsharded run	Sep 22
Bound proposals: acceptance refused when the scenario or the committed recording moved since filing; caches and proposals validated on read	Sep 24
M17 core: execution replay: `trace` config, Gherkin timeline in the trace, `saffron trace` and the Saffron replay page, report and IDE links, secrets masked	Sep 24
M17 follow-up: heal review with accept and reject in the replay, proof replay sessions, the agent's own browser calls in the trace	Sep 24–26
M13: guided first run, `saffron init --examples` installs the Saucedemo suite with config and .env.example	Sep 13
Tag expressions in `--filter`: `and`, `or`, `not`, parentheses (0.9.1); the list forms keep meaning `or`	Sep 29
M5. multi-provider agents: `provider` = `claude`, `codex`, `antigravity` (Google AI), `cursor`, each on the vendor's own client and the machine's subscription login; one recorder behind a loopback tool gateway; `saffron login <provider>` (0.9.2)	Sep 30
Security hardening of the agent's browser tools: allowlist of recordable tools, call-by-call guard (web URLs only, uploads inside the project and never dotfiles, no `filename` writes, private output folder), result cap; `strictMcpConfig` for Claude (0.9.2)	Sep 30
Nested projects: the language server and both IDE plugins resolve a `.saffron` file against the nearest `saffron.config.json`; JetBrains trusted-project gate and root chooser; VS Code runs without a `.cmd` shim on Windows (editors 0.3.3)	Sep 30
M20 step 1: `saffron export` packs a CI run's proposals, report, screenshots and traces into one bundle; `saffron import` checks it against the reviewer's checkout (repository, format, digests, each proposal's binding) and files it there; shard bundles merge (0.9.3)	Oct 1
M20 step 2: `saffron export --branch` commits the proposals onto the tested commit as a new branch and opens a draft pull request with each proposal's verification and diff (GitHub, through `gh`) (0.9.3)	Oct 1
M20 step 3: a CI runs view in VS Code: the branch's GitHub Actions runs through `gh`, Import downloads a run's bundle and runs `saffron import`, each proposal's outcome under the run, Proposals marked with the run they came from (`importedFrom` in `status --json`) (0.9.3; VS Code 0.4.0)	Oct 1
M20 step 4: one CLI contract for both CI services and both editors: `saffron runs` and `saffron import --run` over `gh` (GitHub Actions) and `az` (Azure Pipelines), the JetBrains CI Runs tab, Azure DevOps draft pull requests (0.9.3; VS Code 0.4.0, JetBrains 0.4.0)	Oct 1
`saffron dashboard`: the suite's quality from its run history (failing and for how long, flaky, healed again and again, never run, slowest, trends, per tag), the same numbers as `quality` in `status --json`; the editors' Dashboard shows it (0.9.4; VS Code 0.5.0, JetBrains 0.5.0)	Oct 4
M20 follow-ups: a recording uses only the env vars its scenario names and navigates only to http(s); proposals from CI or a heal branch are never replayed with your secrets; heal once; `import --run` refuses fork runs; `accept --all` skips imported and unchecked proposals; outline rows stay on one shard; report and replay page escape imported data (0.9.5)	Oct 4
M22 v1: a CI run's results viewed without importing: `saffron report --run` and `saffron trace --run` (view only) from a checked download in `.saffron/ci/`, each proposal judged importable or not, a fixed cache cap; results under each run in both editors' CI runs views (0.9.7; VS Code 0.6.0, JetBrains 0.6.0)	Oct 5
One scenario at a time: `saffron run features/login.saffron:12` runs the scenario a line is in (an Examples row runs that row), `:Successful login` the scenario of that name, and Run, Replay only and Headed lenses above each scenario, from `saffron lsp` in JetBrains IDEs with LSP4IJ and drawn by the VS Code extension, plus right-click Run inside a scenario in JetBrains (0.9.8)	Oct 6
Browser matrix in one run (`--browser chromium,firefox,webkit`, one report, a row per browser), device and viewport profiles (`--device`, `--viewport`), parallel recording and healing (`--agent-workers N`, a browser per session), per-worker timing in the report, and Saffron's own data in the replay: tabs, the responses network checks matched, the agent's cost	Oct 6
M15. Several people in one scenario: `As "buyer" ...` runs a step in a browser session of its own (a context with its own cookies, storage and tabs), carried to the steps after it; one agent records and heals across the sessions; storage state per session; a trace per session	Oct 9
M18. Configurable test folder: the `features` key is followed everywhere, `saffron init` and `init --examples` included; `status --json` names it as `featuresDir`	Oct 9
M22 v2: CI runs say what they left before you open them (GitHub), `saffron export --report-only` summaries for every run, and a results panel with screenshots in both editors (1.0.0; VS Code 0.7.0, JetBrains 0.7.0)	Oct 9
1.1.0: agent-blind secrets (SAF-13): the recording agent is given and types only `{env:...}` tokens, values go into the browser on the way and come back masked, for every provider; `saffron eject` exports recordings as Playwright Test specs, never looser than replay (SAF-35); JSON response field checks by pointer, types kept	Oct 22
1.2.0: API test automation (beta): `@api` features test an HTTP API with no browser, recorded once and replayed at zero tokens; status, field, header, timing and pinned OpenAPI contract checks; fixtures and named operations	Oct 26
1.2.1: sturdier recordings, and Codex and Antigravity on an API key for CI machines with no login	Oct 28
1.2.2: proposals reviewed in the run report, a new quality dashboard, and the MCP `verify` tool for coding agents	Oct 28
1.3.0: recordings checked by machine (every Outline row proved, a moved-clock check, writes the page cut off, a check that changed the page held), one review every view agrees on, captured values typed by later steps, data tables for API requests	Oct 30

The road ahead, at a glance

Saffron roadmap: shipped work and the milestones ahead

The diagram groups shipped work by what it lets you do; the milestone numbers below are the build order it came from. The vocabulary track (M9 step sets fed M8 dedupe, which fed M10 IDE support) is shipped, and so is the public release (M6): saffron-ai is on npm. The runtime track (M5) shipped its core and the multi-provider agents, M3 stays a gated decision rather than a scheduled milestone, and the SaaS layer (M7) is the long-term horizon. M22, viewing a CI run's results without importing them, shipped in 0.9.7; its v2 (result markers on each run, a slim summary artifact, a richer results panel) shipped in 1.0.0. Full scope and rationale for each milestone below.

M1. Outer loop: run history & trends

Status: SHIPPED 2026-07-05

.saffron/history.jsonl (one trimmed line per run, commit recommended); report deltas vs previous run + 20-run pass-rate/cost sparklines; chronic detection (2+ heals or 3+ reds in 10 runs, warned in CLI and report); per-run divergence recorded (the M3 evidence stream); recurring failure themes aggregated from red-run errors (the M2 data source).

M2. Step & data vocabulary completion

Status: SHIPPED 2026-07-05 (record tables + doc strings)

Multi-row record tables (N columns + header row, a list of records): per-cell parameterization: recorded values become <table:row:col> references resolved from the scenario's own table at replay. Cell-value edits replay at zero tokens; row-count or header changes alter the key set and are honestly stale (no fragile per-row iteration detection, the agent records the actual actions, the cells parameterize).
Doc strings (""" blocks): parsed, rendered to the agent, and recorded as <docstring> references: content edits replay free; literal recordings stay protected by exact-content staleness.
Cross-step references: unambiguous table/docstring params are scenario-wide (like Examples params), so a later assertion on the note's text follows content edits too. Validated live on the demo guestbook: record $0.53, then full value+content rewrite replayed GREEN at 0 tokens, row addition went honestly stale.
Leftover SHIPPED 2026-07-07: the four known action-vocabulary candidates: handleDialog (armed before its trigger), uploadFile (input or armed file chooser, portable relative paths), dragTo (two semantic targets), and iframe-scoped targets (target.frame). Replay proven by 12 real-browser unit tests plus the four widget scenarios in the example suite, green on all three engines. Recording ergonomics fixed 2026-07-19: the dialog auto-dismiss pathology was Saffron's own bug, not the MCP's, the orchestrator's launching Playwright client dismisses any dialog on a page with no local listener, destroying it before the agent's CDP client could present it. Agent sessions now run with passive dialog listeners armed (withDialogsVisible), and all four capabilities record live: dialog $0.66 (was $5.33 under the workaround), drag $0.18, iframe $0.24, upload $0.30, every proposal proof-verified. Frame-less assertions also gained a replay fallback that searches child frames, so a recording that misses an iframe boundary still verifies.

M3. Level 2: shared step library

Status: gated (decision, not schedule)

Gate: Level-1 divergence data. The step index already reports how often identical step text carries different recordings. If real suites show divergence is rare (the user's Cucumber experience predicts this, authors parameterize instead of forking), Level 2 is safe; if it's common, Level 1

propagation remains the right architecture and this milestone is dropped, not postponed.

Scope if greenlit:

Shared step recordings as the source of truth; scenario caches hold references.
Heal ping-pong detection: a shared step healed in opposite directions within N runs is flagged polymorphic → split the step text or fork it.
Per-scenario overrides (shadowing) as the escape hatch.
Affected-scenario re-verification: updating a shared step queues zero-AI proof replays for every referencing scenario.
saffron show <scenario>: flatten references into the effective action list for review.

M4. Assertion policy v2

Status: SHIPPED 2026-07-05

assertionPolicy: "strict" | "adaptable-mid" (config or --assertion-policy, default strict). Under adaptable-mid, mid-scenario checkpoint assertions may be adapted by the agent, with an adaptation note, suggested .feature edit, proof replay, and the accept gate, while the final assertion block (the trailing run of assertion steps, the scenario's verdict) stays strict under every policy: never merged from agent recordings, always re-verified as written, never refined around. Live-validated both ways on a broken mid checkpoint: strict → red, adaptable-mid → yellow with the checkpoint adapted and verified.

M5. Runtime & providers

Status: SHIPPED 2026-07-07 (core) and 2026-09-30 (multi-provider) · LLM-less replay demand-driven

Shipped:

Cross-browser replay: --browser firefox|webkit (config browser). Replay runs on Playwright's browser engine, so the whole committed suite runs green on all three engines (validated on the example suite). Recording/healing stay Chromium (the agent attaches over CDP): under another browser, a scenario needing the agent fails honestly with "record with --browser chromium, then replay cross-browser" guidance.
Parallel workers: --workers N (config workers). Zero-AI replays run concurrently, one browser context per worker; anything needing the agent is re-run sequentially with full agent support, so healing keeps its exact-browser-state semantics (cost: one extra replay for those scenarios). ~2× on the tiny example suite; scales with suite size.
healModel: --heal-model (config healModel), a cheaper model for heal sessions and their refine passes; recordings keep the main model. Plumbed per-run through the provider (ctx.model).
Multi-provider agents (SHIPPED 2026-09-30, 0.9.2): provider in saffron.config.json or --provider: claude (default), codex (ChatGPT Plus/Pro through the Codex SDK), antigravity (Google AI Pro/Ultra through Google's agy CLI, experimental) and cursor (Cursor SDK). Every vendor runs on its own official client and the machine's existing subscription login; Claude also takes ANTHROPIC_API_KEY and Cursor CURSOR_API_KEY. The non-Claude agents reach the browser and Saffron's tools through a loopback MCP gateway that runs the same recorder hooks as the Claude path, so recordings are identical whatever the provider. saffron login <provider> checks and completes the setup. Codex and Antigravity verified live; Cursor not yet.

Remaining (picked up when demand appears):

LLM-less MCP replay: an alternative replay engine that drives the Playwright MCP server programmatically (no AI) for environments that standardize on MCP.

M6. Public release (free, closed-source)

Status: SHIPPED 2026-07-07, saffron-ai@0.1.0 on npm (reshaped from the original open-source plan, see the design log's "Distribution" entry). Published as saffron-ai, one brand across npm, the site, and the public repo; verified from the public registry on a clean install. The bare saffron npm name is squatted by a deprecated package: a name dispute remains an option later.

Free to use, including commercially, under the Saffron Free Use License v1.0 (replaces Apache-2.0: relicensed cleanly while nothing had been distributed). Users configure ANTHROPIC_API_KEY or a Claude Code login; replay-only CI needs no credentials.
Closed-source packaging done: single bundled+minified CLI (esbuild, no sourcemaps), npm tarball ships dist-pkg + README + LICENSE only; standalone install smoke-tested (--version, steps, run --no-agent all green from the tarball).
First-run preflight done: missing-credentials warning with setup instructions instead of an SDK stack trace.
Feedback channel: GitHub issues on the public site repository (saffron-ai), linked from the site.
Published 2026-07-07; the site quickstart (npm i -D saffron-ai) is now literally true.

M7. Open-core SaaS layer

Status: long-term · after public release traction

Hosted layer on top of the free CLI: team dashboard, cache-review UI with visual diffs (where a Level-2 registry would also earn its place), scheduled cloud runs, run-history analytics across projects. The CLI must never require it.

M8. Step vocabulary & authoring

Status: SHIPPED 2026-07-06 (reader-friendly guide with diagrams: Step Sets & Vocabulary)

Shipped: duplicate-wording rename proposals, after any successful recording, a step whose action list is IDENTICAL (actionsEqual) to an indexed step with different text gets a rename feature-edit appended to the proposal (adaptation note explains; accept --with-feature-edit converges the wording; empty no-op action lists never match; divergent canonicals are skipped). saffron author <prose-file>, drafts a .saffron file from plain paragraphs with the vocabulary injected as required wording; live-validated: authored file reused the vocabulary verbatim, invoked the existing StepSet, and would record almost fully seeded ($0.12 to author). Original scope notes below.

Kills the Cucumber pain of the same step existing under two wordings ("I navigate to the home page" vs "I visit the home page"). In Saffron the pain has a price tag: exact step text is the cache identity, so a second wording is an index miss that re-records at token cost. The step index derived from committed caches is the registry; this milestone surfaces it (no second, hand-maintained registry, those rot):

saffron steps: SHIPPED early with M10 Phase A; list/search the vocabulary (files ∪ caches) with status badges, usage counts, --json export and --snippets; --json export for tooling.
Duplicate detection with rename proposals (mechanical, zero AI): after a recording, a new step whose actions are identical (actionsEqual) to an indexed step with different text is behavior-proven duplicate wording: Saffron files a .feature rename-edit proposal to the canonical wording via the existing feature-edit machinery. Each duplicate costs one recording, then the vocabulary converges. (Cucumber structurally cannot do this, it never observes that two wordings produce identical behavior.)
saffron author <prose-file>: the agent drafts a .feature from plain-paragraph requirements, with the step index injected as the required vocabulary (prefer existing wordings exactly). Synergy: authored files are maximally seedable, so their first recording is mostly zero-AI islands. Interaction: duplicate detection sharpens the Level-1 divergence data that gates M3. IDE-side prevention (completion + near-duplicate diagnostics while typing) is M10, built on this milestone's index plumbing.

M9. Step sets & the `.saffron` dialect

Status: SHIPPED 2026-07-06 (reader-friendly guide with diagrams: Step Sets & Vocabulary; shipped behavior also in Features)

Sets are project-wide (registry scans every .saffron file; library-file convention features/shared.steps.saffron for application-wide flows). Live-validated on the demo guestbook: two .saffron scenarios sharing a step set recorded with 4/5 steps seeded ($0.61 total incl. verification), replayed at 0 tokens; a table-value edit inside the set stayed green; a reworded set step made BOTH invoking scenarios honestly stale; heal feature-edits route to the set definition line.

Kills the repeated-step-group pain (the same 5–6 checkout-prelude steps pasted into every scenario of a file). A step set is a named, reusable sequence of steps, defined with a first-class StepSet: keyword in a .saffron file: a superset dialect of Gherkin (the TypeScript playbook: .ts = .js + keywords, honest new extension, tooling maps back cleanly):

Feature: Checkout Flow

StepSet: Complete guest information
    Given I am on guest information page
    When I enter guest name "Chathuranga Jayasinghe"
    And I click continue button
    Then I am not on guest information page

Scenario: Verify checkout happy path
    Given I am on calendar page
    ...
    And I click choose button
    StepSet Complete guest information
    Then I am on checkout page
    ...

Syntax (locked 2026-07-06): definition is StepSet: <name> (block keyword, with colon, like Scenario:); invocation is StepSet <name> (step position, no colon, like Given). One spelling, symmetric. Sets are a .saffron-exclusive feature: plain .feature files stay fully supported for everything else, and adopting sets in an existing file is a pure rename to .saffron with zero content edits (superset, the TypeScript playbook).

The set shape convention (from the worked example): first step is a guard ("Given I am on guest information page"), last step is an exit assertion ("Then I am not on guest information page"), every invocation carries its own precondition and success check.
Assertions inside sets get position-dependent strictness, for free: after expansion the set's Thens are ordinary assertions of the calling scenario: strict by default, adaptable only under adaptable-mid, and when a set is invoked as the last thing in a scenario its trailing Thens join the final assertion block (strict under every policy, forever). No new rules; expansion does it.
Parse-time expansion, exactly like Background merging: sub-steps keep their exact text identity, so caching, Level-1 seeding, heal propagation, and staleness all work with zero engine changes. Editing a set makes its scenarios honestly stale, but re-recording is almost fully seeded, so convergence is cheap.
Implementation: a pre-processor rewrites StepSet: lines to Scenario: same-line (line numbers preserved 1:1) and feeds the standard @cucumber/gherkin parser, no fork. The real cost is the edit-back path: featureEdits/heal proposals need step provenance ({file, line, fromSet}) because an expanded step's source line lives in the set definition, possibly in another file, and editing it fixes every caller at once (they go stale, re-record seeded).
Guard rails: set names are project-wide unique (same-file definitions resolved first), which also catches the colon-typo at an invocation site (an accidental StepSet: block duplicates an existing name → parse error with a "remove the colon to invoke" hint); invoking an unknown set is a parse error; an And directly after an invocation inherits kind from the set's last step: docs recommend an explicit keyword on the step after an invocation.
v1 rules: sets may contain tables, doc strings, and <param> placeholders (resolved from the calling scenario's Examples row); no nested sets; sets never run standalone but their steps join the seeding vocabulary; reports group expanded steps under the invocation with a "from StepSet" badge.
Docs ship the one-line VS Code files.associations and .gitattributes snippets that give .saffron highlighting and GitHub rendering on day one.

M10. IDE integration: `.saffron` language support & step completion

Status: SHIPPED 2026-07-06 · marketplace publish rides with M6 · reader-friendly guide: Step Sets & Vocabulary

Shipped surface: saffron steps [search] [--json] [--snippets] (the vocabulary command: Phase A also delivered the core of M8's saffron steps), saffron mcp (stdio server: search_steps, list_step_sets), saffron lsp (the same experience for editors beyond VS Code: JetBrains via the free LSP4IJ plugin on Community editions or the built-in LSP API on Ultimate, Neovim via lspconfig; completion with badges, StepSet go-to-definition, hover, diagnostics, served by the real parser), and the Saffron VS Code extension living in the sibling repository ../saffron-vscode (zero-dependency: .saffron grammar, badge-carrying completion, StepSet completion, go-to-definition, hover, unknown/duplicate-set and near-duplicate wording diagnostics). The extension is published on the VS Code Marketplace (ChathurangaJayasinghe.saffron-vscode, 2026-08-15).

While typing a .saffron (or .feature) file, the editor suggests the project's existing steps, so duplicate wording is prevented at the keyboard, before it ever costs a recording.

Two framing decisions:

The completion vocabulary is the union of parsed files + the cache index, not caches alone: files are the source of wording (a teammate's unrecorded steps must complete too); caches add status. Every suggestion carries a badge no Cucumber tool can offer, ✓ recorded (replays at zero tokens), ⚠ divergent, ○ unrecorded, authors literally see which steps are already paid for.
Completion alone is half the cure: the author who confidently types a new wording never opens the popup. The layer that kills the duplication pain is a near-duplicate diagnostic (squiggle: "similar to 'I navigate to the home page', recorded, used in 4 scenarios" + rename quick-fix). IDE prevention before recording; M8's behavior-proven detection after: same pain attacked from both sides.

Phased by cost:

Phase A: zero-extension stopgap (can ship with M9): saffron steps --json (machine-readable vocabulary export) and saffron steps --snippets writing .vscode/saffron.code-snippets, VS Code completes snippets natively, regenerate after each run. Crude matching, no badges, works immediately.
Phase B: AI authoring surface: saffron mcp, a small MCP server exposing search-steps / list-step-sets over the live index. In 2026 "someone manually writing a file" is often a Copilot/Claude session, assistants can't read completion popups, they need the vocabulary as context. Same vocabulary humans get, served to machines (and the interactive twin of saffron author).
Phase C: the real language tooling: a Language Server (LSP, so JetBrains/Neovim reuse it) + thin VS Code client: TextMate grammar with first-class StepSet: highlighting; step completion with the status badges; StepSet-name completion after StepSet ; go-to-definition on invocations (what makes .saffron feel like a real language); hover showing a step's recorded actions, usage count, and last-recorded date; diagnostics, unknown StepSet name, duplicate set definitions, near-duplicate step wording.

Ruled out: piggybacking the official Cucumber VS Code extension, it completes from step definitions (glue-code regexes), which Saffron deliberately doesn't have; it has nothing to index here.

M13: Guided first run from `saffron init`

Status: SHIPPED 2026-09-13

saffron init offers to install a small example suite so a new user's first saffron run has something real to record and replay:

Prompt: "Install example tests? (y/N)". Non-interactive runs (CI, --yes/--no-examples) never block.
The examples target the public https://www.saucedemo.com demo shop (login, cart, checkout), written the way the runner rewards and in .saffron files only: short scenarios, {env:VAR} for the credentials, one Scenario Outline, StepSets in a library file invoked at the start of scenarios and in the middle, tags for --filter.
saffron.config.json is written with baseURL https://www.saucedemo.com, retries: 1, strict: false, and a .env.example documenting SAUCE_PASSWORD.
The examples ship inside the npm package next to the skill (examples/saucedemo/) so they stay versioned with the runner; init copies them into features/ only when the folder is empty.
Docs: a "Your first run" page that walks the user through record, review, accept and the zero-token replay using exactly these files.

M14. Run from the IDE

Status: SHIPPED 2026-09-16 (JetBrains plugin 0.2.0, VS Code extension 0.2.0; side panel in 0.2.1); its follow-up, one scenario at a time, shipped 2026-10-06 in saffron-ai 0.9.8.

Shipped: a Saffron run configuration type in JetBrains IDEs (run with files, folders, tags, replay-only, headed, re-record and extra arguments; report; accept), right-click → Run on .saffron files and folders in both IDE families, the Saffron tool window in JetBrains IDEs (feature files with scenario counts, search, Run Selected / Run All, Open Report, Accept All Proposals, last-run totals, pending proposals), and the matching Saffron side panel in VS Code (feature files with scenarios as children, checkboxes and Run selected, replay-only and headed toggles, last-run totals, pending proposals). Both panels grew into dashboards on top of saffron status --json (2026-09-17): a Proposals list to tick and accept or reject with the narrative and proof-replay verdict, Tags to tick and run, vocabulary health (divergent steps, duplicate wordings proven by identical recordings) with the effective config, and the run report embedded in the IDE (JCEF tab in JetBrains, webview editor tab in VS Code) instead of re-drawn charts. VS Code runs commands in a dedicated terminal from the workspace folder; JetBrains runs them through the Run tool window and keeps the configuration in the run widget.

Follow-up, one scenario at a time (shipped 2026-10-06 in saffron-ai 0.9.8, VS Code extension and JetBrains plugin 0.6.1):

Single-scenario runs in the CLI: saffron run features/login.saffron:12 runs the scenario that line is in, Cucumber-style; an Examples row runs that row alone, and a line no scenario owns stops the run, saying where the scenarios start. features/login.saffron:Successful login names the scenario instead, which an edit above it does not change; the editors use that form. A run of named scenarios folds into the last report, and its history line is marked partial: it counts for its scenarios, not as a suite run.
Run lenses in the editor: the JetBrains plugin colours .saffron files through a TextMate grammar, so there is no PSI to hang a gutter marker on. The language server publishes Run, Replay only and Headed lenses above each Scenario line instead; LSP4IJ renders them and the plugin runs each through a run configuration made for the scenario. The lenses are opt-in (initializationOptions.runLenses), so a client that cannot run them shows none. Right-click Run with the caret in a scenario runs it too, without LSP4IJ. The VS Code extension draws the same lenses, in the file's Gherkin language, and runs them as tasks.

"Everything except" shipped as --filter "not @wip" (tag expressions, 0.9.1), so no separate --exclude flag is planned.

M15. Multi-tab and multi-session scenarios

Multi-tab: SHIPPED 2026-09-18. Plain prose ("switch back to the first tab", "close the preview tab", "open a new tab at the admin page"); the agent's tab operations are cached as switchTab, closeTab and newTab (0-based, creation order) and replay in order. Clicks that open tabs are still followed automatically; an explicit switch is not undone by the follow.

Multi-session (two browsers in one scenario): SHIPPED 2026-10-09. "User A sends a message, user B sees it": a step starting As "buyer" runs in a browser context named "buyer", and the session carries over to the steps after it. One agent records, heals and refines across every session (a browser backend per session, each call sent to the session of the step it is on); replay opens one context per session, each with its own storage state (sessions.<name>.storageState); each session is traced on its own, and the report tags where each takes over. The cache needed no session field: a step's session follows from the step texts, so a change to it is a changed text and the recording goes stale as for any edit. Cross-domain scenarios need nothing: absolute URLs outside baseURL are cached as written.

M19. External test data

Status: SHIPPED in 0.8.0 (2026-09-21). {data:file.key} reads a value from JSON files under data/, stored in recordings as a token and resolved at replay. One assertion checks every value of a list (the enum case), following the file at replay. Also shipped: Examples rows read from a file in .saffron files, values that are unique per run, and data per environment.

M16. Failure evidence: screenshots and video

Status: screenshots and IDE links SHIPPED (0.7.2, with the panels in VS Code 0.2.6 and JetBrains 0.2.3); video planned. A red scenario shows its error text today and nothing of what the page looked like. Planned: a screenshot of the active page at the moment an action or assertion fails, embedded in the report; optional video per scenario (video: off | on | retain-on-failure); a predictable, git-ignored .saffron/artifacts/ layout that saffron prune keeps small; and artifact paths in latest.json so both IDE panels can open them. Pixels can show secrets that text masking cannot reach, and the docs will say so.

M17. Execution replay: time-travel debugging

Status: core SHIPPED 2026-09-24. trace: off | on | retain-on-failure, Gherkin steps and recorded actions as the trace's timeline, saffron trace and the Saffron replay page (steps, live DOM at every action, filmstrip, details, network, Saffron panel), links from the report and both IDE panels, {env:VAR} masked in trace text.

Follow-up implemented 2026-09-24 to 2026-09-26: heal diffs beside the selected step, highlights of recorded target matches, accept/reject with exact proposal identity and binding checks, and agent browser calls on the runner connection. Proof replays retain separate traces and outcomes. The viewer and MCP trace bridge resolve the runner's own Playwright package, including when MCP installs another version.

M18. Configurable test-source directory

Status: SHIPPED 2026-10-09. features/ stays the documented convention, but the folder name is a preference, not a rule: the features key in saffron.config.json (default "features") points the runner at scenarios/, tests/ or e2e/. Explicit CLI paths keep winning; the key decides where no-argument runs and the tools look.

Everything that assumes the convention follows the key: saffron run, status, steps, author, prune, the StepSet registry, the language server, the MCP server and the draft pull request.
saffron init creates the configured folder, and init --examples installs the example suite into it (and its data into dataDir); a folder outside the project is refused.
saffron status --json reports the effective folder as featuresDir. The VS Code and JetBrains panels scan the whole project, so they need no setting.
Docs: the configuration reference names the key and its default; examples keep features/.

M20. CI proposals: review and accept from the IDE

Status: SHIPPED 2026-10-01 in runner 0.9.3 (on npm 2026-10-03), the VS Code extension 0.4.0 and the JetBrains plugin 0.4.0. Azure Pipelines was tested against a stand-in az only, not yet in a real Azure DevOps organization, and Windows az.cmd quoting only by unit tests: please report problems.

A heal in CI used to file its proposal in a workspace that was thrown away, so the repair had to be reproduced locally with a second agent session. M20 made the proposal travel. saffron export in CI writes a bundle (the pending proposals, the run's report, screenshots and traces, and where the run happened: repository, branch, commit, CI run and attempt, shard), and CI uploads it as an artifact. saffron import on a reviewer's checkout refuses a bundle from another repository, of a newer format or with a damaged file, checks each proposal's binding against the files there, and files the rest where a local run would have, so saffron diff, saffron accept, saffron trace, the report and both IDE panels work on it unchanged. Shard bundles merge into one report. After accept the new recordings are an ordinary diff to commit and push with your own git tools; nothing is pushed for you.

saffron runs and saffron import --run <id> list a branch's CI runs and import one, through gh for GitHub Actions and az for Azure Pipelines, and both editors call only these: the CI runs view in VS Code and the CI Runs tab in JetBrains import a run with a click and mark each imported proposal with the run it came from. For teams that review in pull requests, saffron export --branch <name> pushes the proposal files (not the recordings) as a branch on the tested commit and opens a draft pull request on GitHub or Azure DevOps whose description is each proposal's verification and diff, with the commands to accept.

How to set it up and use it: Review CI heals, CI runs in your IDE and Heal pull requests.

M21. API testing: reusable operations, fixtures and schema validation

Status: v1 implemented 2026-10-21 (unreleased): the HTTP medium, @api scenarios, requests with typed bodies and extracted values, status, field and schema checks against a pinned OpenAPI 3 contract (saffron api pin), contract freshness, writes never retried, API calls from browser scenarios. 2026-10-24: polling (until), header capture, form and multipart bodies, list and shape checks, header and duration checks, per-request timeouts, YAML contracts, format validation, captured credentials masked and origin-bound, Fixture: blocks run once per feature, saffron eject of API steps (a scenario reading a fixture stays fixme). Named operations are StepSets. 2026-10-26, after two evaluations: captures an agent segment keeps reach the reused steps after it, a quantity is no status, the checks a step makes show in the trace and the report, polls are bounded, a stale recording says what changed. 2026-10-26: eject exports schema checks, the contract's schemas resolved into the spec and validated with ajv. Left: browser fixtures with inherited cookies, eject of a scenario reading a fixture. Extend the Gherkin → reviewed execution plan → zero-AI replay workflow to standalone REST/JSON tests. Use reusable resource clients expressed as named declarative operations, JSON request templates, explicit response extraction, scenario context and feature fixtures. Reference pinned OpenAPI contracts or standalone JSON Schemas for deterministic validation, alongside explicit business assertions. Start with the customer → product → subscription flow, including three- and twelve-month contracts and reuse through fixtures. API contract failures remain failures: AI assists authoring and diagnosis; changed endpoints, expected statuses, schemas or assertions require reviewed updates. Proof replay and retries must respect server-side side effects. Deliver the HTTP executor and operation model first, then fixtures, schema validation, and AI/IDE integration.

M22. CI run results: view without importing

Status: v1 SHIPPED 2026-10-05 in 0.9.7 (VS Code 0.6.0, JetBrains 0.6.0); v2 (a) to (c) SHIPPED 2026-10-09 (1.0.0, VS Code 0.7.0, JetBrains 0.7.0); (d) waits for a user to ask.

Clicking a CI run in the IDE should show what it found. Until now the only action was Import, which files the run's proposals, copies its evidence into your checkout and replaces your latest report. v1 adds a read-only view:

saffron report --run <id> downloads a run's bundles once into .saffron/ci/<provider>-<id>/attempt-<n>/, runs every check import makes, judges each proposal against this checkout (importable, or not and why) without filing it, and opens the run's report page. --json gives the run, totals, the scenarios with their failing step, error, screenshot and trace, the proposals, and warnings. It never writes .saffron/reports/, .saffron/proposals/, .saffron/artifacts/ or the import ledger.
A run without results says why: still running, expired, or no bundle uploaded (with the advice to export and upload under if: always()). --max-download <MB> refuses a large run by its size, so an editor can ask first.
saffron trace --run <id> <scenario> replays the run's traces from that folder, view only: no Accept or Reject, because nothing was filed.
saffron import --run <id> reuses the download, so viewing then importing downloads once.
The cache keeps the newest 10 runs and at most 1 GB, evicting the oldest first, and ignores itself in git with its own .gitignore.
In both IDEs a run's row expands into its results: totals, the red and yellow scenarios, the proposals and why any would not import, Open Report and Open Trace. Import is unchanged and stays the only action that files proposals.

v2: (a) to (c) shipped: on GitHub, saffron runs marks every run from one listing of the repository's artifacts (results and size, summary only, expired, none, still running), and both editors' rows say it; saffron export --report-only writes a saffron-summary that viewing uses for a shard or attempt with no full bundle, and that alone is never imported; and viewing a run writes it as one page (panel.html; report --run --panel prints it) (cards with screenshots, failures first, the proposals), shown in a VS Code editor tab and below the runs in JetBrains, its buttons opening traces, scenarios, the report and Import. Azure rows are marked when opened. (d) is not built: nobody has asked for the cache limits as settings.

As planned:

(a) Per-run result markers in saffron runs and the IDE rows (results, none or expired, and the size), from one GitHub artifacts call joined to the runs by workflow_run.id; on Azure, asked lazily per run.
(b) A slim summary artifact (saffron export --report-only, uploaded with if: always()), so green runs and quick looks need no full bundle.
(c) A richer results panel (a webview in VS Code, a panel in JetBrains) with screenshot thumbnails.
(d) Settings for the cache's limits, only if users ask for them.

Known issues carried past 1.0.0

Next, in 1.0.x. The 1.0.0 review rounds left a few issues open because each was narrow, failed loudly rather than passing, or behaved the same or worse in 0.9.9. False greens come first:

Outline rows: remaining false greens (SAF-127), implemented for the next 1.0.x release: short values in actions, dates without a year or glued to digits, blank cells, literal element roles and ambiguous whole env/data tokens refuse other rows when the recording cannot follow them. Step reuse refuses changed arguments containing several placeholders. Date tokens inside environment values stay literal when comparing rows. Regression coverage includes glued data/unique tokens and repeated browser replay.
Outline rows that fail or record more than they should: a row value that coincidentally sits inside a step's own value ties the recording to its row; a few shapes (slugs, ISO dates for custom-format cells, lists read from data) record again on every run.
Replay: line breaks drawn by <br> or block elements, and speed on very large containers.
Assertion strictness (SAF-132, fixed in 1.0.1): response checks use requests started since the latest browser action. Attribute patterns preserve case, assertion role names match exactly, and input-value checks require visibility. Partly blank text expectations and omitted expected values are refused. expectDiffers already checks nonempty displayed text. Compatibility changes are documented in HOW-TO-USE.
CLI and files: status and diff --json details; a scenario with no steps passes green.
Secrets: the fingerprint a row-bound recording keeps of an {env:...} cell lets a short secret be recovered by trying every value.
Tools: status and diff wording for row-bound recordings, the trace viewer's session view, the size of the CI results page, and a few language-server and editor details where they still differ from the runner.

M23. Native dynamic assertions

Status: SHIPPED in 1.0.1. Localized dates and ranges are tokens ({date+7:D. MMM:de-DE}, {date+7 - +9:MMM D, YYYY}), resolved on one run clock in the configured timezone, and recorded as format version 3. Amounts that add up are a native check (expectAmount: "before" + "breakfast"), and so is what a downloaded file says (expectDownload). The supporting-script guidance below stands for what is still not native.

Tests often check values that depend on the day they run, in the form the application shows them: a two-night stay booked a week ahead reads "20–22 Oct 2026", "30 Oct–1 Nov 2026" or "31 Dec 2026–2 Jan 2027". {date+N} covers numeric dates, not a localized range, so today a recording freezes the displayed dates or falls back to a date-shaped pattern that does not check the stay, and projects write setup scripts to generate the expected text.

Localized date ranges. A plain-language assertion names the range by relative dates, or by a relative start and a number of nights. The recording keeps that intent with its locale, timezone and display style, and every replay, the verification replay included, recalculates the expected text with a standard date formatter, at zero AI calls. The check runs against the date-summary element itself; healing cannot change the expected dates. Existing {date+N} recordings keep working, and the new form raises the recording format version. If the standard style cannot reproduce a compact form faithfully, the gap is documented before any custom formatting option is added.
Guidance for supporting scripts. Until a capability is native, the skill and the instructions saffron init writes tell an agent when a small project script is justified and how to keep it honest: native features first, expected values derived from test inputs rather than from the output being checked, generated at run time and read through {data:...}, wired through documented hooks, never by editing a recording by hand.

Not a general expression language, and no executable code in recordings. More native capabilities of this kind follow: each supporting script that projects keep writing is a candidate for one.

Parallel and cross-browser: what is left

Both are shipped in their core form (M5): --workers N replays concurrently, and --browser chromium | chrome | msedge | firefox | webkit replays on any engine, branded Chrome and Edge included. Sharding across CI machines shipped on 2026-09-22: --shard 2/4 splits the suite by scenario, or by feature file with --shard-by file, and saffron report --merge puts the shards back together into one report and one history line. Shipped on 2026-10-06: parallel recording and healing under a cap (--agent-workers N, default 1, each session in a browser of its own), per-worker timing in the report, a browser matrix in one run (--browser chromium,firefox,webkit, a row per browser in one report) and device and viewport profiles (--device "iPhone 15", --viewport 1280x720).

M12. Secrets: `{env:VAR}` tokens

Status: Tier 1 SHIPPED 2026-07-07; Tier 2 (agent-blind secrets, SAF-13) SHIPPED in 1.1.0 (2026-10-22). Live-validated end to end: an agent recording that TYPED the real credentials produced artifacts containing only tokens: grep for the secret across proposal, cache, reports and history found zero hits; replay resolved from env at 0 tokens; a missing var failed fast by name; .env filled gaps with real env winning.

Credentials must never live in feature files, and in Saffron that contract has to hold across SEVEN surfaces: feature files, committed caches, proposals, the agent conversation, reports, history.jsonl, and terminal errors.

Tier 1 (scoped):

{env:VAR} tokens, deliberately the {date+N} execute-time family, NOT <param>: they resolve lazily in executeAction, so every replay path (runs, proof replays, seeded islands, assertion guards) is covered by one resolver, and the seeding machinery is untouched.
Caches/proposals store only the token: recording masks referenced env VALUES back to tokens (the deparameterize mechanism).
Output hygiene: narratives, adaptations, error strings, reports, and history get a masking pass over the referenced values only, the environment is never enumerated.
Preflight: missing env vars fail fast, by name, before the run.
.env auto-load from the project root (git-ignored; real env always wins), so local dev and CI behave identically.
Honest limitation, documented: during RECORDING the agent must type the real value, so it transits the AI provider, replays never do. Recommendation: record with rotatable staging credentials.

Tier 2: agent-blind secrets (built 2026-10-22, in agentToolProxy, which every provider's browser calls already pass through), an MCP proxy wrapped around the Playwright MCP server: the model sees and types only the token; the proxy substitutes the real value into the browser and masks the code-echo on the way back. The model never holds the secret. Every result is masked too, so a secret the page displays reaches the agent as its token. Residual: values under three characters, a page that shows the value transformed (reversed, base64), and same-text matches masked along with it (HOW-TO-USE, Secrets).