Design log

Distilled from the founding design discussion (2026-07-02). This is the "why" behind the architecture; the "how" is in the README and the code.

The idea

Tests are Gherkin .feature files. First run: an AI agent executes the scenario in a real browser and records a deterministic cache. Every later run Saffron replays the cache in a real browser at zero tokens. When a cached step fails at runtime, the agent takes over mid-run, adapts, and files a cache proposal the user accepts or rejects: snapshot-testing UX for user journeys. Reports state adaptation, AI calls, tokens, and cost honestly.

Competitive landscape (refreshed 2026-09-19)

Checked against each vendor's own documentation on the date above. Vendor docs establish what a product does, not how reliable it is, so nothing here is a claim about quality. Where a row compares versions, the version is named, because Stagehand's answer changed between v3 and v4.

Tool	What it already does	What is still different about Saffron
Playwright Test Agents (planner, generator, healer)	The planner writes a Markdown plan, the generator turns it into ordinary Playwright tests, the healer patches failing ones and re-runs them. Generated tests run with no model involved, so they also replay at zero tokens.	This is the strongest baseline: "an agent writes Playwright and we review its PR". The healer is a separate maintenance step, not a mid-run fallback; the artifact is code rather than a plan with a protected assertion block; there is no built-in report of what was adapted and what it cost. Saffron does not beat it on replay cost.
Playwright-BDD	Runs Gherkin through the Playwright runner, with an established ecosystem behind it.	Gherkin plus Playwright is already available. Step definitions are hand-written glue; Saffron's case is removing that glue for supported browser flows, and it has to justify a separate runtime to do it.
Karate Agent (Karate Labs)	Reusable flows are .js scripts that run at native speed with no LLM tokens; in autonomous mode an LLM drives an observe-decide-act loop and recovers failures.	Deterministic execution plus AI recovery is not unique to Saffron. Karate Agent is commercial (licence file, evaluation on request; core Karate is open source), and its docs describe no accept or reject step for what a recovery changed.
Stagehand v3 (Browserbase)	Two caches: a Browserbase server-side one, and a local cacheDir whose files are portable. The docs recommend committing that directory so CI does not run inference on first execution.	A cache committed to git is therefore not unique either. Stagehand is an automation SDK, not a test runner: no distinction between actions and assertions, no review gate, and after a DOM change the documented remedy is clearing the cache by hand.
Stagehand v4 (Browserbase)	Caching is managed server-side only: no cache files, and the cache option has no effect with a local browser.	Here the cache is not a reviewable artifact at all, and it requires the vendor's browser.
mabl	Selector auto-heal. Per the retirement notice the external review cites, agentic runtime recovery was introduced and then disabled across workspaces on 2026-08-03 after early-access results. We could not open that page when refreshing this table (bot check), so treat the date and wording as the review's, not ours.	A caution about exactly the workflow Saffron relies on. The notice does not publish enough to infer a technical cause. It is the reason proof replay, sacred assertions and the review gate are mechanical rather than advisory here.
Momentic, testRigor	Natural-language tests with self-healing locators, delivered as SaaS.	Not re-verified in this refresh. The earlier version of this table said tools in this group use AI on every run; that was a claim about a whole category without product-specific evidence, and it has been removed.

Sources: Playwright Test Agents · Playwright-BDD · Karate Agent overview · Stagehand v3 caching · Stagehand v4 caching · mabl retirement notice

What is not unique: zero-token replay (generated Playwright has it, Karate flows have it, Stagehand's cache has it), AI recovery at runtime (Karate Agent), and a cache you can commit (Stagehand v3).

Positioning: Saffron's niche is the combination, as a workflow. Gherkin is the source of truth with no step glue. The recording is a plan a reviewer can read, with assertions the agent is mechanically unable to weaken. A heal happens mid-run, is proof-replayed at zero AI, and lands as a proposal with an action-level diff that someone accepts or rejects. Reports say what was adapted and what it cost. Against ordinary generated Playwright this is not a cost advantage on replay; the honest comparison is total maintenance: authoring, repair review, flake investigation and missed defects.

Locked design decisions

1. Cache = structured JSON action list per scenario. Semantic targets (ARIA role + accessible name, label/text/testId, CSS fallbacks) and <param> placeholders so one cache serves all Examples rows of a Scenario Outline. Raw MCP tool-call replay was rejected: Playwright MCP element refs are snapshot-scoped and unstable across page loads.
2. Assertions are never healed. Then steps compile to expect* actions; the agent may fix how a run reaches an assertion, never what it checks. Enforced mechanically, not just by prompt (see below).
3. Three result states. Green = cached pass. Yellow = passed-with-adaptation (proposal pending review). Red = failed. Yellow exits 0 with a warning; a --strict flag is on the roadmap.
4. Adaptations propose two diffs: the cache diff and a .feature step rewrite (saffron accept --with-feature-edit), so documented truth tracks executable truth.
5. Saffron's own replayer, on Playwright's browser engine. The cache is JSON, not a generated script. The Playwright MCP server only runs in agent mode; agent and replayer share one browser over CDP so healing continues from the exact failed page state.
6. Claude-first behind AgentProvider. Claude stays the default; Codex, Google AI (Antigravity) and Cursor subscriptions joined on 2026-09-29, each through its vendor's official client (see the entry of that date).
7. Free CLI, open-core SaaS path. Originally planned as Apache-2.0 from day one; amended 2026-07-07 (see "Distribution" below): the public release is free to use but closed-source, with open-sourcing as a later, deliberate step.

Key rationale

• Regression-masking is the existential risk. An agent whose goal is "reach the end anyway" will route around real bugs. A test tool that masks regressions is worse than none. This drove decisions 2 and 3.
• Verified in practice: during MVP verification the agent did try to rewrite a failing assertion to a "close enough" check (renamed heading) and reported success. Prompt rules alone did not stop it. Hence the mechanical guards: heal merges never take agent-recorded assertion actions, and when the failed step is an assertion the orchestrator re-executes the cached assertion as written on the live page: red if it doesn't pass.
• Flake vs drift: bounded retries with backoff run before any AI escalation; the agent's first job on takeover is diagnosis (UI drift → adapt; app defect → fail with explanation), not goal-seeking.
• Adaptation breaks the Gherkin contract unless feature files evolve with the cache: hence structured featureEdits in proposals.

Execution flow

Locator maintenance: propagation over a registry (decided 2026-07-03)

Locator duplication across caches is real (a shared login button appears in every scenario; one rename = N heals). The classic Page-Object-Model answer , a shared locator registry with indirection: was considered and rejected for now: agent-recorded caches would need entity resolution to decide two targets are "the same element", and a wrong registry heal silently fans out to every referencing scenario, inverting Saffron's blast-radius guarantee.

Instead: same-target heal propagation (saffron accept --propagate). When an accepted heal rewrites a locator, every other cache whose target has the identical semantic core (role/name/nameRegex/label/text/testId, fallback CSS excluded) gets the same rewrite. Identical core = identical Playwright query, so the identity proof is exact, not heuristic; fan-out is printed per scenario at accept time. Caches stay self-contained. An opt-in named-locator registry remains a candidate for the review-UI era.

Step reuse: Level 1 before Level 2 (decided 2026-07-05)

Step-level sharing (the Cucumber step-definition model) survives the objection that killed the locator registry: step text is author-declared identity: two scenarios using the same text is a human assertion of sameness, not an inference. The spectrum: Level 0 (per-scenario caches + propagation) → Level 1 (shipped): a derived step index seeds new recordings ("record the gaps"), storage stays per-scenario, divergence is instrumented → Level 2 (candidate): a shared step library as source of truth, which additionally needs polymorphic-step (heal ping-pong) detection, per-scenario overrides, and affected-scenario re-verification. Level 1 captures the dominant economics (new-scenario recording cost scales with novel steps only) without shared-mutation risk; its divergence data decides whether Level 2 is warranted. Non-prefix (island) seeding shipped 2026-07-05: seeded islands replay zero-AI, bounded agent segments (max 3 per scenario) record the gaps, agent overshoot past a segment boundary is adopted instead of replayed twice. No remaining Level-1 scope gaps: the open question is Level 2, decided by divergence data. Quoted-value generalization shipped 2026-07-05: lookups fall back to a quote-generalized pattern and substitute differing args by exact-field equality only (substring occurrences and ambiguous duplicate args bail). Data tables shipped 2026-07-05: tables are part of the staleness contract; 2-column key/value tables parameterize recorded values as <table:key> (value edits replay free, key edits re-record), and seeding matches on table keys.

Wording duplication: surface the derived index, prove duplicates by behavior (decided 2026-07-05 · SHIPPED 2026-07-06)

The Cucumber pain: the same step exists under two wordings ("I navigate to…" / "I visit…"). In Saffron the pain is sharper, exact text is cache identity, so wording drift is an index miss that re-records at token cost, but the architecture also enables a fix Cucumber cannot have.

Challenged and rejected: a manually-curated step registry. The step index derived from committed caches already is the registry; a second, hand-maintained one duplicates truth and rots. Decided instead (scoped as M8): surface the index (saffron steps); behavior-proven duplicate detection: two wordings whose recordings produce identical action lists (actionsEqual) are a duplicate by definition, not by string similarity, and trigger a mechanical rename proposal through the existing feature-edit machinery; and saffron author: AI drafts feature files from prose with the index injected as required vocabulary, which makes authored files maximally seedable (authoring quality becomes recording cheapness). IDE completion (VS Code extension over .saffron/cache) deferred to the public-release era. Prevention converges wording before recording; detection converges it after, at the cost of exactly one recording.

Step sets: parse-time expansion, and the .saffron superset dialect (decided 2026-07-05 · SHIPPED 2026-07-06)

The second Cucumber pain: the same 5–6 prelude steps pasted into every scenario of a file (Background only covers scenario starts, file-wide). Decision: named step sets, StepSet: blocks invoked by a reference step (scoped as M9).

Why expansion beats the alternatives:

• Composite steps (one declarative step recording N actions, the Gherkin-purist answer) fork the vocabulary: the composite can't share recordings with scenarios that use the fine-grained steps individually.
• Runtime references (cache points at a shared recording) reintroduce the shared-mutation risk that deferred Level 2.
• Parse-time expansion (like Background merging) preserves each sub-step's exact text identity, so caching, Level-1 seeding, heal propagation, and staleness work with zero engine changes. A set edit makes its scenarios honestly stale; re-recording is almost fully seeded. Step sets are parser sugar with outsized leverage, a feature uniquely cheap in Saffron because identity is text.

Why a new file extension (initially challenged, then adopted with conditions): the user's .saffron proposal is a superset dialect, normal Gherkin plus the StepSet: keyword, which is the TypeScript playbook (.ts = .js + keywords, honest new extension). A keyword genuinely reads better than a tag convention, and the deciding argument: a .feature file containing StepSet: would break standard parsers while claiming to be standard: worse than an extension that declares itself. The pre-processor rewrites StepSet: lines to Scenario: same-line (line numbers preserved) and reuses @cucumber/gherkin (no fork); docs ship the files.associations / .gitattributes snippets for day-one highlighting. v1: unique project-wide set names; tables/doc strings/params allowed inside sets; no nesting; sets never run standalone but their steps join the seeding vocabulary.

Global step sets (discussed 2026-07-06): a dedicated .safset extension for library files was proposed and rejected, research showed cross-file resolution already shipped with M9 (the registry is project-wide by construction), so the extension would split the dialect into two grammars and double the IDE-tooling bill for a distinction the registry never needed. Adopted instead: the .d.ts-style naming convention: a sets-only features/shared.steps.saffron library file , documented across the guide/FEATURES/HOW-TO-USE. Deferred until real pain appears: header-less library files, a stepSets config path for sets outside the features tree, and namespacing for very large suites (the unique-name parse error will surface that pain visibly).

Syntax locked from the user's worked example (2026-07-06): definition StepSet: <name> (colon, block keyword), invocation StepSet <name> (no colon, step position), grammatically consistent with Gherkin (block keywords carry colons, step keywords don't). Challenged and resolved: (a) the colon-typo hazard at an invocation site (an accidental block would swallow the scenario's tail) is caught by the project-wide unique-name rule: the accidental block duplicates an existing set name and parse-errors with a "remove the colon to invoke" hint; a distinct Use keyword was considered and rejected to keep one symmetric spelling. (b) Sets are .saffron-exclusive, the earlier * steps: legal-Gherkin fallback for plain .feature files was dropped (two spellings, one feature = teaching cost); adopting sets in an existing .feature is a pure file rename, since the dialect is a superset. (c) Assertions inside sets inherit position-dependent strictness automatically after expansion: mid-scenario invocation → checkpoint semantics; final invocation → the set's trailing Thens join the always-strict final assertion block. (d) The set shape convention: guard step first, exit assertion last, every invocation carries its own precondition and success check. (e) The true implementation cost is the edit-back path: featureEdits need step provenance ({file, line, fromSet}); editing a set definition fixes every caller at once (callers go stale, re-record seeded).

IDE integration: vocabulary to humans via LSP, to AI via MCP (decided · SHIPPED 2026-07-06)

Design for step completion while writing .saffron files (scoped as M10). Key decisions from the discussion:

• Vocabulary = parsed files ∪ cache index, not caches alone, a teammate's written-but-unrecorded steps must complete too, or parallel authors re-diverge. Caches contribute status, which is the differentiator: completion items badge each suggestion ✓ recorded / ⚠ divergent / ○ unrecorded: the editor shows which steps already replay for free. No Cucumber tool can offer this because no Cucumber tool knows what a step costs.
• Diagnostics over completion: the author who confidently types a fresh wording never opens the popup, so completion alone can't kill the duplication pain. A near-duplicate squiggle with a rename quick-fix is the prevention layer; M8's behavior-proven detection remains the cure after recording.
• Two consumers, one index: humans get an LSP (portable to JetBrains/Neovim; go-to-definition on StepSet invocations, hover with recorded actions and usage counts); AI assistants get saffron mcp (they can't read popups: they need the vocabulary as context; the interactive twin of saffron author).
• Cheap first rungs: saffron steps --snippets (native VS Code snippet completion, zero extension code) and --json ship long before the language server; the extension itself waits for the .saffron parser (M9) and publishes with M6.
• Shipped 2026-07-06: the VS Code extension went first (plain-JS providers, sibling repo); saffron lsp followed same-day for JetBrains/Neovim: LSP4IJ covers Community editions, so the "paid IDEs only" limitation of JetBrains' native LSP API doesn't bite. Near-duplicate threshold calibrated at 0.7 token-Jaccard: catches typo-level drift, leaves synonym-level duplicates to the runner's behavior-proven rename proposals (lexical similarity cannot prove synonymy; identical recorded actions can).
• Rejected: reusing the official Cucumber extension, it indexes glue-code step definitions, which Saffron deliberately lacks.

Distribution: free first, open-source later (decided 2026-07-07)

The public release ships as a free, closed-source npm package, the Claude Code model: npm install, configure ANTHROPIC_API_KEY (or use a Claude Code login), free for any use including commercial, no redistribution or modification. Mechanics: the published artifact is a single bundled+minified CLI (no sourcemaps, files whitelist), under the Saffron Free Use License v1.0. Relicensing from the earlier Apache-2.0 plan was clean because nothing had ever been distributed and there are no outside contributors: the door that closes at first publish, which is why the decision was made now. Open-sourcing is a scheduled later milestone, not abandoned: shipped JS is legally protected but never secret, and the community value of open source remains part of the plan once the tool has traction. Feedback runs through GitHub issues on the public site repository (saffron-ai) until the runner repo itself goes public.

Secrets: seven leak surfaces, two tiers (decided · Tier 1 SHIPPED 2026-07-07)

"Read secrets from env vars" is the easy fifth of the problem. Saffron persists more artifacts than Cucumber, so a password in a feature file reaches seven places: the file, the committed cache (the sneaky one), proposals, the agent conversation, reports, history.jsonl, and terminal errors. The design is therefore a pipeline property:

• Syntax {env:VAR}, not <env:VAR>, the <param> family resolves early (prompts, resolvedText) and participates in seeding resolvability checks; env tokens there would leak wider and break seeding. The {…} family resolves lazily at action execution, where {date+N} already lives: one resolver covers every replay path.
• Masking is scoped to referenced values: only vars the scenarios actually use are masked/checked; the environment is never enumerated into params or prompts.
• Tier 1 kept secrets out of everything Saffron writes, but the agent had to type the real value while recording. Tier 2 (SAF-13) removes that: agentToolProxy, which every provider's browser calls already pass through, substitutes {env:NAME} into the arguments on the way in (after the recording hooks and the trace label see the token, before the URL and upload guards judge the real value) and masks every result, error and refusal on the way out, as do Saffron's own tool replies. Residual: values under three characters, transformed spellings, and same-text matches masked along with the value.
• .env auto-loaded from the project root (real env wins), decided for dev/CI parity.

Roadmap

Maintained in ROADMAP.md: milestones M1–M10 with statuses and decision gates (run history, vocabulary completion, the gated Level-2 shared step library, assertion policy v2, runtime/providers, public release, SaaS layer, step vocabulary & authoring, step sets & the .saffron dialect, IDE integration). This log stays the record of decided designs; the roadmap is the single source of truth for pending work.

2026-07-19: Network-aware steps (vocabulary, not keywords)

The user raised the classic UI-test synchronization need: wait for a backend call (e.g. a 200/201, or a polling endpoint reporting READY) instead of a page signal, and asked whether Saffron should introduce keywords. Decision (challenged and locked): no new Gherkin keywords. StepSet: earned syntax because it is structural; a network wait is behavioral, and behavior belongs in the recorded action vocabulary behind plain-prose steps: keywords would start a glue DSL and break the "any Gherkin is valid" superset story. Two actions instead: waitForResponse (healable synchronization) and expectResponse (assertion: sacred under the existing kind system, so API contract checks can never be healed). Matchers: URL regex (path-only, volatile IDs must be patterns: the nameRegex lesson applied to URLs), method, status exact-or-class, body regex (v1 includes body matching by user decision, for polling waits). Replay uses a context-level rolling response log (covers new tabs); waits match responses observed since the previous step began: covers both "click and wait" in one step and a standalone wait right after its trigger: while expectResponse audits the whole scenario. Recording is verify-first via the MCP's network log. Live proof: both example scenarios recorded verified on the first attempt, including the body-pattern polling wait.

2026-09-13: Ship guidance for agents with the package

Most test code is now written by AI assistants, and Saffron's economics depend on wording discipline those assistants do not know by default. The user proposed shipping a skill with the npm package. Decisions:

• Format and placement: an Agent Skill (SKILL.md + references/) in a skills/ directory at the package root: the convention the ecosystem converged on for packages (skills-npm, TanStack Intent). .agents/, .claude/, .cursor/, .github/ skill folders are the consumer project's discovery locations, not something a package should contain.
• Installation is an explicit command, never a postinstall hook: npm cannot legitimately write into a consumer's project at install time. saffron init copies the skill into the agent directories (default .claude/skills + .agents/skills), idempotently.
• No shipped AGENTS.md: it is the consumer's file. init upserts a small marked block (load the skill, reuse the vocabulary, never hand-edit caches, secrets as tokens) into AGENTS.md (created) and CLAUDE.md (only if present).
• MCP registration belongs in the same command: a bundled MCP server is inert until the host knows about it, so init merges a saffron entry into each host's project-scoped config (.mcp.json, .cursor/mcp.json, .vscode/mcp.json) rather than leaving a manual step. Codex has no project-level file and gets a printed one-liner.
• The skill states the runner's rules, not its internals: exact step text as cache identity, sacred Then placed last, no volatile or secret literals, StepSet syntax (including the bare-name mistake), network waits and page furniture phrasings, the run → review → accept loop, and an anti-patterns table. Progressive disclosure per the spec: a short SKILL.md, detail in reference sheets.

2026-09-21: Test-data isolation: setup and teardown (SAF-33)

The problem

A scenario assumes the application is in a known state. "Cancel an order" needs an order. Most scenarios also leave something behind: "create a customer" leaves a customer after every run. Without a way to prepare state before a run and clean it up after, a suite passes on a fresh database and fails on the second run, or passes alone and fails in parallel because two scenarios touch the same record.

Two halves:

• Setup puts the application into the state the scenarios need.
• Teardown removes what the run created or changed, whether it passed or failed.

Cypress and Playwright answer this with code: beforeEach, afterEach, API calls, database tasks. Saffron has no code layer by design, so it needs its own answer.

What already exists

Need	State
Two runs creating the same record	Solved: {unique:name} (M19)
Browser state leaking between scenarios	Solved: a fresh context per scenario
Logged-in state	Solved: storageState
Data a scenario needs before it starts	Only through the UI (Background, a StepSet): slow, and itself a recording that can break
Cleaning up afterwards	Nothing: data piles up in the environment
Cleanup when a scenario fails midway	Nothing

Options

• A. UI only. Keep Background and StepSets. Nothing to build. Slow, and teardown through the UI is fragile: it never runs when the scenario fails before reaching it.
• B. Hook commands in the config. "setup": "npm run seed" and "teardown": "npm run clean": shell commands Saffron runs around the run. Zero tokens, no new syntax, works with whatever the team already has (a seed script, a SQL file, a curl). Coarse: per run, not per scenario. The team writes the script.
• C. API steps in the feature file. A step that makes an HTTP call instead of driving the UI. Fast and per-scenario, and readable in the scenario. Costs a new step type, and a home for endpoints and auth.
• D. Tagged hooks. @setup / @teardown scenarios or StepSets that always run, teardown even on failure. Pure Gherkin, still UI speed.

Decision

B now, C later. B is small, covers what most teams need on day one, and takes nothing away from C. Together with {unique:name} it is enough for a suite to be re-runnable and parallel-safe: setup seeds the shared read-only data, unique values keep the written data apart, teardown sweeps up. D is rejected for now: it is A with better failure behaviour, and C makes it unnecessary. C remains the proper per-scenario answer and gets its own ticket when a user asks for it.

B, as built

• Config keys setup and teardown: one command, or a list run in order. Run through the shell, in the project root, once per saffron run.
• Setup runs before any browser opens. A non-zero exit stops the run, exit code 2: running scenarios against a half-seeded application only produces misleading reds.
• Teardown always runs once setup was attempted: after green runs, red runs, a failed setup (it may have seeded half of it), and Ctrl-C. On Ctrl-C a running setup is killed, process tree and all, and awaited before teardown starts, so a cancelled seed cannot write behind the cleanup. Teardown itself is not interruptible.
• A failed teardown does not change a scenario's verdict. A run whose scenarios failed still exits 1; a run that otherwise passed exits 2 and says so: a dirty environment is the next run's false red, and CI should hear about it now. The remaining teardown commands still run.
• The commands receive SAFFRON_BASE_URL and SAFFRON_ENV, so one script serves every environment.
• hookTimeoutMs (default five minutes) bounds each command.
• --no-hooks skips both, for when the state is already prepared, such as re-running one scenario from the IDE against a seeded app.
• Hook output goes to stderr, so it never mixes into anything a tool parses from stdout. The report records each command, its exit code and its duration.

2026-09-24: The baseURL is a prefix; heals report flow changes

Found in a live run against a booking site's A/B variants.

The baseURL is a prefix, path and query included. Recorded navigations were cached by stripping the baseURL as a string and replayed with new URL(path, baseURL), which drops the base's query and, for a leading-slash path, its path too. A cache recorded against ?flow=standard could therefore never replay against ?flow=checkout-guest. Now relativizeUrl and resolveUrl (src/util/url.ts) treat the baseURL as origin + path prefix + default query: / is the baseURL itself, /rooms is the prefix plus /rooms with the base query, a cached URL with its own query keeps it, and a bare ? records "no query" under a base that has one. Caches keep storing plain paths, so nothing changes on disk and old caches replay unchanged.

Heal notes name the kind of change, and Saffron checks the flow. The agent's account of a heal described a removed page as a locator tweak. Two layers now:

• The heal prompt gives the page replay stopped on and asks the agent to classify the change (locator or label, page removed or merged, page added or reordered, timing) in its adaptation and narrative.
• The orchestrator compares the healed recording with the committed one, per step, and files flow change: notes for navigation actions (click, press, goto, tab moves) that were dropped or added. Same-type rewrites are paired first, so a renamed button is a locator change and adds no note.

Considered and rejected for now: storing the page URL on every recorded action (next to pageFingerprint) to compare "expected page" with "actual page" at the failed step. That is a new cache field touched by the step index, the diff, table tokens and status output; the action comparison already catches the removed-page case without it.

2026-09-24: The agent works on the live page, and only on its steps

Found in two live runs: a seeded recording whose proposal held only "switch to tab 1" for step 1, and a heal whose step 1 held a tab switch plus the search and room clicks of steps 2 and 3.

Why the agent was on the wrong page. Playwright MCP attaches with connectOverCDP and works in browser.contexts()[0]. Over CDP, pages of browser contexts that client did not create land in its default context, so the agent sees every Saffron page as one tab list and starts on the first. During a refinement the scenario's own page is still open beside the proof replay's page, so the agent started on the page where the heal ended, selected a tab, and in one run navigated back to the start.

Fix, deterministic first. Agent sessions run one at a time and Saffron owns every context in the browser, so before a heal, a refinement or a recording segment it closes the pages of every context except the live one (focusLivePage, src/orchestrator/live-page.ts). The MCP tab list is then the live context's tabs, in the order replay counts them for switchTab. Within the live context the tabs are real (a popup the scenario followed), so nothing there is closed; the prompt names the live tab's index and URL instead.

Attribution. A session now knows the first step it may record: the failed step for a heal or refinement, the segment start, or the first unseeded step. Actions under an earlier step are discarded.

What happens before the first saffron_step depends on the session:

• Heal and refine start at a known failed step, so what the agent does before announcing it is diagnosis and is discarded. First tried: holding those actions for the first announced step. A live re-run showed why that is wrong: the heal re-clicked the room button to see where it leads, the click landed in the failed step, and the proof replay clicked a room button on the guest information page. The prompt now says to announce the failed step before any state-changing action and to look with snapshots.
• A seeded recording segment is recording, not diagnosing: the page is where the seeded steps left it and nothing failed there, so early actions are almost always the agent starting its first step before announcing it. They are held and filed under the first announced step at or after the segment start (a tab selection is dropped, since replay is already on the live page). Discarding them would lose real work and leave the proof replay to fail.
• Fresh recordings keep filing early actions under step 0.

Considered and not done: dropping an action after the announcement when it repeats one from a completed step (the agent redoing an earlier step). Repeating a control is often legitimate (a "Continue" or "Add to cart" button on several pages, a second search), and a target match cannot tell the two apart, so a drop could silently break a correct heal. The flow change: notes and the proof replay already surface a real redo.

Merge safety net. mergeHealedSteps keeps the committed recording for every step before the failed one, and the refine merges keep every step the proof replay passed, whatever the agent's session holds. Each discard or re-filing leaves a recording: adaptation note.

2026-09-24: Bound proposals, and recordings validated on read

The slogan is "changes you approve". Two gaps made that a claim rather than a mechanism.

Acceptance was of a file, not of a review. saffron accept promoted whatever proposal was there. Between filing and accepting, the scenario text, the committed cache it was diffed against, or the proposal file itself could move (an edit, a colleague's heal, --propagate, a pull, a hand edit), and the promotion went through unread. Feature edits already had a drift check; the recording did not.

Decision: every proposal carries three sha1 digests (16 hex chars each) of what it was made against, computed when it is filed: the scenario's steps as [kind, text, table, docString] (Examples values excluded: rows share a recording and the cache's own staleness rule judges a changed cell), the committed cache's contract (steps and data snapshot, fingerprints excluded because they are advisory), and the proposal's own executable content: its recording plus the feature edits --with-feature-edit applies (narrative and adaptations are read, not applied, so they are not bound). The runner judges staleness against the scenario it has already expanded; the CLI and status parse with the StepSet registry, so a StepSet invocation never reads as a parse error and a stale proposal. saffron accept recomputes all three and refuses by reason when any differ. No override flag: an override would be the old behaviour with a name, and the honest path (re-run, get a proposal bound to now) costs a replay, not a recording, when the committed cache still works. Staleness is shown wherever a proposal is listed (accept, diff, status, both IDEs) so nobody meets it at the button. A run that finds a stale pending proposal records afresh instead of proving it again. Unbound proposals from older runners are accepted with a note, not refused: refusing would strand every proposal in flight at upgrade time.

Caches were parsed and trusted. loadCache checked version === 1 and nothing else. The schema has gained expectEach, opensTab, data, each and owner.json in three weeks; a mistyped action in a hand edit or a file from a newer runner replayed as garbage or threw mid-run.

Decision: one validator at the door (src/cache/validate.ts) for caches and proposals, naming the first problem by JSON path. It checks the operands each action type needs, not only that present fields are typed right: an expectUrl without a URL used to compare "" with every page and pass. An explicitly empty value stays valid (a recorded clear); the recorder's own guard judges those. The schema version stays 1: fields added within a version are optional and ignored by older runners, so a newer cache still reads on an older runner, while an unknown action type or a version: 2 is refused with "update saffron-ai". Bumping to 2 was rejected because it would refuse every cache in every repository for no on-disk difference. An unreadable recording is a red with the reason, never a re-record over the file: the person who committed it decides. Scans (step index, propagation, status, listings) skip an unreadable file and report it; explicit commands on a named file refuse it.

2026-10-01: CI heals travel as bundles (M20 step 1)

A heal in CI filed its proposal in a workspace thrown away with the job; getting it into the repository meant a second agent session on someone's machine, paid again and free to heal differently.

One zip, byte for byte. saffron export writes the pending proposals, the last run's report (latest.json, or a shard's report on a shard's machine: whichever finished last), the screenshots and traces the report names, and manifest.json: format version, runner version, repository, branch, commit, CI run, attempt and link, shard, and each proposal's binding and revision. Proposals travel as their exact bytes, because the revision a reviewer accepts is the sha256 of the file and the binding is what makes a proposal from another machine safe to accept. A zip rather than a folder: one file to upload, download or link, and the reader and writer already existed for traces (src/util/zip.ts), so no dependency. Pictures and traces are stored, not deflated again. Export streams the zip to disk (writeZipFile): each evidence file is read in chunks once to check its digest and take its CRC, then copied into the zip in chunks, so no trace is ever held in memory whole; a file that changes between the two reads is refused, and the zip is renamed into place only when complete.

The imported run is the latest run. saffron trace, the HTML report, status --json and both panels all read .saffron/reports/latest.json. Import installs the bundle's report there (marked imported, with the CI run's link, and its proposal paths rewritten to this checkout) and its evidence under .saffron/artifacts/, so every surface works unchanged. A second place for imported runs was rejected: every reader would have needed teaching, for a report that the next local run replaces anyway.

Stale is refused at import, and checked again at accept. A proposal whose binding does not fit the checkout could only be rejected there, so it is not copied: it is named with the reason, with a hint to check out the bundle's commit when the checkout is elsewhere. The rest import; nothing is all-or-nothing except integrity. The binding is rechecked by saffron accept as always, because the branch can move in between. An unbound proposal is refused, since nothing could tell whether it fits, and so is one naming a feature file outside the project, which --with-feature-edit would otherwise write to.

Repository identity is a guard, the binding is the safety. A bundle is refused when no remote of the checkout is its repository. Paths are compared, and a different host name only warns: an SSH host alias or a mirror is still the repository, and every remote counts, so a fork with an upstream remote imports. HTTPS and SSH spellings (Azure DevOps has three) normalize to one, and credentials in a CI remote URL are dropped before anything is written. A commit mismatch warns, with the distance when git can tell, and never refuses; the commit recorded for a pull request build is the head of its branch, which the reviewer has, not the merge commit CI tested, and only when HEAD is a merge commit whose second parent is that head. GitHub sets GITHUB_HEAD_REF only for pull_request and pull_request_target, so a pull_request_review build reads the branch from its event (pull_request.head.ref), never <n>/merge. A pull_request_target build checked out the base branch: it records that branch and the commit checked out, never the fork's. A bundle with no repository (made outside git) is refused unless --allow-unknown-repository, which imports it with a warning; a checkout with no remote gets a warning, not a refusal.

Integrity before anything is written. Every file's digest is in the manifest, and every path must be a slug path under proposals/ or artifacts/, so nothing lands outside .saffron/. An entry is inflated no further than its header says and never past a gigabyte. Only each bundle's central directory and manifest are held in memory: every entry is read from the bundle file on disk one at a time (a URL's bundle is downloaded to disk first), so one import takes up to 8 GB unzipped, every shard of a large run together. Export applies the same limits, and drops green scenarios' traces first when a bundle would be over the 4 GB a zip holds. A damaged bundle, an unknown format or a foreign repository is refused whole with exit 2. Importing twice rewrites nothing and says so; after accept it says the proposal is already accepted.

Shards. One bundle per shard. Import merges their reports with the report --merge rules, relaxed on one point: a partial set imports, because CI exports the shards that failed, not the ones that passed. The report says which shards it holds.

Settled from the open questions. Names: saffron-bundle.zip, and saffron-bundle-shard-<i>-of-<n>.zip on a shard, under the git-ignored .saffron/reports/ by default; CI examples pass --out because upload-artifact leaves hidden folders out. Retention is the CI's own setting. The imported report links to the run that produced it. Green runs' traces travel when the export step runs on every run; no flag was needed for that.

2026-10-01: CI opens a draft pull request (M20 step 2)

For teams that review in pull requests. The design allowed either saffron export --branch or a documented workflow step around export. Built in, because the hard parts are the ones a copied workflow gets wrong: which commit to build on, committing only the proposals, and never pushing onto a branch that exists. The workflow step is one line.

A commit beside the checkout. The proposal files are written as blobs and committed on a temporary index read from the tested commit (read-tree, update-index, write-tree, commit-tree), then pushed as <commit>:refs/heads/<branch>. The CI checkout is never switched or staged, the commit holds the proposal files and nothing else, and it works where .saffron/proposals/ is git-ignored. The caches are not committed: accepting is the reviewer's act, on their machine, with the bindings checked there.

The commit the reviewer has. The branch starts on the bundle's commit, which on a pull request build is the head of the pull request's branch, not the merge commit CI tested. Building on the merge commit would drag the base branch's changes into the heal's pull request. A shallow checkout does not have the head, so it is fetched by hash, but only a commit the checkout's HEAD is, merged in or descends from: anything else is refused before a fetch or a push. Each proposal is judged against that commit's own scenario, recording, StepSets and data folder, materialised in a temporary tree, not against the CI checkout's. The pull request goes to the branch the run tested.

Never onto an existing branch. ls-remote first and refuse: pushing a commit whose parent is the tip of main onto main would be a fast-forward, landing unreviewed proposals on it. No --force.

gh on GitHub, a file elsewhere. gh is on every GitHub runner and takes the workflow's token from GH_TOKEN, so Saffron holds no credentials. Off GitHub the branch is pushed and the description is written to a file for the team's own pull request step; Azure DevOps comes with step 4, where it can be verified against a real organization. A failed gh leaves the branch pushed and exits 1 with its reason (often the repository setting that lets workflows open pull requests).

The description is data in code blocks. The narrative, adaptations, proof errors and diff come from a model and from the page under test; in markdown an @team would notify and a link would render. Everything of theirs is fenced (with a fence longer than any backtick run inside), names are inline code, and only the CI run link, which the runner builds from its own environment, is a link. Cut before GitHub's 65,536-character limit with a count of what was left out. The branch goes into the description's shell block as one quoted word, and a --branch or --base with a space, a quote or any of $ ` ; | & < > ( ) \ is refused up front: a reviewer pastes it into a shell.

Exit codes. 1 means one thing: the branch is pushed and the pull request could not be opened. Every other failure, of the export itself too (an --out that is a folder, an invalid config), is one line and exit 2. The same holds for every command: an unexpected error reaching the top is printed as saffron: <message> with exit 2, the stack only under SAFFRON_DEBUG.

Identity. Git's configured identity when there is one, else github-actions[bot] on GitHub Actions (GitHub's documented identity for workflow commits) and Saffron elsewhere, rather than failing on a CI machine nobody configured.

2026-10-01: CI runs in VS Code (M20 step 3)

From a red run to an accepted repair without leaving the editor.

The CLI does the work; the extension calls two commands. The CI runs view lists the current branch's runs with gh run list --json, and Import is gh run download <id> --pattern 'saffron-bundle*' into a private temporary folder followed by saffron import <folder> --json. Every check (repository, format, digests, bindings, commit distance) is the runner's; the extension shows the JSON: under the run, each proposal's outcome and every warning. gh holds the credentials, so the extension never stores or sees a token, and the download is deleted once imported, because the bundle's contents now live in .saffron/.

Which run a proposal came from is the runner's to say. Proposals travel byte for byte, so the provenance cannot go inside them. The imported report knows it, but the next local run replaces that report, usually before the reviewer accepts. So import notes it beside the proposals (.saffron/reports/imported.json, git-ignored: it describes this checkout, not the repository), keyed by file and revision, and status --json gives a proposal its importedFrom only while its revision is the one imported. JetBrains reads the same field in step 4.

What is not here. No polling: runs load when the view opens and on Refresh, because GitHub rate-limits and a run list is not a feed. No filtering to Saffron workflows: knowing which runs carry a bundle costs an API call per run, and Import on one without a bundle says so in one line.

2026-10-01: Azure Pipelines and JetBrains, through one CLI contract (M20 step 4)

"Done when both providers work in both IDEs through the one CLI contract." Step 3 had the VS Code extension call gh itself; adding az the same way would have meant two services implemented twice, in JavaScript and in Kotlin. So the CI work moved into the runner, behind two commands, and step 3's view was rebuilt on them before it shipped.

The contract. saffron runs --json lists the branch's runs as one shape: id, name, number, status (queued, in_progress, completed), conclusion (success, failure, cancelled, partial, or the service's word), event, branch, commit, link, and pending, the proposals from that run still pending here (from the import ledger). saffron import --run <id> --json is the import's own JSON. A problem (the tool missing, signed out, the repository not recognized) is a problem with a kind and a message saying what to do, on exit 2 with the JSON all the same, and an error with the same message so a client that reads only error works too; any other failure is {"error"} alone, as every import --json failure is (an invalid config included), and a runner that predates a command is told apart by its "unknown command". The editors render; they decide nothing.

Azure specifics. The provider follows the remote (Azure Repos means Azure Pipelines) unless ci.provider says otherwise, with ci.organization and ci.project for a GitHub repository built by Azure Pipelines. A pull request build's source is refs/pull/<n>/merge, so az's own branch filter would miss it: runs are listed wide and filtered by the branch the build validates (pr.sourceBranch), and its commit is the pull request's head (pr.sourceSha), the one a reviewer has. Every query, the pull request builds' included, starts at min(max(limit, 50), 500) builds and doubles up to 500 only while too few are the branch's. Azure has no artifact pattern: the run's artifacts are listed and the saffron-bundle* ones downloaded by name, plain names only, since they become folder names. The draft pull request is az repos pr create, its description passed as @file (the Azure CLI's own way of reading an argument from a file) so it survives cmd.exe, and cut to Azure's 4,000 characters. On Windows az is a batch file, which Node runs only through cmd.exe: every argument is quoted for it, with the rules the VS Code extension already used for npx.

JetBrains. A CI Runs tab after Proposals, on the same two commands, with the import's outcome as children of the run, and Proposals rows naming the run they came from. The tab's note gives way to the status the import itself reloads, so the outcome lives in the rows, not the note.

Verified so far. GitHub end to end, live, in a throwaway private repository with the documented workflow (the one stand-in: with no AI credentials there, a step filed the proposal the agent would have filed for a deliberately stale recording). The run went red and kept its screenshot and trace; export uploaded the bundle and --branch opened the draft pull request through gh, its one commit by github-actions[bot] on the tested commit. Locally, saffron runs listed the run, import --run pulled the real artifact, saffron trace replayed the runner's trace, and the VS Code import path did the same. Following the pull request's own instructions, a reviewer accepted with no agent, replayed green, pushed and merged, and the next run on main was green. One finding, fixed: the instructions staged all of .saffron/, which swept in the reviewer's local history line; they stage .saffron/cache and .saffron/proposals now. Azure only against a stand-in az that answers like the documented commands: no Azure DevOps organization was reachable from the session. The first real Azure run is the check that remains.

2026-09-29: Other subscriptions, through each vendor's official client

The ask: whoever already pays for Codex, Google AI Pro/Ultra or Cursor should record and heal on that plan, with nothing new to buy, the way a Claude login works today. API keys for those vendors come later.

Official clients only. The Claude provider runs Claude Code through the Claude Agent SDK on the user's own login. The others do the same with what each vendor ships for programs: @openai/codex-sdk (it runs the Codex CLI on the ChatGPT login), @cursor/sdk, and Google's Antigravity CLI (agy) headless. Reading a vendor's OAuth token and calling its backend ourselves was rejected outright: Google banned accounts for exactly that in February 2026, and it would break whenever a vendor changes an endpoint.

Google means Antigravity, not Gemini CLI. Gemini CLI stopped serving Google AI Pro, Ultra and free personal accounts on 2026-06-18; its SDK is unpublished and API-key only, and the Python Antigravity SDK takes API keys too. agy in print mode is the only client that runs on a personal Google subscription, and Google documents it for programmatic use. Its terms still forbid third-party tools reaching Antigravity with the user's login, and no Google statement covers a tool that runs the official binary for the user. The provider ships marked experimental, with that risk stated in the docs, and is named antigravity so gemini stays free for the API-key path.

One recorder: the tool gateway. Claude records through SDK hooks that see every tool call in process. The other agents run in their own process, and their SDKs have no blocking pre-tool hook (Codex reports a call after the fact; Cursor's tools run in its SDK; agy has shell-command hooks). Rather than three recorders, Saffron serves the Playwright and Saffron MCP servers to them over streamable HTTP on 127.0.0.1, behind a random bearer token sent as a header (a token in the URL would sit in Codex's command line, readable by any local user; Codex reads it from an environment variable instead), stateless per request, and runs each call through the same beforeTool/afterTool hooks the Claude SDK calls (agent/session.ts). An error result goes to failedTool there, as Claude Code reports it, so a refused call (an upload with no file chooser open) records nothing. A call runs to its end even when the vendor gives up on it (a cancellation, a closed request): Playwright MCP 0.0.77 finishes a click or goto anyway, so the recording keeps what the browser did (review, 2026-09-29, after a fix that aborted such calls left recordings without actions the page had taken). Every recording rule (tables, doc strings, fingerprints, tab attribution, discard-before-announce, proof replay) therefore holds for every provider without a line of vendor code. A vendor driver is only "run this prompt, with these MCP URLs, and report usage": about a hundred lines each.

Containment per vendor. No provider may use a shell, edit files or search the web; the agent gets the browser and Saffron's tools. Claude: tools: []. Codex: features.shell_tool and friends off, a read-only sandbox, MCP tools set to approve (under approval never, an unapproved MCP call is refused), and a two-line wrapper adds --ignore-user-config --ephemeral, because the SDK merges the user's ~/.codex/config.toml (their own playwright MCP server, plugins and hooks) into every run and has no option to skip it. The config's auth keys (cli_auth_credentials_store, forced_login_method, forced_chatgpt_workspace_id) are passed back as overrides, since a keyring login is only found with the first of them. Windows cannot spawn the wrapper without a shell, and running unwrapped would let the user's own browser server bypass the recorder, so there the provider refuses to start (decided 2026-09-29 after review). Cursor: tools: ["mcp"], settingSources: [], a throwaway store. Antigravity (measured on agy 1.2.13): MCP tools reach the model through one built-in, call_mcp_tool {ServerName, ToolName, Arguments}, and the model first reads each tool's schema from ~/.gemini/antigravity-cli/mcp/<server>/. Headless agy auto-denies MCP calls, since it cannot ask, unless an allow rule sits in the user's settings; a hook's allow and its permissionOverrides do not grant it. Two ways through were measured. --dangerously-skip-permissions with the hook as the only fence worked, but no longer failed closed: a hook that did not load would leave the agent unfenced. Chosen instead (with the user, 2026-09-29): saffron login antigravity adds two allow rules, mcp(saffron_browser/*) and mcp(saffron_tools/*), to ~/.gemini/antigravity-cli/settings.json, once and with every other setting kept, and sessions skip nothing. Those server names exist only in Saffron's sessions. The workspace's PreToolUse hook is the second fence: it allows call_mcp_tool to the two servers, and view_file or list_dir inside their schema folders (paths resolved, so .. does not pass; denying a guessed file name there sent the agent guessing parameters by trial calls), and denies the rest; --sandbox confines terminal commands besides. agy reads its own schema folder without a rule. Until the rules are in, a session with tools refuses to start and names the command; saffron author, which has no tools, needs none. The prompt goes in on stdin, where agy reads it when there is no -p (verified on 1.2.13): it carries resolved {env:VAR} values, and a command line is readable by every account on the machine. The first live Codex session used Playwright's browser_run_code_unsafe to read a file from the home directory; Playwright calls that tool RCE-equivalent.

Hidden from every provider, Claude included (decided 2026-09-29). The tool runs any JavaScript in the Playwright server's Node process, with the user's files and processes in reach. It is also a recording hazard: its result echoes the script as await (<code>)(page);, and the recorder files every Playwright statement line in a tool result as an action, so a scripted loop or branch records as its flattened body. Nothing depends on it: no committed recording or test uses it, and replay never calls MCP tools at all, so hiding it changes only how future recordings are made. One allowlist (AGENT_TOOLS in agent/tool-cap.ts) drives all three proxies (the gateway, the trace bridge and the capped stdio proxy, built by one agentToolProxy): a tool not on it is dropped from the list and a call to it by name is refused. An allowlist, because a denylist missed what PLAYWRIGHT_MCP_CAPS adds, which the user's own environment reaches Playwright MCP with (coordinate mouse, storage, cookies, routes). Older releases' plain browser_run_code stays out the same way. browser_evaluate stays visible: it runs through page.evaluate inside the page under test, the browser's sandbox, so it can read the app but not the host, and the evaluate() line it echoes records nothing. Agents use it to read computed values.

The allowlist holds only tools that record as an action or only read, so it also leaves out browser_navigate_back, browser_navigate_forward, browser_reload, browser_resize and browser_drop (review, 2026-09-29). An agent that opened the wrong link and went back recorded both clicks and not the way back, so replay left the page and failed the second click every run. browser_navigate reaches any page the agent would return to, and records as a goto.

Agents that batch. Codex's current models call tools from a script, so a refused saffron_assert and the saffron_done after it ran together, and the retry arrived after the run was final: an unverified proposal. Two changes, both for every provider: Saffron's refusals are MCP errors (isError), which stop the script, and saffron_done with success is refused, once, while an announced assertion step has no check. After both, the same Codex session fixed its assertion and filed a verified proposal. Once, not always: an agent that cannot express a check would otherwise burn its turns to the limit, and a second pass is caught anyway by the proof replay, which marks the recording unverified. A vendor that fails after saffron_done keeps the verdict it gave.

Packaging. @openai/codex-sdk brings a 316 MB native binary, so both SDKs are installed by the project itself, no dependency of saffron-ai (they were optional peer dependencies until a supply-chain scan counted them as ours), loaded on first use, with an install hint naming the supported version when one is missing, too old or of another major; agy is a system binary. Cost: these vendors report tokens but no price, so the report says the cost was not reported rather than showing $0.

2026-10-06: A browser matrix, parallel agents, and where the time went

A browser per agent session. Parallel recording was held because concurrent sessions burn money faster. It ships with a cap of 1 by default (agentWorkers), so nothing changes until a team raises it. The question was where a second session runs. Contexts in one browser looked cheaper, but the agent attaches to a whole browser over CDP: the orchestrator finds the page it acts on, watches for new tabs and guards dialogs across every context, and the agent's own tab list is the browser's. Two sessions in one browser would see each other's tabs. So each agent worker is an Orchestrator of its own, with its own browser and port, sharing the one agent provider (which keeps no state between sessions). Scenarios that share a recording (an outline's rows) stay on one worker in order, or two rows would record over each other.

A matrix is browsers in sequence, keyed by browser. Each browser runs the suite in turn, Chromium-family first: those can record and heal, and the replay-only engines then replay the proposals they filed, at zero tokens, instead of failing for want of a recording. Rows carry browser, and every place that keys results by scenario keys by browser too (the report fold, chronic scenarios, the dashboard's flips and standing). Rejected: suffixing the display name (Login [firefox]). Proposals bind to the display name (recordedFor), and an outline's rows are told apart by it, so a suffix would read a recording made in Chromium as another row's in Firefox. Evidence goes one folder deeper, per browser. The matrix refuses --shard: shard positions assume one row per scenario, and a CI matrix runs browsers side by side, which is the faster form anyway.

A device is a context option, not a recording property. Recordings stay browser- and size-neutral (roles, names, labels), so a desktop recording replays under --device "iPhone 15" and fails at the step where the page really differs. Considered: storing the profile in the cache and refusing a mismatch. That would turn every cross-device run into a re-record, when finding the step that breaks on a phone is the point. Firefox accepts a mobile profile and then times out on every click, so that pairing is refused at start.

Per-step cost stays per scenario. The replay shows what Saffron knows per step (retries, tab moves, the response each network check matched) and the agent's cost per scenario. Providers report usage per session; dividing it over the steps the agent announced would print precise-looking numbers that are guesses.

2026-10-21: One orchestrator, many mediums (the seam before API testing)

Two architecture reviews, one by Saffron's own agent and one external, reached the same verdict before M21: keep the modular monolith, no rewrite, and cut one seam before a second kind of test arrives. API testing and native mobile are real second mediums (new actions, new targets, their own recorder and replayer); accessibility, security and performance checks are new assertion kinds on a medium that exists, and need no architecture of their own. This entry records the seam, what it deliberately leaves for M21, and what was rejected.

The orchestrator owns the flow; a medium owns the performance. The orchestrator keeps everything every medium shares: whether the recording is fresh, which proposal is pending and whose it is, the Outline row rules, the escalation from replay to record or heal, the assertion guard that runs a failed Then as written, the proof replay and its one refine, duplicate-wording detection, secret masking, the proposal it files and the verdict. It drives the medium through two interfaces in src/orchestrator/medium.ts. A Medium starts and stops, says whether the agent can attach to it, clears a scenario's evidence, opens an Execution, and runs an agent session inside whatever it must hold back meanwhile. An Execution is one isolated performance of a scenario (the run, a proof, a pending proposal's replay): it replays a recording or a slice of one from where its sessions are, checks an assertion's actions as written, resets for a fresh recording, hands the agent what it needs to attach (attach), takes a snapshot, keeps or discards its trace, and closes. The browser medium (src/browser/medium.ts) is the code the orchestrator used to hold: contexts, sessions, tabs, downloads, dialogs, CDP, screenshots and traces, moved rather than rewritten, with sessions.ts, live-page.ts, fingerprint.ts and the agent's trace bridge (traced-browser.ts) beside it. The orchestrator imports nothing from Playwright. A test drives it through a fake medium with no browser and watches it record, prove, file and guard (src/orchestrator/medium.test.ts): what a second medium plugs into.

Why the seam went in before the second medium, against the usual rule. An interface with one implementation is normally refused here: the seam is cut where the second consumer pulls. It went in first because the second consumer's shape is known from M21 and the reviews (an HTTP execution has a cookie jar and named responses instead of pages and tabs, an agent attaches through an HTTP tool server instead of CDP, evidence is a request transcript instead of a screenshot), and because landing it alone, with every browser test green and a byte-identical recording format, is cheaper to review than landing it inside the API feature. The contract was shaped by that consumer, not by the browser: nothing in it names a page.

Named for what joins next. The recorded action shape is WebAction and Action is its union, so an HTTP family joins by adding to the union and to the replayer that owns it; validateCache already refuses an action name a runner does not know, so a new family needs no schema version. Config, scenario filters, run hooks, the quality model, the project status model and the CI run result types moved out of the CLI module, so nothing outside src/cli imports it: the CLI is a client of the runner, and a core package could be cut from it without a rewrite.

Left for M21, on purpose. Four things the reviews asked for are the API feature's own, and would be speculative before it: typed operation values (bodies hold numbers, booleans, objects, null and absent fields, and the string substitution that serves <param>, {data} and {env} must be re-proven for them); retries and proof replays that respect server-side writes (a proof is a second execution, and for a POST that is a second record, so the medium must say which actions may run again and the orchestrator must obey); the pinned contract as a freshness input (an unchanged scenario over a changed OpenAPI document must read as stale); and the agent-side seam (the agent session still parses Playwright code lines out of the browser tool's results, which no other medium will do; an HTTP medium records from explicit Saffron tools, the path saffron_assert and saffron_wait_api already take). Also left: the replay result types, which still carry the browser's activePage and live in src/replay, and the worker scheduling in the CLI, which a physical device will want exclusive, when mobile arrives and not before.

Rejected. A plugin loader or a package per medium: one team, one npm package, one agent prompt; a package the project installs itself already carries "different libraries" (the vendor SDKs use that pattern). Rewriting the browser code into the new shape: it moved with its comments and its tests, so the diff is the seam and nothing else. Changing AgentRunContext: the orchestrator's browser tests read traceContext, sessionPage and liveTab off it, and the API feature will decide what an attachment looks like for a medium with no pages; until then the browser's fields stay where every provider reads them.

2026-10-21: API testing v1 (M21), on the medium seam

The first second medium. What it settled, beyond the roadmap's direction:

The recording holds the request as sent, typed. A request is a method, a path, headers and a JSON body (JsonTemplate), not a string template substituted into JSON: a row or data value that stands where a number or boolean was sent is a typed leaf ({ "$value": "<seats>", "$type": "number" }), resolved then read as its type, so the recording never sends "3" for 3. Every pass that reads or rewrites an action's text (the row rules, the secret masker, the data and date passes) walks a body's strings through one helper (cache/json-template.ts), so a value in a body is found as a value in a URL is. Rejected: bodies as strings with tokens in them, which the reviews warned would substitute strings into JSON.

Captured values are tokens, written back at record time. An id a response returns is kept by extract (name to JSON Pointer) and reaches later steps as {captured:name}, in requests and in browser checks alike. The agent copies literal ids out of responses; the tool writes them back as the token before the action is recorded (tokenizeCaptured), so a recording never carries the id its recording run happened to get.

Checks are verified before they are recorded. saffron_assert kind status, field and schema run the check on the latest response, as replay will, and refuse one that does not hold: a recorded check is one that held at recording. The three checks read the latest response of the step's session, so a Then that needs a request sends it first, under its own step.

Writes never run twice. Replay retries only an action that may run again: a browser action, a GET, HEAD or OPTIONS, or a request the recording marks idempotent. A POST that times out fails the step. A proof replay is a second execution by design, so a scenario that creates records names them with {unique:...}, as the browser scenarios do, and each execution makes its own; cleanup stays the project's teardown. Rejected: skipping the proof for scenarios with writes, which would file unverified proposals.

The contract is pinned and part of freshness. saffron api pin writes the OpenAPI 3 document into the project; a Swagger page is an entry point whose document is pinned, never the page. A recording with a schema check carries the contract's SHA-256 (contracts), written by the medium (recordingFacts) and read by the freshness check with the data values: an unchanged scenario over a changed contract is stale, red without an agent. Rejected: a digest per check action, which would scatter one fact over every Then.

One replayer for the HTTP family, two mediums. http/replay.ts sends and checks; the browser replayer hands HTTP actions to it with the browser context's own request client (so an API call after a UI sign-in carries the session's cookies), and the HTTP medium runs it with clients of its own, one per As "..." session. @api tags a scenario for the HTTP medium; the CLI opens one API orchestrator beside each browser orchestrator and routes by tag. The agent gets the same Saffron tools in both: no browser tools in the API medium (cdpEndpoint absent), saffron_request and saffron_extract in both.

Left for later. Reusable named operations and fixtures shared between features (the roadmap's next steps), YAML documents and eject of API steps. (A browser matrix runs API scenarios once, not per pass: done the same week.)

Rendered after the run, never during it (added the same day). The owner asked for a request and response viewer like cypress-plugin-api, which draws each call inside the Cypress runner as the test runs, and which they had seen lag and freeze runs. Saffron's answer is the transcript: the medium keeps each exchange as text while the scenario runs (that is the whole cost), writes it once the verdict is in, and the report and the trace viewer render it afterwards from the file. api.transcript sets how much is kept ("on-failure" by default, "on" for green scenarios too, "off" for the lines without headers or bodies). Rejected: a live panel in a headed run, which would put the rendering back inside the execution.

2026-10-23: API tests, after their review

A review of the API branch found five things to fix before it ships and several to fix soon. The decisions:

A pointer is a field's address, written like the step. A request's extract pointer that names the row (/plans/Basic/id) was replayed as recorded, outside what row binding inspects, so another row passed on Basic's field. A whole segment spelling a column the step names is now that column (/plans/<plan>/id), resolved and escaped per segment at replay as a check's path is; any other row value in it binds the recording to its row (rowBound, reason "value"), so another row is refused, never recorded again in a loop.

A status is expected only where the step names it. The agent could resend a failing request with the status it got, as the tool itself suggested, and record a failure as expected. saffron_request now refuses an expectStatus that the step does not name (its code, its reason phrase, or a class and a code in it) before anything is sent, and suggests resending only when the step names that status.

The orchestrator owns freshness and the Then guard's trigger. Every replay goes through replayIn: a recording that no longer matches its scenario is stale without asking the medium, and a failure's kind is the recorded step's, so a medium that calls a failed Then an action cannot get it healed. The seam test proves both with a fake medium that would pass.

API concerns stay out of browser-only runs. A broken api.contract fails the schema checks that need it, not a browser run; the API tools and the contract's summary go to the agent only in a project that configures api or a scenario whose replay called the API.

Smaller guards. A request that gets no response reports the first line of Playwright's error, never its call log, which lists the request's headers. No request goes to a link-local address (a cloud's metadata service). OpenAPI 3.0's boolean exclusiveMinimum and exclusiveMaximum are read as draft 7 writes them, a 204 the contract gives no body passes a schema check when it has none, and the API base URL's path prefix is stripped before an operation is looked up. api pin stops after 30 seconds and refuses a document over 20 MB, and api.transcript takes only its three values. Heal, refine and seeded prompts in the API medium speak of requests and name the values replay kept; the IDE plugins pick screenshots by kind and offer a replay for API rows.

2026-10-23: Several APIs, named in the config

The owner asked how a project with more than one API defines their base URLs. One api.baseURL served the default API; a second could only be a full URL in the recording, which fixed its host, and the guard that sends {env:...} values to the API's origin alone refused its secrets.

A request names its API, never its host. api.services maps a name to a base URL and a contract, and a request carries service with a path relative to that base. The agent passes the name to saffron_request; a full URL it writes under a known base URL is recorded as that API and a relative path, so no host reaches a recording. --api-url overrides a base URL per run, the way --base-url does for the app. Rejected: hosts in the recording with a rewrite table at run time, which would make every recording depend on the table's spelling.

Every rule that was per API stays per API. The {env:...} guard allows the origin of the request's own service; expectSchema reads the service of the exchange it checks; recordingFacts records the contracts of the services a recording called, and each pinned file is active for freshness. A broken contract keeps the run's existing behavior: it stops an API run and fails only schema checks in a browser run.

HTTP recordings are version 6. saffron-ai 1.1.0 knows no HTTP actions and would refuse such a file as malformed, which reads as "record it again". At version 6 it says to update instead. The service field adds no version of its own: no released runner reads HTTP recordings at all.

2026-10-24: Fixtures: steps a feature runs once (M21)

A feature whose every scenario needs a signed-in client or a seeded customer repeated the login or the creation in each scenario, and the proof replay repeated it again. Background runs per scenario, and a StepSet reuses step text and its recording, not a run. The smallest thing that removes the repetition is a block of steps the runner performs once per feature and whose captured values every scenario reads.

Fixture: is a .saffron block, not a tagged Background. Gherkin gives Background no tags, so @once Background: is not Gherkin and would break the .feature promise. Fixture: <name> follows StepSet: exactly: the pre-processor rewrites it to Scenario: on the same line, so the Gherkin parser and every line number are unchanged, and a .feature that uses it gets the rename advice StepSet gives. One fixture per file, at the feature level, named, without Examples, invoking StepSets freely. It is parsed as a scenario of its own (ParsedScenario.isFixture) and hung on each of the feature's scenarios (ParsedScenario.fixture), so a worker that holds a scenario knows what must run first without a second parse. The feature must be @api: the HTTP medium shares values, not cookie jars, and a browser fixture whose cookies the scenarios inherit is a different design (a shared storage state), left out for now.

A fixture is recorded, replayed, healed and proved like a scenario. Its recording lives under .saffron/cache beside the feature's others, named after the fixture; a stale one is re-recorded; its Then steps are never healed; its proposal is proved by a replay of its own, with its own {unique:...} values; saffron accept takes it like any other. The step index seeds its steps across features as it does any step. Nothing in the recording format changes: the fixture's recording is an ordinary one, and a scenario's recording names its values as {captured:name}, which a runner without fixtures fails to resolve (red, never green) after having refused the Fixture: line itself. cacheSchemaVersion stays at 7.

Its captured values are read-only seeds. The orchestrator runs the fixture before the first scenario of its feature and keeps the values its extracts produced (a green replay's, or the proof's when it recorded). Every execution of a scenario starts with a copy of them in vars: the run, its proof replays, a pending proposal's replay and the agent's view (captured, named in the prompt as the fixture's). A scenario's own extract under the same name shadows the copy and never reaches the fixture or another scenario. A credential the fixture captured stays a credential in every scenario's execution and agent session (ExecutionReplayOptions.credentials, AgentRunContext.credentials), so a token from a fixture's login is masked everywhere one from the scenario would be.

Once per run, shared between workers. fixtureRuns is a map the CLI shares between every Orchestrator of a run, like filedThisRun: the first scenario of a feature to ask starts the fixture, the rest await the same promise, so the replay workers, the agent workers and the API orchestrator beside each browser one run it once. It is keyed by the fixture's cache file and by whether the agent may run: the parallel phase runs with the agent disabled, and a fixture that is red only for want of an agent runs again in the agent phase. A shard is a run of its own and runs the fixture itself. A proof replay of a scenario never re-runs the fixture: it has no {unique:...} of the scenario's execution, and the values it produced exist on the server for any client.

A fixture's verdict is every scenario's floor. Red (a failed step, a stale recording with no agent, a recording that did not prove) makes every scenario of the feature red with the fixture's error and its steps skipped; a fixture that recorded a proposal pending review makes a green scenario yellow, naming the fixture's proposal. The fixture itself has no row in the report: the scenarios carry its verdict, and the run's list of pending proposals names its file.

Named operations are StepSets. A StepSet step that sends a request and keeps a value with extract is a reusable named operation: invoked by name from any scenario or fixture, recorded once, seeded from the index everywhere else, its result addressed as {captured:name}. The roadmap item needs documentation, not a new construct.

Left out. Fixtures shared between features (put the steps in a StepSet and invoke it from each feature's fixture: it records once per feature). Browser fixtures and inherited cookies. A fixture's values reaching a Background. Making a scenario's recording stale when the fixture's capture names change: a renamed name fails the scenario at the unknown token, and the next run heals it.