Design log

Distilled from the founding design discussion (2026-07-02). This is the "why" behind the architecture; the "how" is in the README and the code.

The idea

Tests are Gherkin .feature files. First run: an AI agent executes the scenario in a real browser and records a deterministic cache. Every later run Saffron replays the cache in a real browser at zero tokens. When a cached step fails at runtime, the agent takes over mid-run, adapts, and files a cache proposal the user accepts or rejects: snapshot-testing UX for user journeys. Reports state adaptation, AI calls, tokens, and cost honestly.

Competitive landscape (refreshed 2026-09-19)

Checked against each vendor's own documentation on the date above. Vendor docs establish what a product does, not how reliable it is, so nothing here is a claim about quality. Where a row compares versions, the version is named, because Stagehand's answer changed between v3 and v4.

Tool	What it already does	What is still different about Saffron
Playwright Test Agents (planner, generator, healer)	The planner writes a Markdown plan, the generator turns it into ordinary Playwright tests, the healer patches failing ones and re-runs them. Generated tests run with no model involved, so they also replay at zero tokens.	This is the strongest baseline: "an agent writes Playwright and we review its PR". The healer is a separate maintenance step, not a mid-run fallback; the artifact is code rather than a plan with a protected assertion block; there is no built-in report of what was adapted and what it cost. Saffron does not beat it on replay cost.
Playwright-BDD	Runs Gherkin through the Playwright runner, with an established ecosystem behind it.	Gherkin plus Playwright is already available. Step definitions are hand-written glue; Saffron's case is removing that glue for supported browser flows, and it has to justify a separate runtime to do it.
Karate Agent (Karate Labs)	Reusable flows are .js scripts that run at native speed with no LLM tokens; in autonomous mode an LLM drives an observe-decide-act loop and recovers failures.	Deterministic execution plus AI recovery is not unique to Saffron. Karate Agent is commercial (licence file, evaluation on request; core Karate is open source), and its docs describe no accept or reject step for what a recovery changed.
Stagehand v3 (Browserbase)	Two caches: a Browserbase server-side one, and a local cacheDir whose files are portable. The docs recommend committing that directory so CI does not run inference on first execution.	A cache committed to git is therefore not unique either. Stagehand is an automation SDK, not a test runner: no distinction between actions and assertions, no review gate, and after a DOM change the documented remedy is clearing the cache by hand.
Stagehand v4 (Browserbase)	Caching is managed server-side only: no cache files, and the cache option has no effect with a local browser.	Here the cache is not a reviewable artifact at all, and it requires the vendor's browser.
mabl	Selector auto-heal. Per the retirement notice the external review cites, agentic runtime recovery was introduced and then disabled across workspaces on 2026-08-03 after early-access results. We could not open that page when refreshing this table (bot check), so treat the date and wording as the review's, not ours.	A caution about exactly the workflow Saffron relies on. The notice does not publish enough to infer a technical cause. It is the reason proof replay, sacred assertions and the review gate are mechanical rather than advisory here.
Momentic, testRigor	Natural-language tests with self-healing locators, delivered as SaaS.	Not re-verified in this refresh. The earlier version of this table said tools in this group use AI on every run; that was a claim about a whole category without product-specific evidence, and it has been removed.

Sources: Playwright Test Agents · Playwright-BDD · Karate Agent overview · Stagehand v3 caching · Stagehand v4 caching · mabl retirement notice

What is not unique: zero-token replay (generated Playwright has it, Karate flows have it, Stagehand's cache has it), AI recovery at runtime (Karate Agent), and a cache you can commit (Stagehand v3).

Positioning: Saffron's niche is the combination, as a workflow. Gherkin is the source of truth with no step glue. The recording is a plan a reviewer can read, with assertions the agent is mechanically unable to weaken. A heal happens mid-run, is proof-replayed at zero AI, and lands as a proposal with an action-level diff that someone accepts or rejects. Reports say what was adapted and what it cost. Against ordinary generated Playwright this is not a cost advantage on replay; the honest comparison is total maintenance: authoring, repair review, flake investigation and missed defects.

Locked design decisions

1. Cache = structured JSON action list per scenario. Semantic targets (ARIA role + accessible name, label/text/testId, CSS fallbacks) and <param> placeholders so one cache serves all Examples rows of a Scenario Outline. Raw MCP tool-call replay was rejected: Playwright MCP element refs are snapshot-scoped and unstable across page loads.
2. Assertions are never healed. Then steps compile to expect* actions; the agent may fix how a run reaches an assertion, never what it checks. Enforced mechanically, not just by prompt (see below).
3. Three result states. Green = cached pass. Yellow = passed-with-adaptation (proposal pending review). Red = failed. Yellow exits 0 with a warning; a --strict flag is on the roadmap.
4. Adaptations propose two diffs: the cache diff and a .feature step rewrite (saffron accept --with-feature-edit), so documented truth tracks executable truth.
5. Saffron's own replayer, on Playwright's browser engine. The cache is JSON, not a generated script. The Playwright MCP server only runs in agent mode; agent and replayer share one browser over CDP so healing continues from the exact failed page state.
6. Claude-first behind AgentProvider. Multi-provider later.
7. Free CLI, open-core SaaS path. Originally planned as Apache-2.0 from day one; amended 2026-07-07 (see "Distribution" below): the public release is free to use but closed-source, with open-sourcing as a later, deliberate step.

Key rationale

• Regression-masking is the existential risk. An agent whose goal is "reach the end anyway" will route around real bugs. A test tool that masks regressions is worse than none. This drove decisions 2 and 3.
• Verified in practice: during MVP verification the agent did try to rewrite a failing assertion to a "close enough" check (renamed heading) and reported success. Prompt rules alone did not stop it. Hence the mechanical guards: heal merges never take agent-recorded assertion actions, and when the failed step is an assertion the orchestrator re-executes the cached assertion as written on the live page: red if it doesn't pass.
• Flake vs drift: bounded retries with backoff run before any AI escalation; the agent's first job on takeover is diagnosis (UI drift → adapt; app defect → fail with explanation), not goal-seeking.
• Adaptation breaks the Gherkin contract unless feature files evolve with the cache: hence structured featureEdits in proposals.

Execution flow

Locator maintenance: propagation over a registry (decided 2026-07-03)

Locator duplication across caches is real (a shared login button appears in every scenario; one rename = N heals). The classic Page-Object-Model answer , a shared locator registry with indirection: was considered and rejected for now: agent-recorded caches would need entity resolution to decide two targets are "the same element", and a wrong registry heal silently fans out to every referencing scenario, inverting Saffron's blast-radius guarantee.

Instead: same-target heal propagation (saffron accept --propagate). When an accepted heal rewrites a locator, every other cache whose target has the identical semantic core (role/name/nameRegex/label/text/testId, fallback CSS excluded) gets the same rewrite. Identical core = identical Playwright query, so the identity proof is exact, not heuristic; fan-out is printed per scenario at accept time. Caches stay self-contained. An opt-in named-locator registry remains a candidate for the review-UI era.

Step reuse: Level 1 before Level 2 (decided 2026-07-05)

Step-level sharing (the Cucumber step-definition model) survives the objection that killed the locator registry: step text is author-declared identity: two scenarios using the same text is a human assertion of sameness, not an inference. The spectrum: Level 0 (per-scenario caches + propagation) → Level 1 (shipped): a derived step index seeds new recordings ("record the gaps"), storage stays per-scenario, divergence is instrumented → Level 2 (candidate): a shared step library as source of truth, which additionally needs polymorphic-step (heal ping-pong) detection, per-scenario overrides, and affected-scenario re-verification. Level 1 captures the dominant economics (new-scenario recording cost scales with novel steps only) without shared-mutation risk; its divergence data decides whether Level 2 is warranted. Non-prefix (island) seeding shipped 2026-07-05: seeded islands replay zero-AI, bounded agent segments (max 3 per scenario) record the gaps, agent overshoot past a segment boundary is adopted instead of replayed twice. No remaining Level-1 scope gaps: the open question is Level 2, decided by divergence data. Quoted-value generalization shipped 2026-07-05: lookups fall back to a quote-generalized pattern and substitute differing args by exact-field equality only (substring occurrences and ambiguous duplicate args bail). Data tables shipped 2026-07-05: tables are part of the staleness contract; 2-column key/value tables parameterize recorded values as <table:key> (value edits replay free, key edits re-record), and seeding matches on table keys.

Wording duplication: surface the derived index, prove duplicates by behavior (decided 2026-07-05 · SHIPPED 2026-07-06)

The Cucumber pain: the same step exists under two wordings ("I navigate to…" / "I visit…"). In Saffron the pain is sharper, exact text is cache identity, so wording drift is an index miss that re-records at token cost, but the architecture also enables a fix Cucumber cannot have.

Challenged and rejected: a manually-curated step registry. The step index derived from committed caches already is the registry; a second, hand-maintained one duplicates truth and rots. Decided instead (scoped as M8): surface the index (saffron steps); behavior-proven duplicate detection: two wordings whose recordings produce identical action lists (actionsEqual) are a duplicate by definition, not by string similarity, and trigger a mechanical rename proposal through the existing feature-edit machinery; and saffron author: AI drafts feature files from prose with the index injected as required vocabulary, which makes authored files maximally seedable (authoring quality becomes recording cheapness). IDE completion (VS Code extension over .saffron/cache) deferred to the public-release era. Prevention converges wording before recording; detection converges it after, at the cost of exactly one recording.

Step sets: parse-time expansion, and the .saffron superset dialect (decided 2026-07-05 · SHIPPED 2026-07-06)

The second Cucumber pain: the same 5–6 prelude steps pasted into every scenario of a file (Background only covers scenario starts, file-wide). Decision: named step sets, StepSet: blocks invoked by a reference step (scoped as M9).

Why expansion beats the alternatives:

• Composite steps (one declarative step recording N actions, the Gherkin-purist answer) fork the vocabulary: the composite can't share recordings with scenarios that use the fine-grained steps individually.
• Runtime references (cache points at a shared recording) reintroduce the shared-mutation risk that deferred Level 2.
• Parse-time expansion (like Background merging) preserves each sub-step's exact text identity, so caching, Level-1 seeding, heal propagation, and staleness work with zero engine changes. A set edit makes its scenarios honestly stale; re-recording is almost fully seeded. Step sets are parser sugar with outsized leverage, a feature uniquely cheap in Saffron because identity is text.

Why a new file extension (initially challenged, then adopted with conditions): the user's .saffron proposal is a superset dialect, normal Gherkin plus the StepSet: keyword, which is the TypeScript playbook (.ts = .js + keywords, honest new extension). A keyword genuinely reads better than a tag convention, and the deciding argument: a .feature file containing StepSet: would break standard parsers while claiming to be standard: worse than an extension that declares itself. The pre-processor rewrites StepSet: lines to Scenario: same-line (line numbers preserved) and reuses @cucumber/gherkin (no fork); docs ship the files.associations / .gitattributes snippets for day-one highlighting. v1: unique project-wide set names; tables/doc strings/params allowed inside sets; no nesting; sets never run standalone but their steps join the seeding vocabulary.

Global step sets (discussed 2026-07-06): a dedicated .safset extension for library files was proposed and rejected, research showed cross-file resolution already shipped with M9 (the registry is project-wide by construction), so the extension would split the dialect into two grammars and double the IDE-tooling bill for a distinction the registry never needed. Adopted instead: the .d.ts-style naming convention: a sets-only features/shared.steps.saffron library file , documented across the guide/FEATURES/HOW-TO-USE. Deferred until real pain appears: header-less library files, a stepSets config path for sets outside the features tree, and namespacing for very large suites (the unique-name parse error will surface that pain visibly).

Syntax locked from the user's worked example (2026-07-06): definition StepSet: <name> (colon, block keyword), invocation StepSet <name> (no colon, step position), grammatically consistent with Gherkin (block keywords carry colons, step keywords don't). Challenged and resolved: (a) the colon-typo hazard at an invocation site (an accidental block would swallow the scenario's tail) is caught by the project-wide unique-name rule: the accidental block duplicates an existing set name and parse-errors with a "remove the colon to invoke" hint; a distinct Use keyword was considered and rejected to keep one symmetric spelling. (b) Sets are .saffron-exclusive, the earlier * steps: legal-Gherkin fallback for plain .feature files was dropped (two spellings, one feature = teaching cost); adopting sets in an existing .feature is a pure file rename, since the dialect is a superset. (c) Assertions inside sets inherit position-dependent strictness automatically after expansion: mid-scenario invocation → checkpoint semantics; final invocation → the set's trailing Thens join the always-strict final assertion block. (d) The set shape convention: guard step first, exit assertion last, every invocation carries its own precondition and success check. (e) The true implementation cost is the edit-back path: featureEdits need step provenance ({file, line, fromSet}); editing a set definition fixes every caller at once (callers go stale, re-record seeded).

IDE integration: vocabulary to humans via LSP, to AI via MCP (decided · SHIPPED 2026-07-06)

Design for step completion while writing .saffron files (scoped as M10). Key decisions from the discussion:

• Vocabulary = parsed files ∪ cache index, not caches alone, a teammate's written-but-unrecorded steps must complete too, or parallel authors re-diverge. Caches contribute status, which is the differentiator: completion items badge each suggestion ✓ recorded / ⚠ divergent / ○ unrecorded: the editor shows which steps already replay for free. No Cucumber tool can offer this because no Cucumber tool knows what a step costs.
• Diagnostics over completion: the author who confidently types a fresh wording never opens the popup, so completion alone can't kill the duplication pain. A near-duplicate squiggle with a rename quick-fix is the prevention layer; M8's behavior-proven detection remains the cure after recording.
• Two consumers, one index: humans get an LSP (portable to JetBrains/Neovim; go-to-definition on StepSet invocations, hover with recorded actions and usage counts); AI assistants get saffron mcp (they can't read popups: they need the vocabulary as context; the interactive twin of saffron author).
• Cheap first rungs: saffron steps --snippets (native VS Code snippet completion, zero extension code) and --json ship long before the language server; the extension itself waits for the .saffron parser (M9) and publishes with M6.
• Shipped 2026-07-06: the VS Code extension went first (plain-JS providers, sibling repo); saffron lsp followed same-day for JetBrains/Neovim: LSP4IJ covers Community editions, so the "paid IDEs only" limitation of JetBrains' native LSP API doesn't bite. Near-duplicate threshold calibrated at 0.7 token-Jaccard: catches typo-level drift, leaves synonym-level duplicates to the runner's behavior-proven rename proposals (lexical similarity cannot prove synonymy; identical recorded actions can).
• Rejected: reusing the official Cucumber extension, it indexes glue-code step definitions, which Saffron deliberately lacks.

Distribution: free first, open-source later (decided 2026-07-07)

The public release ships as a free, closed-source npm package, the Claude Code model: npm install, configure ANTHROPIC_API_KEY (or use a Claude Code login), free for any use including commercial, no redistribution or modification. Mechanics: the published artifact is a single bundled+minified CLI (no sourcemaps, files whitelist), under the Saffron Free Use License v1.0. Relicensing from the earlier Apache-2.0 plan was clean because nothing had ever been distributed and there are no outside contributors: the door that closes at first publish, which is why the decision was made now. Open-sourcing is a scheduled later milestone, not abandoned: shipped JS is legally protected but never secret, and the community value of open source remains part of the plan once the tool has traction. Feedback runs through GitHub issues on the public site repository (saffron-ai) until the runner repo itself goes public.

Secrets: seven leak surfaces, two tiers (decided · Tier 1 SHIPPED 2026-07-07)

"Read secrets from env vars" is the easy fifth of the problem. Saffron persists more artifacts than Cucumber, so a password in a feature file reaches seven places: the file, the committed cache (the sneaky one), proposals, the agent conversation, reports, history.jsonl, and terminal errors. The design is therefore a pipeline property:

• Syntax {env:VAR}, not <env:VAR>, the <param> family resolves early (prompts, resolvedText) and participates in seeding resolvability checks; env tokens there would leak wider and break seeding. The {…} family resolves lazily at action execution, where {date+N} already lives: one resolver covers every replay path.
• Masking is scoped to referenced values: only vars the scenarios actually use are masked/checked; the environment is never enumerated into params or prompts.
• Tier 1 accepts recording-time exposure (the agent must type the real value once; replays are AI-free): documented with a rotate- staging-credentials recommendation. Tier 2 removes even that via a Saffron-owned MCP proxy substituting tokens inward and masking echoes outward; deferred, not dropped. Inherent residual either way: secrets the page displays appear in accessibility snapshots.
• .env auto-loaded from the project root (real env wins), decided for dev/CI parity.

Roadmap

Maintained in ROADMAP.md: milestones M1–M10 with statuses and decision gates (run history, vocabulary completion, the gated Level-2 shared step library, assertion policy v2, runtime/providers, public release, SaaS layer, step vocabulary & authoring, step sets & the .saffron dialect, IDE integration). This log stays the record of decided designs; the roadmap is the single source of truth for pending work.

2026-07-19: Network-aware steps (vocabulary, not keywords)

The user raised the classic UI-test synchronization need: wait for a backend call (e.g. a 200/201, or a polling endpoint reporting READY) instead of a page signal, and asked whether Saffron should introduce keywords. Decision (challenged and locked): no new Gherkin keywords. StepSet: earned syntax because it is structural; a network wait is behavioral, and behavior belongs in the recorded action vocabulary behind plain-prose steps: keywords would start a glue DSL and break the "any Gherkin is valid" superset story. Two actions instead: waitForResponse (healable synchronization) and expectResponse (assertion: sacred under the existing kind system, so API contract checks can never be healed). Matchers: URL regex (path-only, volatile IDs must be patterns: the nameRegex lesson applied to URLs), method, status exact-or-class, body regex (v1 includes body matching by user decision, for polling waits). Replay uses a context-level rolling response log (covers new tabs); waits match responses observed since the previous step began: covers both "click and wait" in one step and a standalone wait right after its trigger: while expectResponse audits the whole scenario. Recording is verify-first via the MCP's network log. Live proof: both example scenarios recorded verified on the first attempt, including the body-pattern polling wait.

2026-09-13: Ship guidance for agents with the package

Most test code is now written by AI assistants, and Saffron's economics depend on wording discipline those assistants do not know by default. The user proposed shipping a skill with the npm package. Decisions:

• Format and placement: an Agent Skill (SKILL.md + references/) in a skills/ directory at the package root: the convention the ecosystem converged on for packages (skills-npm, TanStack Intent). .agents/, .claude/, .cursor/, .github/ skill folders are the consumer project's discovery locations, not something a package should contain.
• Installation is an explicit command, never a postinstall hook: npm cannot legitimately write into a consumer's project at install time. saffron init copies the skill into the agent directories (default .claude/skills + .agents/skills), idempotently.
• No shipped AGENTS.md: it is the consumer's file. init upserts a small marked block (load the skill, reuse the vocabulary, never hand-edit caches, secrets as tokens) into AGENTS.md (created) and CLAUDE.md (only if present).
• MCP registration belongs in the same command: a bundled MCP server is inert until the host knows about it, so init merges a saffron entry into each host's project-scoped config (.mcp.json, .cursor/mcp.json, .vscode/mcp.json) rather than leaving a manual step. Codex has no project-level file and gets a printed one-liner.
• The skill states the runner's rules, not its internals: exact step text as cache identity, sacred Then placed last, no volatile or secret literals, StepSet syntax (including the bare-name mistake), network waits and page furniture phrasings, the run → review → accept loop, and an anti-patterns table. Progressive disclosure per the spec: a short SKILL.md, detail in reference sheets.