Changelog

All notable changes to Saffron, in one place: the saffron-ai runner (npm), the VS Code extension (ChathurangaJayasinghe.saffron-vscode) and the JetBrains plugin (ai.saffron.jetbrains). Dates are publish dates.

Unreleased

Editors (JetBrains plugin 0.2.2)

• LSP4IJ is an optional dependency. A first upload of 0.2.2 was rejected by the Marketplace verifier on 2026.3 EAP because LSP4IJ had no release for that build yet, and a mandatory dependency that cannot be resolved blocks the whole plugin. Highlighting, run configurations and the tool window now work without it; completion, go-to-definition, hover and diagnostics switch on when LSP4IJ is installed, and a one-time notification says so when it is missing. The presence check uses no plugin-manager API, so it stays clean as that API turns internal.

0.7.1

Released 2026-09-20.

Runner

• expectUrl checks the active page, not any open tab. A URL sitting in a leftover tab could satisfy the assertion for the wrong reason. A recording made before tab openers were marked, with a slow popup and retries: 0, can now fail where it used to pass: re-record it, or leave the default retry, which re-follows the newest tab.
• A changed table value only replays if the recording really uses it. Matching key sets used to be enough, so a recording that still carried the old value as a literal, or never referenced the cell, stayed "valid" and typed stale data. The old value is looked for only where a value can live (typed, matched or named text, with tokens removed), so an empty cell or a short value like "on" does not force a needless re-record.
• Propagation never rewrites an assertion, on either side: assertion steps and assertion actions are skipped when a fix is extracted and when it is applied, the step wording must match, and a locator inside a frame is a different locator. When one heal sends the same locator to more than one place, nothing propagates and saffron accept says so.
• saffron prune and saffron status respect features outside the configured directory. A scenario run as saffron run e2e/login.feature had its recordings listed as orphans. Feature files named by existing recordings are now inspected too, and one that fails to parse blocks the prune like any other.
• Step reuse no longer brings back a stale table value. Changing alice to bob correctly invalidated the scenario's cache, and step reuse then seeded the same actions straight back, matching on table keys alone: a verified proposal at zero AI calls whose table said bob while its actions still typed and asserted alice. Seeds now pass the same rule as cache validation, and a step with no table of its own is refused when it carries another step's old value as a literal. The old value is looked for in locators built from data as well (a test id like user-alice, fallback selectors, a frame, a response URL pattern), not only in typed and named text: a fill can be parameterized while the assertion beside it still targets the old user.
• A feature edit cannot land on the wrong step. Edits address steps by index, so a step inserted while a proposal waited shifted every index after it, and an edit meant for "I click login" rewrote the new step. saffron accept --with-feature-edit now compares the scenario with the steps the proposal was recorded against and refuses before anything is consumed: the file, the cache and the proposal stay as they were.
• A skipped feature edit no longer reaches the cache. When one edit applied, the accepted cache was given the new text of every edit, including a Background edit that was correctly skipped, so the feature and its cache disagreed at once. Only edits that landed are synced. Two edits that reach the same StepSet line through two invocations are applied once when they agree, and both refused when they do not, naming the two wordings: the line can only say one thing.
• saffron status parses each feature file once. The vocabulary and the orphan scan each parsed the whole project again; both now reuse what status already read. IDE panels call status on every refresh.

Editors (VS Code extension 0.2.5)

• Completion badges and duplicate-step diagnostics follow cache creation and deletion. Only changes to existing JSON files were watched, so accepting a first recording or pruning one left the vocabulary stale until something else triggered a rebuild. Recordings are watched for all three events (and only recordings, not every JSON file), and the watcher is disposed with the extension.
• Vocabulary is per project in a multi-root workspace. Step sets, wording and recorded badges were built once for the whole workspace, so project B's Login step set could answer go-to-definition in project A, and a step recorded in A showed as recorded in B.
• The inline diff button on a proposal row works. An inline action passes the tree item, not the file path the row click passes; both forms are accepted, and the project guard applies to both.

0.7.0

Released 2026-09-19.

New: saffron diff, saffron prune, a configurable pollIntervalMs, and a configuration reference in the docs. The rest of this release is correctness work from several rounds of external review.

Upgrading. Three fixes can change what an existing suite does, all in the direction of telling the truth:

• A scenario that was green because replay quietly clicked a near-match ("Delete all orders" for a missing "Delete"), or because a Then step never checked anything, now fails and heals or asks to be re-recorded.
• Feature files in subdirectories get a new cache directory name (a short digest of their path is appended), so their scenarios record again once. Files directly under the features root are unaffected. saffron prune lists the old directories afterwards.
• saffron status --json gains cacheState and orphans; nothing was removed.

Runner

Correctness fixes from an external review of the runner. Four of these could let a run look green or yellow when the goal was never checked.

• Secrets could reach your feature files. Machine-applicable feature edits were the one agent output that skipped {env:VAR} masking, so saffron accept --with-feature-edit could write a resolved password into a scenario. Feature edits, the narrative and the suggested edit are now masked everywhere they are stored or reported.

• An assertion that recorded nothing now fails. A Then step with an empty action list used to replay as a pass, so a scenario could be green without its verdict ever being checked. Replay fails such a step, and the recording is filed unverified. This can turn an existing green scenario red: that assertion was never being checked. Re-record it.

• Negative assertions consider every recorded locator. expectNotVisible returned as soon as one candidate was hidden, so a real failure could hide behind a missing primary locator while a fallback was visible.

• Recorded names match exactly first. A step recorded against "Save" could click "Save draft" when it came first in the DOM. Substring matching remains the fallback for reading the page. The agent is no longer told to shorten names.

• Cache files are unique per scenario. The cache directory now includes the feature file's directories, so two same-named files in different folders stop overwriting each other's recordings. Flat layouts are unchanged. A run refuses to start if two scenarios would still share a file.

• Suspicious recordings are saved unverified. The check ran after the proposal was written, so the file said verified while the run said otherwise, and accept --all took it.

• An unverified proposal shows the proof replay's timeline, not the agent session's, so the report says which step could not replay instead of marking every step passed.

• A step that changes between an action and an assertion is stale. Cache matching compared the step's words but not its keyword, so turning When I see the total into Then I see the total kept the old recording and replayed the action instead of checking anything.

• Waiting for a new tab happens where a tab opens. Every click paid a 150ms wait, which taxed scenarios that never use tabs and still lost the race with a slower popup. The proof replay now records which action opens a tab, and only that action waits, for as long as any other action may take. A five-step login scenario replays in about a third of the time.

• saffron status distinguishes a valid cache from a stale one. It reported cached by asking whether a file existed, so a scenario whose text had changed still looked recorded. Each scenario now carries cacheState: valid, stale, invalid or missing, and the printed view marks stale ones.

• saffron lsp no longer reports a valid step set as defined twice in projects whose path contains a space: it compared a file system path with a percent-encoded URL path.

• saffron diff shows what a proposal changes: the committed recording against the proposed one, action by action, with a rewritten action shown as one before-and-after pair and changed assertions counted separately. The narrative is the agent's account of what it did; this is what a reviewer is approving. It also appears in the HTML report and in both IDE panels.

• An assertion step must contain an assertion. The earlier guard only rejected an empty action list, so a Then that recorded a waitFor and nothing else still passed.

• A control that is found but cannot be used no longer hands the action to a different control. With a disabled "Save" next to an enabled "Save draft", replay clicked the draft button. An interaction now stops at the candidate that matched, so drift goes to the failure and healing path instead of quietly doing something else.

• Negative assertions check every match of every locator. Checking only the first match let a hidden first element vouch for a visible second one. A locator that errors no longer counts as proof of absence.

• Secrets are discovered through Examples rows. A {env:VAR} reaching a table cell or doc string through a placeholder was invisible to masking, so it could survive into recordings, reports and feature edits.

• Cache identity follows the path, not its spelling. admin/login.saffron and admin-login.saffron produced the same cache directory; nested files now carry a digest of their path. The collision check also looks at the whole project, because two features run separately never met before.

• A recording that says a tab opens must see one. The marker was advisory: when no tab appeared the timeout was swallowed and the next assertion ran against the old page. Tab openers are also captured while recording now, not only during the proof replay, which could miss them.

• An interaction never substitutes a different control. A recorded name that no longer matches exactly could still act on any control containing it, so a missing "Delete" could click "Delete all orders" and the run stayed green. Interactions now require the exact name; assertions and captures keep the substring fallback, because reading the page is not acting on it. A scenario whose control was renamed now fails and heals instead of passing.

• The proposal diff compares actions, not their descriptions. Two actions that read the same could differ where it counts: dropping a response body check or repointing a selector from #save to #delete both reported no change. Actions are compared field by field, and a rewritten action now lists exactly which fields moved.

• Repeated steps stay separate in the diff. Steps were matched by their wording, so a scenario that adds the same item twice showed no change when the proposal dropped one of them.

• A popup that arrives late is pinned on the action that opened it. The opener marker waited for the next recorded action, so a slow window was blamed on whatever the agent did afterwards, and replay then waited for a tab that action never opens. Only an interaction can be credited: a wait or an assertion recorded between the click and the popup no longer takes it, and the marker now follows the agent's actual tool calls instead of inferring them. Any action marked as opening a tab waits for one, not only clicks. A popup nobody can be credited with is dropped rather than passed to a later action: one that arrives while the agent is idle and the last interaction is too old, one that lands during a call that failed or could have opened it itself, and one the agent opened with a new tab. Only known observation tools (a snapshot, a wait, listing tabs) may vouch for the click before them, and failed tool calls are now drained so they cannot hold an attribution open.

• saffron prune removes recordings nothing owns. Deleting a scenario or a feature file left its cache behind, where it stayed in git and no reviewer could tell it from a live one. saffron prune lists every orphaned cache and proposal with the reason, deletes nothing without --yes, and --check exits 1 for CI. Ownership follows the path, so a stale or unreadable cache that a live scenario claims is kept, and while any feature file fails to parse prune refuses to judge anything. saffron status --json carries the same list for IDE panels.

• pollIntervalMs is configurable (default 100ms, minimum 10). It sets how often a polled assertion re-checks the page inside actionTimeoutMs: lower reacts sooner, higher costs less on a busy CI box.

Docs

• A configuration reference, grouped by what you are tuning (timing, project, runtime, the agent, gates) with every option's default, replacing the paragraph that listed the keys in prose. It ends with how timing actually works: Playwright's actionability wait, then polled assertions, then action retries, and why there is no whole-scenario retry.

Editors (JetBrains plugin 0.2.2, VS Code extension 0.2.4)

• JetBrains plugin: Accept Selected with nothing ticked accepted every proposal. Blank paths mean --all to the runner, so the deliberate review workflow could be bypassed by clicking the wrong button. It now does nothing and says so; bulk acceptance stays its own action.
• Both IDEs list recordings no scenario owns any more: a JetBrains Orphans tab (grouped by cache and proposal, double-click to open one, Remove All after a confirmation) and a VS Code Orphaned recordings section with the same action in its title bar. Both name the reason, and both say plainly when the installed runner is too old to report them rather than claiming there are none. prune is also a JetBrains run configuration command, where listing is the default.
• JetBrains plugin: the Proposals tab could show an old diff beside a new narrative. Diffs were cached by file name, and a new run overwrites a proposal under the same name, so Accept could promote something the reviewer never read. A refresh now invalidates every diff, a late result from before the refresh is discarded, and the cache lives on the UI thread.
• JetBrains plugin: a path with a space stays one path. features/order checkout.saffron became two arguments, even for right-clicked and ticked files. Paths are parsed like a command line, and what the plugin writes itself is quoted to match.
• VS Code extension: Run selected ran project B's files from project A in a multi-root workspace, with A's config and install. Selections are now grouped by owning folder, one run per project, from the Explorer too. The status-backed sections name the project they show, and Select project switches it, instead of silently using the first folder. Switching clears those sections at once rather than when the new status arrives: rows left on screen stayed clickable, so a proposal ticked in project A could be accepted in project B when both had the same relative file. Ticks are bound to the project that listed them, and accept, reject, run tagged and prune refuse to act while a status is still loading.
• VS Code extension: the proposal diff could not start on Windows, where Node refuses to launch a .cmd without a shell. Status and diff now share one launcher: the installed CLI runs under node with a plain argument array, and only the npx fallback goes through cmd.exe, fully escaped.
• Both IDEs show the proposal diff: JetBrains under the narrative in the Proposals tab, VS Code by opening the diff as a document when a proposal is clicked or from its inline action.
• VS Code extension: runs the CLI as a task with an argument array and the owning folder's working directory, instead of writing a command line into a shared terminal. A file name with a dollar sign or a backtick is no longer shell syntax, and a file from a second workspace folder runs against that folder's config and install.

0.6.0

Released 2026-09-18.

Runner

• Multi-tab scenarios. Plain prose, no keyword: "switch back to the first tab", "close the preview tab", "open a new tab at the admin page". The agent's tab operations during recording are cached as switchTab, closeTab and newTab and replay in the same order. New tabs opened by clicks are still followed automatically, and an explicit switch back is no longer undone by that follow. Replay gives a click a moment for its popup to appear and waits for a tab before switching to it.
• When a pending proposal fails its zero-AI replay and the agent records again, the run now says which step failed and why.

Editors

• VS Code extension 0.2.3: the Marketplace listing is named "Saffron", its short description fits without being cut off, and the overview introduces the tool and how to get started before listing features.

0.5.7

Released 2026-09-17.

Runner

• saffron mcp gains a project_status tool (the saffron status --json overview) next to search_steps and list_step_sets, and reports the real package version.

Agents

• Claude Code plugin: claude-plugin/ packages the language server, the MCP server and the skill as one plugin (/plugin marketplace add s-chathuranga-j/saffron-claude-plugin, /plugin install saffron@saffron). saffron lsp now adopts the workspace root the editor sends in initialize, so plugin-launched servers index the right project.
• The bundled skill covers saffron status, accepting or rejecting chosen proposals, subscription versus API-key cost reporting, Scenario Outline recording, and the IDE integrations; the AGENTS.md managed block points at saffron status. Run npx saffron init again to refresh both in a project.

0.5.6

Released 2026-09-17.

Runner

• saffron status no longer lists the same step with different quoted arguments ("<password>", "{env:SAUCE_PASSWORD}", a literal) as duplicate wordings; steps are compared by their quote-generalised pattern.

0.5.5

Released 2026-09-17.

Runner

• saffron status reports duplicate wordings by behaviour: different step texts whose recorded actions are identical once the step's own arguments are masked. The previous wording-similarity list flagged "enter X in the username field" against "enter Y in the password field", which are different steps.

0.5.4

Released 2026-09-17.

Runner

• saffron status [--json]: the project overview that IDE panels and dashboards read: feature files and scenarios with cache state, tags, pending proposals with narratives, last run, history, vocabulary health (divergent and near-duplicate steps) and the effective config.
• saffron accept and saffron reject take several proposal files at once.

Editors

• JetBrains plugin 0.2.1 tool window gains Proposals, Tags, Health and Dashboard tabs; VS Code extension 0.2.2 gains Tags, Proposals and Health sections and an in-editor dashboard. All read saffron status --json.

0.5.3

Released 2026-09-17.

Runner

• Cost is reported by how the run was paid for. On an API key the dollar figure is what was billed. On a Claude subscription (a Claude Code login with no ANTHROPIC_API_KEY) the summary line and the report headline the 5-hour window before and after the run, for example 12% → 19%, and label the dollars as the API-equivalent. The weekly window is not shown.
• Reports and the summary line name the models that answered during the run.

Editors

• saffron lsp (JetBrains and other LSP editors) now offers the StepSet: definition keyword as a snippet when you type step s, Steps or stepset, and the StepSet invocation keyword, even in a project with no sets yet. Before, the keyword only appeared once a set existed, and the definition form never did (the VS Code extension already had both).
• JetBrains plugin 0.2.0: a Saffron run configuration type (Run/Debug Configurations → Add New Configuration → Saffron: run with files, folders, tags, replay-only, headed, re-record and extra arguments; report; accept), right-click → Run on .saffron files and folders, and a Saffron tool window listing feature files with scenario counts, search, Run Selected / Run All, Open Report, Accept All Proposals, last-run totals and pending proposals. Runs use the project's own saffron-ai install with the login shell's environment.
• VS Code extension 0.2.0: right-click → Saffron: Run (also replay only, headed, re-record) on .saffron files and folders, a play button in the editor title, and Command Palette entries for run all, open the latest report and accept all proposals. Commands run in a "Saffron" terminal from the workspace folder.
• VS Code extension 0.2.1: a Saffron side panel: an activity bar view listing feature files with their scenarios (click to jump), inline Run and Replay-only buttons, checkboxes with Run selected and Run all, replay-only and headed toggles that apply to every run from the extension, the last run's totals with the report one click away, and the proposals pending review with Accept all.

0.5.2

Released 2026-09-15.

Runner

• Scenario Outlines record correctly: an assertion recorded from the page's longer sentence now becomes the <placeholder> outright when it contains the row's value (substring replacement used to leave a residue that no other row could satisfy), and rows after the first replay the pending proposal at zero AI instead of recording again and overwriting it. When the proposal does not fit a row, the run is red with a clear message rather than silently replacing another row's recording.
• The agent is told to record a Then step's quoted text exactly as expected text, never the page's longer sentence.
• A Given step or StepSet guard that describes a state the page is already in is no longer reported as an adaptation. The saucedemo example's checkout set used to show a warning on its first recording for this.
• Yellow scenarios are labelled "pending review" in the run summary and the HTML report instead of "adapted", since a proposal is not necessarily an adaptation.

Editors

• {env:VAR} tokens are navigable: Ctrl/Cmd+click (go to definition) opens the variable in .env, or .env.example when no .env exists; hover says where it is defined and never shows the value. In the VS Code extension and in saffron lsp for JetBrains and other editors.

0.5.1

Released 2026-09-14.

Runner

• Example suite: .saffron files only, with StepSets invoked both at the start of scenarios and in the middle ("Recover from a locked-out login", and a checkout set that carries a data table).

Editors

• Grammar: quoted arguments, <placeholders> and {env:…} / {date…} tokens get their own scopes, so editors colour them like Cucumber parameters instead of the whole step line. Ships in the VS Code extension, the JetBrains plugin and the package's TextMate bundle.

0.5.0

Released 2026-09-14.

Runner

• Guided first run: saffron init offers to install the Saucedemo example suite (--examples / --no-examples): five short scenarios against the public demo shop, {env:VAR} credentials with a .env.example, a Scenario Outline, a data table, and a StepSet in a library file, plus a matching saffron.config.json with retries: 1. Only installs into an empty features/ directory. Documented as "Your first run".

0.4.2

Released 2026-09-13.

Runner

• saffron run --rerecord: discard the selected scenarios' caches and let the agent record them fresh, seeded from the other recordings. The sanctioned fix for a wrong recording instead of editing cache JSON.
• The recorder understands every way the agent types (fill, pressSequentially, type) and drops the clearing fill('') that precedes typing into the same field. This was the cause of caches with an empty value for a step that passes one.
• A fill recorded with an empty value on a step that passes a value marks the proposal UNVERIFIED with the reason, even if the proof replay passed.
• saffron accept --all skips UNVERIFIED proposals; --include-unverified overrides, and accepting a single file still works with a warning.
• Retries are visible: a retried ×N chip per step in the report and a ↻ line in the terminal. Retries stay per action with backoff; they are not scenario re-runs.
• Scenario Outline rows display as (example N); report, terminal and docs use plain punctuation throughout.
• Reports and history carry the real package version.

Docs and site

• Documentation moved to the site's Docs section, generated from the runner's markdown; the previous docs page became Features; the landing page was trimmed.
• Assertion policy documented on the Features page; cache lifecycle gained the staleness and wrong-recording stages.

0.4.1

Released 2026-09-13.

Runner

• JetBrains highlighting without the plugin: the package ships the TextMate grammar at textmate/saffron for Settings → Editor → TextMate Bundles.
• saffron lsp run from a terminal prints the editor setup steps instead of waiting silently for a client.

Editors

• JetBrains plugin 0.1.0 (Marketplace listing 34240): .saffron file type, bundled grammar, saffron lsp wired through LSP4IJ automatically.
• JetBrains plugin 0.1.1: the no-package fallback runs npx -p saffron-ai saffron lsp (the bare saffron name resolves an unrelated npm package); the install hint shows once per project; dark theme file icon; verified on 2024.2 through 2026.2.
• VS Code extension 0.1.3: plain punctuation in README, hover and diagnostic messages.

0.4.0

Released 2026-09-13.

Runner

• Agent skill shipped in the package (skills/saffron/SKILL.md plus reference sheets) following the Agent Skills specification.
• saffron init: installs the skill into .claude/skills and .agents/skills (Copilot and Cursor on request), registers the saffron mcp server in .mcp.json, .cursor/mcp.json and .vscode/mcp.json, appends a managed block to AGENTS.md (and CLAUDE.md when present), and scaffolds saffron.config.json, features/ and the .gitignore entry.
• Multi-tag --filter (OR semantics, comma-separated or repeated); scenario tags are written to the report JSON and history.jsonl so external tools can key results by a test-case tag.
• saffron --version reports the real package version.

Editors

• VS Code extension 0.1.1: case- and spacing-tolerant StepSet keyword completion plus a definition snippet.
• VS Code extension 0.1.2: warning and quick fix when a set name is written without the StepSet keyword.

0.3.1

Released 2026-08-15.

• saffron --version no longer reports a hardcoded value.

0.3.0

Released 2026-08-12.

Runner

• Network-aware steps in plain prose: waitForResponse (healable) blocks until a matching backend response, expectResponse (assertion, never healed) audits that one occurred. Matchers: URL pattern, method, status (exact or class), optional body pattern for polling waits. Waits match from the triggering step onward, so "click and wait" never races.
• Recording ergonomics: native dialogs, drag-and-drop and iframes record live. The dialog auto-dismiss was the orchestrator's own launching Playwright client, not the MCP; agent sessions now keep dialogs visible.
• Positive assertions recorded without frame scoping fall back to searching child frames; poll-based assertions sweep all candidates under one deadline.

Editors

• VS Code extension 0.1.0 published to the Marketplace.

0.2.0

Released 2026-07-19.

Runner

• Secrets: {env:VAR} tokens resolve from the environment (or a git-ignored .env) at replay; caches, proposals, reports and history are masked back to the token; missing variables fail fast by name.
• Action vocabulary for real-world pages: handleDialog, uploadFile, dragTo, and iframe-scoped targets, replaying on Chromium, Firefox and WebKit.
• Cross-browser replay (--browser firefox|webkit), parallel replay workers (--workers N), and a cheaper model for heal sessions (--heal-model).

0.1.0

Released 2026-07-10.

First public release, free for any use under the Saffron Free Use License.

• Gherkin-native runner: an AI agent records each scenario once; later runs replay with plain Playwright at zero tokens; runtime healing files reviewable proposals; assertions are never healed.
• Verified proposals (zero-AI proof replay), honest cost reporting, run history and trends, chronic-scenario detection.
• Level-1 step reuse: new recordings seed from every step already recorded.
• Data vocabulary: <param> placeholders, key/value and record tables, doc strings, {date±N} templates, captured values.
• Assertion policy: strict (default) or adaptable-mid; the final assertion block is always strict.
• Step sets and the .saffron dialect (StepSet:), project-wide library files, parse-time expansion.
• Step vocabulary tooling: saffron steps, saffron mcp, saffron lsp, saffron author; behavior-proven duplicate wording detection.