Changelog
46 entries. The runner on npm, the VS Code extension and the JetBrains plugin, in one place.
Runner
- An Outline row no longer passes on another row's data in three more
ways: a check naming a column that no step of the recording writes ("the
<plan>plan shows ...") binds the recording to its row, a row's number is found where the page shows it grouped or with a decimal comma ("Up to $1,000" for 1000), and a request body's number is taken for a constant only where the body itself writes the column. A style check's property is a value a row can hold. A check run in the session a column names (As "<who>") follows that column; one run in another session (As "admin") does not, and a step writing the column counts only when it comes before the check in the same session. A row's number is also found negative ("−$1,000") and grouped by an apostrophe or a no-break space ("CHF 1'000"). - An amount check fails when the target shows another currency than the
captured amounts, and captures in two currencies are refused. A symbol
is compared with the codes it can stand for: "$5" may be USD 5 or CAD 5,
never NOK 5 or EUR 5, "€" is EUR, and a dollar its letters name is that
dollar ("US$", "CA$", "A$", "R$" and others); every capture is compared
with every other. A lowercase word that is also a code ("Top 10",
"150 eur") is compared with codes only. A URL check reads a
path whole past
-,.,_,~and+, and past:before a letter ("/checkout" is not "/checkout-failed" or "/checkout_failed"; a host before its port, "localhost:3000", still matches), and a word starts as it ends: after a letter and a separator it is part of another ("paid" is not in "/orders/not-paid"; in a query "+" and a space still separate words, so "shoes" is in "?q=red+shoes"), a number also after a digit ("42" is not in "/1-42"), while an id-prefixed slug keeps its words ("blue-shirt" is in "/products/42-blue-shirt") and "example.com" is in "shop.example.com". A doubled separator ("/checkout--failed") is one, in replay and in specssaffron ejectwrites. A committed check of a path the app shows with a slug after it ("/products/42" on "/products/42-blue-shirt"), or of one word inside a slug ("shirt" or "US" in "/en-US/cart"), fails after upgrading: record it again with the whole slug, or check the id another way. A currency code is read beside the number, capitals first, so "Top up NOK 150" is in NOK, and "NET 150 EUR", "150 eur" and "EUR (150.00)" are in euros. Text inside<noscript>, or inside an SVG definition never drawn where it stands, is not shown text, whether the check's target is the container or that element itself. - API tests: a group's
samecheck compares values exactly (two int64 payment references a digit apart differ, as do 0.1 and 0.10000000000000001, and a number and a string of its digits), a number in a body that JSON cannot hold as written is refused, in any spelling (9007199254740993,9007199254740993.0,9.007199254740993e15,0.10000000000000001; it would be sent rounded to another value: send it as a string). A row or capture that held such a number and passed before is refused after upgrading, adifferscheck whose row value is empty is refused, an expected decimal with more digits than a number holds is refused, and a group's contract check ties the recording to the contract, soapi pinre-records it. - A provider's daily quota, "exhausted your capacity" and "hit your
limit" stop the run's AI sessions as other usage limits do. A key a
vendor quotes with its middle starred or elided (
sk-...XwVu) is masked, every Cursor error masks CURSOR_API_KEY, including one the SDK throws, and an Antigravity error that ends in JSON names its message. Averifyscenario path that leaves the project through a link is refused. - Specs
saffron ejectwrites open web pages only (neverdata:,file:orjavascript:) and upload only the project's own files (no dotfiles, nothing outside the project, no link to a dotfile or out of the project, no hard link, no folders), as replay does; they refuse adifferswhose value a blank row or capture empties, and a body number they cannot send exactly. An Examples cell holding such a number marks its test fixme instead of stopping the eject. - A
"""or a triple backtick at the start of a description line no longer opens a doc string, which left everyStepSetline after it unread: as in Gherkin, a doc string opens only after a step in a scenario, Background, StepSet or Fixture, never in a Feature, Rule or Examples description, in the runner and the language server alike. - An API that answers with a refused connection in its body or a field (its own database's) is a failure, never "not run, environment missing": only a request that got no response, or the browser's own navigation error, says nothing answered. The dashboard's recent failures pass over reds the app did not cause when telling whether a scenario still fails and for how long.
- Recording: a credential an API handed out during the session is masked
in everything the browser tools show the agent (a token in a page URL
was shown whole, and an earlier value of a credential that changed); a
page the agent opens with an
{env:...}value in its URL must be on the app's origin or one the scenario writes, a URL with no scheme read as the browser opens it; code a check runs throughbrowser_evaluatethat clicks, submits, sets a value, navigates or sends a writing request is refused, as a click in a check is; with a browser already running (transport.browseror atransport.module), recordings run one at a time, as two sessions on one browser closed each other's pages. - A password or user name in a base URL is masked wherever a URL quotes
it (
https://qa:***@staging.test): the report's errors, the run history, proposals,status,diff, bundles, traces and the API transcript, and the agent is never shown it. Only there: the same word elsewhere in a message is left as it is. - A local notation beside the number (
100 kr,Rs. 500,5,00 zł) is a currency of its own in an amount check, so captures in "kr" fail a total shown in euros. A URL check reads a percent-encoded letter beyond a separator as a letter ("/checkout" is not in "/checkout-%C3%A9chou%C3%A9"). - A proposal or recording the run cannot read (no permission, a folder in
its place) is listed as unreadable instead of stopping
status,diff,accept --alland the run's closing summary. The editor flags a{data:...}reference only where the runner reads it (a step, its table or doc string), never in a title, a description or a comment. - A Then recorded again under the same words that now expects another
value ("Pending" where it checked "Paid", a status 500 where it expected
201, another JSON value or URL), or reads another element named the same
way (test id "status" become "badge"), is held for review as a weakened
check (a row or data token holding the same value is the same): a stale re-record no longer replaces a check that just
failed. A heal that changed nothing is red, never a proposal, also with
--no-verify. The usage an agent session spent stays counted when its proof replay cannot start. - An Outline row's short value (a word under 5 characters with no digit,
such as "dev") is written as its
<column>only where it stands alone: inside "devices panel" it stays as recorded, so another row no longer checks a text never on the page. A secret Playwright cut short with "…" in a trace line, or a captured credential cut short in a message, is masked. - Replay: a "not shown" check looks inside the app's own iframes too (its origin, or one it writes, shown on the page), as a "shown" check does, so an error drawn in one no longer passes both; another origin's frame (a chat widget's "Close") and a hidden one (a silent sign-in renewal) are not searched. A "not shown", count, "differs" or "URL does not contain" check, and a response check, passes only once the requests the last action started have answered and the passing state has held for half a second: a list still growing, a placeholder drawn during a refresh or a row gone for a moment no longer passes. A long-poll the page opened earlier is not waited for. A response check fails when the very request it matched (its URL and method, and the body or JSON it checks) was answered again with another status (a POST answered 200 and then 500); a refetch, a child path or a GraphQL query on the same endpoint is another request.
accept --step-setrefuses a StepSet that ran in a named browser session (As "seller") and a proposal that fails an Examples row: filed for reuse, its steps would seed other sessions' or rows' steps. A Nordic price ("150,- kr", "150:- kr") keeps its currency. A base URL's Basic header an API echoes is masked; a URL in a query beside an email is not taken for a password; step text accept writes into the feature file keeps its URLs whole.- Recording: a capture's target is read as replay reads it before it is recorded, and a value it does not show is refused, so a later check never compares with another element; a status the step names exactly ("answer 404") is expected exactly, never as its class, and a status check names a status the step states; a step performed again drops its requests' responses too, so a later check never reads a dropped one; a response check is verified with the step's table values, as replay reads it.
- Config: a key that names a known one misspelt (
maxCostUSD,stict) is refused, naming the key meant, and any other unknown key is said; acapabilitiesvalue in quotes, an unknownsnapshotModeand ahookTimeoutMsof 0 are refused; an emptySAFFRON_ENVkeeps the config'senv.status --jsonand the MCPproject_statusmask a password in any URL and a header value written whole.run --planjudges a pending proposal as the run does: a stale one is a recording, one filed on another machine is not replayed, a row-bound one is refused. The MCPverifytool says when the run did not finish cleanly (interrupted, scenarios not run, a failed hook). - A heal whose agent session could not start keeps the replay's failure, never "not run"; a fixture that failed in the app fails the scenarios behind it, never "environment missing"; the usage earlier agent sessions of a recording spent stays counted when a later one fails.
- A password in the base URL, or a user name alone (
https://TOKEN@host), is masked in the report, the run history and bundles. A run with--filteror of part of the features folder is partial in the history, so trends compare suite runs only. A red the app did not cause never makes a scenario flaky on the dashboard. - A CSV data file with a quote never closed is refused, naming the row, instead of reading every line after it as one field. The language server no longer stalls on a line holding a long run of spaces.
- Replay lets the first candidate that finds the recorded element decide a
check, shown or not: a hidden recorded element no longer passes a text,
visible, value or enabled check on a fallback elsewhere, a list check
(
every {data:...}) no longer passes on a fallback or another frame listing more, and a watched message counts only on the element watched, never on a log line elsewhere. A currency-first value ($5.00) in a pattern no longer matches its negative (-$5.00). - A refused API request's reply to the agent masks a credential an API handed out in the URL, as the reply to a request that answered does.
- Restoring a column over a secret no longer splices a short env value
into a word (
devin/devices); a row value an API request's path holds URL-encoded is written back as its column, as in a browser URL. - A red the agent cannot fix (an unreadable recording, a row refused, the committed recording failing while another row's proposal waits) counts as a failure in the report and the run summary, never as "needs recording" with a command that would change nothing.
- The HTML report sets the reds the app did not cause apart from
failures: a stale recording in a run with
--no-agent, a missing environment, an agent that could not start or a run over its budget. They get their own not run group under the failures (needs recording when that is the only reason), a ○ on the run map, and a failed count that says how many more did not run and why. Every scenario that needs recording names the others, with onesaffron runcommand for their feature files. The run map counts them per feature file, and the recurring failure themes leave them out (run history now keeps the mark). They are still red, and the run still exits 1. Found in real-world use, where one edit to a shared StepSet made 45 of 48 reds stale recordings. - A request group's step seeds another scenario that sends the same
requests with its own values: the same table header, request names and
rows, each cell read through
<row:column>, and so does a group check that reads each response's own row. Before, a group step was never seeded, so every scenario recorded it with the agent. - A StepSet accepted on its own (
accept --step-set) after a scenario's recording was made seeds the steps that come from that StepSet over the scenario's older recording, with noseeded from another recording:note; the same words written straight in a scenario keep its own. saffron status --jsonlists the StepSets each proposal recorded (stepSets), for an editor to offeraccept --step-set.saffron ejectexports a count check as Playwright's count of shown elements,expect(locator.filter({ visible: true })).toHaveCount(n), the number as replay reads it (010is 10).- A pattern check that fails where the pattern writes an optional part right after a value says the value is matched whole and how to write the part instead.
- A currency code after a number counts as a price in a locator's name only after two digits or more: "Section 4 SEK" is no price.
Editors (VS Code extension 0.8.1, JetBrains plugin 0.8.1)
- Both editors offer
saffron export(the review bundle), a draft pull request with the pending proposals (export --branch, after asking for a branch and confirming),saffron eject(after asking, for the open file or every one) andaccept --step-setfor one ticked proposal's StepSet, from the Proposals view's menu (VS Code) or toolbar (JetBrains).
- An Outline row no longer passes on another row's data in three more
ways: a check naming a column that no step of the recording writes ("the
Runner
Upgrading from 1.3.2:
Recordings with a request group (several API requests sent at once) are schema version 19, and recordings with a count check use an action 1.3.2 does not know: 1.3.2 refuses both rather than replaying them wrong. Upgrade every machine that replays them, CI included.
Two new notes hold a proposal from
accept --all:check weakened:(aThenrecorded again with a looser check than the committed one) anddata not followed:(a step that names a{data:...}value its recording keeps as text). Accept such a proposal by its file after reading why.While recording, each browser check the agent writes is run on its page by replay's own engine, and refused when it does not hold there. A recording that used to fail its proof on such a check now fixes it before the proof.
A scenario recorded again because its recording is stale seeds the steps it did not change from its own recording, never from another scenario's newer one.
A downloaded file's check resolves a
{captured:...}value, while recording and at replay, so "the calendar file contains the booking number{captured:bookingNumber}" checks this run's booking; one that nothing captured fails saying so. Before, replay checked for the token as text and the proof failed every file.A date the scenario writes as a token stays that token when the agent types it glued to a time (
20261111T140000Zfor{date+7:YYYYMMDD}), and a pattern that writes the time after it ({date+7:YYYYMMDD}T[0-9]{6}) matches it. A time written as optional ((?:T[0-9]{6})?) still ends the date, so the step writes it as required. A date left as text in the format the scenario's own token writes getsdate not following the clock:and is held fromaccept --all.A short data value the step names (
{data:party.size}, 2) is written back as its reference where it stands alone once in a check's or fill's value, a text pattern or an amount ("Table for 2 people","price" * 2), never in a selector, URL or path, nor a digit of regex syntax, a time or a date. A step whose recording still holds a value it names as text, or a list's items one by one, gets a newdata not followed:note and is held fromaccept --all. A price in a locator's name (Info Lunch SEK 150) is avolatile locator:.The language server sees a StepSet's steps filed with
accept --step-set(.saffron/step-sets) as recorded when they change.The plugins' keyword tables carry each language's Feature, Background and Rule, so a file in another Gherkin language ends a scenario at them.
A
StepSet:orFixture:in a file written in another Gherkin language (# language: de) defines its set or fixture: it was read as the EnglishScenario:, which that language does not know, so no set was found. A StepSet is called in a language with no*step too (Slovenian), and the# language:line is read as Gherkin reads it:# language: de # Deutschis English to both.The language server reads an
As "..."session through Background and Rule headers in the file's own language, and ends one at aFixture:, as the VS Code extension does.A file the agent uploads while recording is found in the project when its path is relative, wherever the run was started:
saffron -p app runfrom a monorepo's root refuseddemo-app/sample-upload.txtas outside the project.The benchmark's recording measure counts every note held from
accept --all, and names the scenarios not verified on the first attempt or held, with why.A browser check the agent writes is run on its page by replay's own engine before it is recorded, with replay's values, captures and clock, and refused with replay's reason when it does not hold: a value not whole in its text or an element not shown fails while recording, not at the proof.
A
Thenrecorded again under the same words with a looser check than the committed one gets acheck weakened:note and is held fromaccept --all: a value it no longer reads (the same value written another way still counts), a pattern's or name pattern's lost^or$, a text it checks less of, a kind of check it no longer makes or covers, a JSON field it no longer compares, an element it no longer names.--rerecordkeeps the recording it removes for this comparison.A scenario recorded again because its recording is stale seeds each step it did not change from its own committed recording where that fits and the page shows what it acts on, never from another scenario's newer one (a shared StepSet's update brought those in). A step that replayed another scenario's recording is named in a
seeded from another recording:note.A check of an element that runs out of time (shown, text, pattern, value, style, attribute) says what replay saw last: nothing matching the target, an element there but not shown, or the text or value it showed, with secrets masked before it is cut.
A count check: "3 rooms should be listed" is one check of how many shown elements the target finds in the page (or the frame it names), from a number, a
<column>or a{data:...}value; a hidden element never counts, and none is written as "not shown".saffron ejectdoes not export it yet, and saffron-ai 1.3.2 refuses a recording with one as an action it does not know, never replaying it wrong.The report page's headline counts and cost are this run's own after a run of named scenarios, with the number of rows kept from earlier runs beside them; before, it added up the kept rows' results and AI calls.
Several API requests at once, each with its own values, and checks over all their responses: an idempotency test that registers one charge from several requests at the same moment. A step's table names the requests, a header row and a row per request (
When I send these registrations concurrently as "registrations":); the request is recorded once and sent once per row, every row before any answers, each row's cell written as<row:column>, so editing a cell needs no recording again; a<row:column>naming no column of the table is refused rather than sent as text, and the group's name may come from an Examples column. Every response stays with its row whatever order it arrives in, a request that gets none is kept and fails the step once all have settled, and a group is never sent again.Thensteps check every response, those of one status, a named one, at least some, or all of them sharing a field's value (all 200 responses should have the same "/paymentReference"); a response's own cell in a check is its row's. The common checks are exact phrases recorded with no agent. A failure lists each request with its status and why it failed. "The response" after a group fails instead of reading one of them. Recordings with a group are schema version 19: saffron-ai 1.3.2 and earlier refuse them.saffron ejectdoes not export a group yet.saffron accept --propagatefixes a healed API request everywhere it is sent, as it already did for a locator. Scenarios that share a step's wording hold copies of its request; when a heal rewrites that request (a field the API now requires, a renamed path), accepting it with--propagaterewrites every other cache that sends the same request under the same step text, so one heal fixes them all instead of each scenario healing on its own run.--propagategoes on the accept itself: without it, accept names the caches it left as they are, which heal on their own run. Either way it prints what the request was and is now. A fix is not copied when it writes a value of the healed scenario's own (a table cell, a doc string, its Outline row), adds a<column>or{captured:...}value, drops a value the request extracted, sends the request to another API, or when the heal kept the request as it was in one place and rewrote it in another; accept says which. A step whose heal added or removed an action is not copied either.saffron acceptwithout--propagateno longer tells you to re-run it with--propagate: the proposal is accepted by then, and a re-run had nothing left to fix. It says the other caches were left to heal on their own run.
Editors (VS Code extension 0.8.0, JetBrains plugin 0.8.0)
- Both editors mark a proposal with a
check weakened:ordata not followed:note, or one that fails an Examples row, as held from Accept All, assaffron accept --allskips it. JetBrains asks before Accept All runs, as VS Code does. - In a file written in another Gherkin language (
# language: de), Right-click Run reads that language's Feature, Background and Rule headers (JetBrains), and VS Code's completion, recording badges, hovers and wording hints read its step keywords and headers, a keyword with no space of its own ("假如") included. - A
Fixture:block is a block of its own: it ends the StepSet above it (VS Code), and Right-click Run inside it runs no scenario (JetBrains). - VS Code shows the steps a StepSet's proposal filed for reuse
(
.saffron/step-sets) as recorded, and no longer marksExamples: {data: roles}as an error when a description line and then a table follow it. - JetBrains passes a run's files after
--, so a file whose name starts with a dash is a file, never an option.
Runner
Upgrading from 1.3.1:
- A URL check (
expectUrl, contains) reads its text as a whole part of the URL, and a path (text with a/) outside the query, unless the text names a query (?,=,&):/dashboardno longer holds on/login?next=%2Fdashboard, nor/orders/42on/orders/421. A word with no slash (shoes) is still found in/search?q=red+shoes, but not in a query value that is a path (dashboardin/login?next=/dashboard), nor in a query value whose encoded#or&makes it look like a fragment or another field (/dashboardin?next=%2F%23%2Fdashboard). A check that the URL does not hold a text still reads it anywhere:erroris in?errorCode=5. A text that names a query is the page's own: its?is where the URL's query starts and a field it names is one of the URL's own, so/dashboard?tab=settingsandtab=settingsno longer hold on/login?next=%2Fdashboard%3Ftab%3Dsettings, nor/dashboardon an OAuth callback's#state=%2Fdashboard. A fragment's fields are the page's own too:#tab=settings,/app#tab=settingsandtab=settingshold on/app#tab=settings. A recording that passed only that way turns red; the exported test checks the same. Examples: {data: roles}(a space, a capitalData) is an error naming the file and line, where the Outline used to run no row at all. The language server marks it while it is typed.- An expected text never matches inside a longer word or number, whether
a row value or written as it is (SAF-197):
"$10"no longer holds on$100,"1 item"on11 items, nor a text targetSavedon "You have unsaved changes". A role's name may still have grown a suffix. A number written in the step may be followed by a zero decimal part ("$10"in$10.00, never$10.50,₹1,00,000or2.0.1); a row value keeps its decimals whole (<id>1 is not in1.00), and neither is found before a digit in an inline tag (Order 1<b>2</b>,$10<sup>.50</sup>). A text ending in a number's point or comma takes no more digits ("Refund of $10."is not inRefund of $10.50), and a number is not found in its negative ("5.00"is not in-5.00or−5.00;"5"is still inpages 3-5). A value that starts or ends in Chinese, Japanese, Thai, Lao, Khmer or Myanmar script is not bounded there, as those scripts put no space between words, but for a negating prefix:"成功"is not in不成功nor没有成功,"保存"not in无法保存. Only a negation right before the value is read: a short value in a sentence that negates it further back can still be found. A recording that passed only that way turns red, and the agent records the whole word next time. - An expected text never runs across a block, a table cell or a
<br>(SAF-198):"15.00"no longer holds on the cells1and5.00, nor"Qty 10"on the itemsQty 1and0 left in stock. Items of a flex row or a grid read as one line, a space apart:"Total: $10"holds on the itemsTotal:and$10. A value written on several lines still matches the lines the page draws. - A style or attribute check needs its element shown (SAF-199): "the
error is shown in red" no longer holds on an error that is not on
screen, nor on a shown fallback when the recorded element is on the page
but hidden. A check of what hides an element (
display,visibility,opacity,content-visibility;hidden,aria-hidden,inert) still holds on a hidden one, and so does one of aclass, astyle,openordata-stateattribute, or a height, width, transform or clip: how many apps hide an element (d-none, a closed dialog).
The exported Playwright tests check text, blocks and styles the same way (SAF-197, SAF-198, SAF-199).
- A text check of an amount written with its currency (
$5.00,€5) no longer holds on the negative amount (-$5.00,−€5), as5.00already did not on-5.00. A URL check of a number takes no decimal part (amount=10is not in?amount=10.50, nor/orders/42in/orders/42.5), and a letter or digit the URL writes percent-encoded belongs to the word beside it (name=Josis not in?name=Jos%C3%A9, norid=42in?id=42%35). The exported test checks the same.
Fixed, from a review of the runner:
A credential stays masked wherever it travels: a fixture's token that a later capture replaced, or one inside an extracted object, in the scenarios the fixture seeds; and a replaced token in what the agent reads while recording, which it used to show. An exported test masks a secret as a URL encodes it (
%2B,%2F,%3D) and in the errors of its JSON checks, as replay does.A
differscheck on an object holding an integer beyond JavaScript's precision no longer passes against the same object a capture kept.The dashboard's suite health no longer reads 100% while a scenario failed (473 of 475 reads 99.58%).
The language server no longer marks
Examples: {data: roles}as an error inside a doc string, nor when a description comes before its table: the runner reads both.No false green from a fallback selector or an iframe: the first locator that finds a shown element decides a check, so a fallback that happens to show the expected text no longer vouches for the recorded element that failed it. A recorded element that is gone, or hidden for a check other than a style or attribute check, still gives way to its fallback.
No false green from an API check whose pattern matches any text: a field, header or
untilpattern such as.*,^or an empty one is refused while recording, as page checks already were.No Outline row passes on another row's data when the app writes a row value without its accents (
Joséasstatus-jose), or when a column names a section without quotes (the <plan> card shows "Popular").A captured id of one or two characters is recorded as its token where a path segment or field names its capture (
/orders/42fororder), so replay no longer reads the record made while recording. A field that names another id (productId: 1) is left as it is.A secret in an API query that holds
/together with+,=or&(a base64 key) is masked in errors, transcripts, reports and the agent's replies.A page or API message cut for a report is cut with
…, so a secret across the cut is masked too, also when the message carries it URL-encoded.A request method taken from a column (
a <verb> request) is kept as sent, which binds the recording to its row, instead of an invalid recording made again on every run.An integer above 2^53 extracted from a response (a snowflake id) keeps the digits the API sent, instead of being rounded to another id. A JSON check reads the same digits, so "no longer the same id" fails on the same id, and a request naming the captured id still sends its token.
A differs check finds the element again when the app draws it anew (a React remount), where it used to fail for good.
A shard run removes its own earlier report as it starts, so a shard that then fails setup, crashes or is stopped is missing from
report --mergeinstead of standing in with the earlier run's results. It also goes when the run stops before running anything (a feature file error, a--filterthat matches nothing).saffron ejectwarns when the project's ownplaywright.configrecords a trace and the exported tests read secrets,{env:...}values or credentials the API hands out, which Playwright's trace keeps unmasked (SAF-200). It no longer only checked on the first eject, and it leaves the file as it is. A config it writes leaves the trace out for captured credentials too.A pending proposal stays with the Outline row whose values it was recorded on when a row is inserted above it or the rows are reordered (SAF-201), where another row used to take its "(example N)" and record over it.
saffron statusand the plugins name that row as the rows stand now, and the review page shows that row's pictures, where they used to name and show the row that took its place.An API credential is masked wherever it could still show: a token a polled response handed out before the one waited for, a token a later capture replaced (a poll that rotates it, then quotes the old one as
previous_token), and an error raised while a request is built (a body value that is no number quoting the captured token it resolves to). The exported test learns a token from every polled response too, where it learnt only the last one's and an error quoted the one before.A compiled
Thenkeeps a quoted value as written:the /name field is "Ada Lovelace"checks for both spaces, where it used to check for one.saffron accept <file>#<revision> --step-setrefuses a proposal replaced since it was listed, as a whole accept does, where it took the replacement's steps unseen.A request sent several times at once waits for every copy when one gets no response, and keeps the copies that answered in the transcript, where the failure came first and a late write went unrecorded.
An exported test extracts an integer beyond JavaScript's precision as the digits the API sent, as replay does, where it rounded an int64 id into another record's.
Each API response is let go once read, in replay and in exported tests, so a long poll no longer holds every response it read until the scenario ends.
An exported text check reads the copy replay reads, a shown one after a hidden copy with the same text (a mobile menu's), where it read the hidden one and failed.
A step naming its column for the statuses of copies sent at once (
refused with <refusal>) records them as that column, as it records a single status, so the other rows replay where they were refused as recorded for one row.A JSON check reads a number JavaScript cannot hold as written as a number, written as the API sent it: a
minormaxcompares it exactly with the bound as written, in any notation (9007199254740993,9007199254740993.0and9.007199254740993e15are all over a maximum of9007199254740992,9007199254740991.1over one of9007199254740991, and1e9999999999over1, where it stopped the run), and apattern(which reads text) does not hold on it. Anequalsstill checks its digits, as the error for a typed number advises.A compiled
Thenleaves a number JSON cannot hold (the /n field is 007) to the agent, where it compiled a check that could never pass.
Fixed, from real-world use of 1.3.1:
The report's pass rate never reads 100% while a scenario failed: 473 of 475 reads 99.58%, where it was rounded up to 100%. The dashboard's suite health, recorded and tag shares read the same way, and 0% means none.
passRateinsaffron status --jsonand the history stays a whole number, now never 100 unless every scenario passed (99 for 473 of 475).An Outline row that fails in a step before any step uses its own values (a lunch sold out in a shared checkout step) is blamed on the app's state, blocked by the environment, where the report used to call it the recording's fault and point at a column the step never read. A value used in a step's data table or doc string counts as used.
A pending proposal stays with its row when a column is added to the Examples: only the columns its row had tell rows apart, where every row was called another row's and none recorded again.
The sensitivity check judges a check against the action before a step that only reads (a captured value, a watch armed), as it already did for a wait: reading a booking number is not what confirms a payment.
api_contract(the MCP tool) and the agent's contract summary list a request body's alternatives with their fields (one of: { ... } | { ... }) and anallOf's parts as one body, where they saidany.The Saffron skill's syntax reference describes the negative and exact URL checks 1.3.1 added, where it said there were none.
A recorded API step no longer keeps the agent's failed attempts (SAF-202). A request whose body the pinned contract refuses is not recorded, even when the API answered 2xx, unless the step expects a refusal. The body is judged as sent, its tokens resolved:
{date+1}as a date, and{captured:id}as the text it sends, so an integer field takes the number itself. A request sent again to the same operation and expected answer right after the step's last one takes its place, unless the step asks for several ("twice", "two customers", "for each", "once more"), the earlier one sent a value the step names that this one does not ("the currency to SEK and guest names on" is two writes), this one sends the same fields with other values the step does not name ("a mug and a hat"), or both carry the same idempotency key. A value counts as named in any case and in the step's data table or doc string, and as sent in a form or the query too (?status=draft, then?status=archived); a step with a table of several rows keeps every request.A
Thenthat names a section in plain words (the confirmation Summary shows the package,the order card shows Paid) has its checks scoped to that section, as a quoted name already did (SAF-203): a check that reads the whole page is refused while recording, instead of passing on text that was there before the action.An Outline whose recorded row held one value in two columns (
currencyanddisplaybothNOK) no longer refuses the other rows for a step that names only one of them (I choose "<currency>"): the recorder writes the column the step names. A step naming both, or neither, still records again on a row that types them apart.A request that sends its header's credential in the body or URL as well (
Authorization: Bearer {env:KEY}and a body fieldapiKey) is refused while recording, unless the step names both the body and the header. Only an auth header counts:X-Auth-Token, notX-AuthororX-Auth-User.A "not shown" check naming a control by the start of its name (a textbox
Firstname, where the page showsFirstname Required) gets an insensitive check note saying no element has that name, and naming the control shown before the step.A check that finds its element by the text it checks (an amount found as the text
SEK 495.00anywhere on the page) is refused while recording: it holds wherever that text shows. The agent finds the field the step names, or records a visible check when the step only says the text is shown.
Docs
- How a recording is checked. A new section of the guide (§24), and a
page of its own on the site, maps every check Saffron runs on a
recording before it is filed, in order: what each note means, which ones
hold a proposal from
accept --alland when to accept it by file, what is on by default, and how to turn on stability proofs.
Editors (VS Code extension 0.7.8, JetBrains plugin 0.7.9)
- Both editors mark
Examples: {data: roles}or{Data:roles}with no table under it as an error where it is written, as the runner stops on it: VS Code in every project, JetBrains through the runner's language server. Neither marks one inside a doc string or with a description before its table. - VS Code: the "Proposals pending review" row asks before it runs
saffron accept --all. - JetBrains: Accept All and an accept or reject of selected proposals no longer become the selected run configuration, so the Run button never repeats them.
- A URL check (
Runner
Upgrading from 1.3.0:
A recording with a negative or exact URL check, a check of a message that vanishes, a computed-style check, or a value captured earlier inside the element a step looks for is cache schema version 14, 15, 16 or 17, which 1.3.0 and earlier refuse ("Update saffron-ai"). An Outline recording whose cell is a
{data:...}value holding a date is version- Upgrade every machine and CI job that replays the suite together.
saffron.config.jsonrefuses a quoted number ("actionTimeoutMs": "10000"), a negative one, or ascreenshotother thanoff,viewportorfull-page, where it used to take them as they were.A row, data or captured value inside an API request's path is percent-encoded where it would change the URL (
#,?, a space;&,=and+in the query):A#1is sent asA%231, no longer cut at the#. A/and an existing%20are sent as they are.A
*that opens a scenario after a Background ending in aThenis an action now, as Cucumber reads it, not a check: a recording of such a step records once more.A data table or doc string written under a
StepSet <name>call is an error naming the file and line, where it used to be dropped.With
resetcommands, recordings run one at a time whenagentWorkersis above 1 and the agent is on, and the run says so: one session's reset would wipe the data another is recording on. Replays still useworkersas before.strict,verifyProposalsandreuseStepsinsaffron.config.jsontaketrueorfalsewithout quotes; a quoted"false"used to read as on.In an Outline, a number or boolean a request body or JSON check holds only by coincidence with a row's value (a field not named like the column, or a step not naming it) is kept as recorded and binds the recording to its row, where it used to be sent and checked as each row's value. Other rows are refused with a message until the field is named like the column or the step names it. A constant beside a column the step writes (
active: truebesideadmin: <admin>) is kept as recorded without binding the row.The recorder refuses checks that cannot fail: a URL check every URL holds (
/), a text, style, download or response pattern that matches any text (.*), and a watch for a message other than the one theThennames.A watched message counts as seen only when it appears after the watch was armed (an element already showing it must stop showing it once), and a seen check uses its watch up: a second check of the same message needs a watch of its own. A recording that relied on a message left on screen goes red and heals.
saffron diff --jsonprints a list for one proposal too: a script that read an object for one proposal reads the first entry now.SIGTERM stops a run as Ctrl-C does and exits 143, not 2.
--workers 0,--agent-workers 0and a worker count that is not a whole number exit 2 instead of running one worker.init --examplesno longer writes into an existingsaffron.config.json.From real-world use of Saffron.
- A data value an API request takes through its step's table (
| lastName | {data:signup.lastName} |, sent as<table:lastName>) can change and the recording replays at zero AI: the freshness check follows the table cell to the data it names, and a header name HTTP defines (User-Agent) is never taken for an old value ("User"). (SAF-177) - A URL check with no URL is refused while recording, saying that a URL check reads what the URL contains, and that there is no "does not contain" or "is exactly" URL check yet: four such checks had been recorded empty, and only the paid proof refused them. (SAF-178)
saffron run --plan --jsonprints only the JSON on stdout, with a shard too: theshard 1/2 ...line goes to stderr. (SAF-179)- Fewer false sensitivity notes: a check is judged against the last step
that acted on the page, a step that only waits passed over; a
StepSet's exit guard is not probed against the set's own last action,
as it says the whole set got where it should (a set of checks only is
probed against the step before it); and a step that reads a whole
body, page or message gets no
broad check:note. (SAF-180) - From a review of these fixes:
--plan --jsonalso keeps the note on Examples rows that share a recording off stdout, and an empty shard's plan is an empty JSON plan, with no shard report written; a StepSet used twice in a row has each use's exit guard left unprobed; an old data value written into a body key or a custom header's name, whole or in part, still makes the recording stale; "the body of a card", "the entire row" and a card's body are not read as the whole page; and a URL check's text is recorded trimmed. A header's value is still searched for an old data value (Authorization: Bearer <old token>); only the header names HTTP defines are left out, and a header an extract reads by name is not taken for data.
- A data value an API request takes through its step's table (
From a three-round review of the runner.
- No false green from a blank row value: a role name pattern, a response URL, a JSON field or header pattern, a watched message and a style pattern that a blank value leaves matching more than they mean are refused, as text and URL checks already were. An expected status left blank is refused, never read as any 2xx.
- Outline rows never record over each other's proposal: a row whose
pending proposal was recorded for another row is red, with a committed
recording too, and a proposal from another machine never turns a
failing protected
Thenyellow. - A fixture takes values only from a proof that passed, and a proposal from another machine is never replayed for its values.
- A scenario whose run throws (a busy proposal lock, a full disk) is that scenario's red; the run goes on and writes its report. Reports, shard reports and the history are written whole, never half.
- A contract check repaired by a heal keeps every operand of the check: a URL, pattern or attribute changed is review, not repair.
- Secrets: a value set with a trailing newline is masked as the page
shows it, trimmed; credential headers and cookies in a trace are
replaced whole (
Authorization: Basic ...); names written as one word (authtoken,JSESSIONID) count as credentials; a header the transcript keeps is judged by the same rule as the trace. - A pattern holding a harmless escape (
\#,\:) next to a row value no longer fails as an invalid regular expression. {captured:...}in the element a step looks for is the captured value, and a retried "message was shown" check can still see the message.- API scenarios: YAML merge keys (
<<: *base) in a contract are applied, the response schema is chosen by the response's content type,differscompares lists and objects,the /x field is nullcompiles, a phrase-compiledThenserves every Outline row, and withapi.transcript: "off"no error quotes a response body. - Recording: a multi-line typed text keeps its
{captured:...}, a{unique:...}inside an Examples cell is typed resolved, a feature edit writes<plan>only for the whole word (Productionstays), and Antigravity stops when the run aborts. saffron ejectwrites tests never looser than replay: an exact path or style is compared whole, enabled and disabled checks need the element shown,differscompares amounts, a not-visible check reads every locator at once, timed waits are capped, the device's own engine is dropped, and Playwright's trace is left out when the tests read{env:...}values, since it would record them unmasked.- CLI:
run --planvalidates its flags asrundoes,coverage --write --jsonprints only JSON, the HTML report escapes every proposal field, andinitsets up the rest before naming an MCP file it cannot merge. - Amounts and dates:
(1.5 kg)is positive, and a date range that ends before it starts is refused.
From a second two-round review of the runner.
- Outline rows:
accept --propagateno longer copies a healed locator holding its Outline row's value into other recordings, andsaffron statusmarks an Outline stale when any row would record again, naming the row. - Secrets: a value set with a trailing newline is masked where the page holds it trimmed (in a URL, in another case, escaped in a selector or glued into a word), and a warning about an unhandled rejection masks the credentials a session captured.
- Recording: a target with nothing to find the element by is refused, a
saffron_stepwhose text names another step is refused so actions are never filed a step late, the firstsaffron_doneverdict stands, a pass past the time budget that leaves a step unrecorded is a failure, and a Claude session stopped at its turn limit says to raisemaxTurns; Claude token counts include compaction. - Files and processes:
prunekeeps a merge-conflicted recording of a live scenario outsidefeaturesand the recording of an Outline whose Examples have no rows for now; a scenario name of any length can be saved, and recordings saved under the whole name are still found; Ctrl-C stops a runningreset; ahookTimeoutMspast a timer's limit no longer ends every hook at once; a hook timed out on Windows ends with its child processes; a run killed while writing history no longer spoils the next line; a browser that fails to close or a history file that cannot be written no longer costs the run its report; writes ride out a file briefly locked on Windows; a failed browser close leaves no video folder behind;initgit-ignores the temp and lock files a killed run leaves. saffron ejectwrites tests never looser than replay: a schema check finds its operation with the row's value in the path, a response with a schema per JSON media type is refused, a blank expected status and the NAT64 and IPv4-compatible link-local forms are refused, attribute and style checks read the element replay reads, a blank captured value in a checked text fails, a dialog answer from a row is that row's, a whitespace-only text pattern is fixme, and header patterns compile as replay compiles them.- Status, plan and coverage: a feature in another Gherkin language shows
its name,
run --plancounts a feature's fixture,coveragerefuses a suite with a file it cannot parse andverifysays so, and a scenario named like an Outline row keeps its own last result. - Editors and reports: the language server reads doc string lines as data, not StepSet calls, and the CI run panel shows a cost under a cent as the report does.
- Outline rows:
From a two-round review of the runner.
- No false green from another row or a leftover: an Outline cell that reads a date through a data value has its day checked as a date cell does; a step reused by swapping a quoted value no longer offers the old value's other recordings; a message already on screen before an action no longer counts as seen; a capture pattern a blank row value leaves open is refused.
- Recording types a captured value as plain text, as replay does, and
refuses a browser call whose captured value holds an
{env:...}token rather than type the secret. - Checks without a frame also find what an iframe attached after the check began shows; checks with fallback locators end at their timeout; a drag finds its drop zone by its fallbacks; the heal guard runs a check on the tab an earlier action moved to or opened.
- Seeded recording keeps a watch armed in one stretch for its
Thenin the next, and seeded steps and the sensitivity check read values another step's table or doc string gives. - API: a schema check finds the operation for a path with a parameter
inside a segment (
/files/{name}.json); a row value after a captured URL that brings its own query is encoded as a query value, in replay and in exported tests; with no agent, an Outline's rows file one contract re-check, not one each;api pinpasses over a fallback URL that does not answer with OpenAPI 3. - Proposals:
acceptnotices a hand edit of a proposal's API contract fingerprints,accept --step-setrefuses to replace another StepSet whose name makes the same file name, andrunreports two StepSets or a fixture and a scenario whose recordings would share a file; an accepted heal no longer copies a captured value into other scenarios; a "capture differs" note leaves credentials out. - Reports: a password holding an
@in a URL on the command line is hidden whole,report --mergerefuses a shard it cannot show, naming the field, an exported or imported bundle's report names only the evidence the bundle carries, and the trace page marks a request the agent did not record. - CLI and editors:
saffron authorfails with the provider's error instead of writing it as the draft;initrefuses an instructions block with no end marker instead of deleting the text after it; a config file holding something other than an object is refused by name; an exported timed wait written with a row value waits as long as replay; feature edits apply to steps whose keyword is several words (Étant donné que); the language server started by an editor reads a project with nofeatureskey as the default folder.
A
Thenis read-only while recording. A check refuses a browser action that enters, presses keys, picks, ticks, uploads, drags or navigates, and a click on a button that submits or commits (Log in, Save, Submit, Confirm and the like), before the browser runs it: the agent is told the step is a check and to fail it, suggesting the action as aWhen, if it does not hold as the page stands. Before, such a click was recorded and proved, and held for review afterwards; a login suite had a check click Log in, and others submit again to bring back a toast that had vanished. A click that opens what is checked (a menu, a tab) is still recorded and held as before, and proposals filed before this stay held fromaccept --all. (SAF-181)A "not shown" check must see something disappear. The sensitivity check after a proof runs each "not shown" check in a
Thenthat holds other checks on its own, before the step it follows; one that holds there too (its element was never on the page, as with an invented selector) gets aninsensitive check:note naming it, held fromaccept --all. Before, the step's other checks failing there hid it, and it stayed green whatever the page showed. The recording agent is also told to target the element a named field labels (the booker, not any paragraph showing the name) and only elements it saw for a "not shown" check. (SAF-183)A StepSet's own data value stays its own in an Outline. A step that enters a
{data:...}value itself, while not naming the column (a StepSet typing{data:guests.first.surname}), no longer has the rows refused because one row's cell reads the same: the email row of a login suite was refused for it. A value a step only checks is still refused for the rows that differ, as the page may show it from the row. (SAF-182)Trial navigations stay out of the recording. A navigation the agent leaves for the same page with another query or fragment, before doing anything there, is discarded and noted in the proposal: a login suite kept each query format the agent tried as a replay action. A navigation to another page, or one acted on, is kept. (SAF-191)
A part of a name the agent wrote replays. A click through a locator the agent wrote, naming a part of the element's name, is recorded with the full name the page it saw last shows for the one element of that role holding it: replay matches a name whole, and timed out on "Sign in". Where the page showed none or several, the proposal notes it while recording, before the proof. (SAF-184)
Negative and exact URL checks. A URL check can now say that the URL does not contain a value, or that its path is exactly one, recorded from the step's own words: before, a URL check was containment only, and steps saying the URL no longer holds the login could not be recorded. Replay and
saffron ejectcheck them alike. A recording holding one is cache schema version 14, so an older runner refuses it instead of checking containment. (SAF-186)Checks for messages that vanish. A toast or error flash gone before its
Thenruns can be checked: the agent arms a watch (saffron_watch) in theWhenstep before the action that shows it, and theThenchecks that it was seen since. Replay watches the same way. Before, a login error toast vanished before the check, and recordings submitted again inside theThento bring it back. A recording holding a watch is cache schema version 15;saffron ejectwrites its scenario astest.fixmefor now. (SAF-185)Computed-style checks. A step can check an element's computed CSS property (an invalid input's
border-colorisrgb(220, 38, 38)), or match it to a pattern: before, only classes, attributes and the visible error could be checked. Replay andsaffron ejectcheck it alike, in Chromium, Firefox and WebKit. A recording holding one is cache schema version 16.saffron ejectalso writes the negative and exact URL checks now, which it had written astest.fixme. (SAF-187)A navigation to a captured URL stays the capture. When the agent goes to
{captured:signin}and the value is an address on the base URL, the recording keeps{captured:signin}: before, it kept the day's path relative to the base, and replay went to the old link instead of the one its own request got. (SAF-188)Accept one StepSet from a proposal.
saffron accept <file> --step-set "Sign in"files only that set's steps, which its proof replayed green, for every other scenario to reuse at zero AI, while the proposal and its failing check stay pending. Before, a failed check kept a verified sign-in from being reused, and each scenario paid to record it again. A proposal whose proof failed records how many steps passed before it (meta.provedSteps), and a set the proof did not get through is refused. (SAF-189)Reuse picks the recording the page's state fits. A step recorded in two page states (a promo field opened first, or found open) keeps both recordings, and seeding replays the one whose controls the page shows now: before, it always took the newest, which closed a field already open and paid the agent to record the step again. (SAF-190)
CLI and files, from the 1.0 review. (SAF-133)
- A scenario with no steps is red, saying so, and starts no agent session: it had recorded nothing and passed.
- A broken link or a folder named
*.jsonunder.saffron/proposalsno longer stopsstatus,diffand the end of a run. saffron diff --jsonprints a list for one proposal too, as it does for several.accept,rejectanddifffind a project-relative file (asstatusprints it) under--project-root, not only from the current folder.accept --with-feature-editedits a step line that ends in spaces.saffron statusprints an unreadable recording's warning on a line of its own.saffron initkeeps a skill you edited under.saffron/backups/skills/before installing the shipped one.- Already fixed and checked: an empty run exits 2 naming why, a StepSet file that does not parse is named, recordings are written atomically, a CRLF file takes feature edits, and status counts an unreadable recording apart.
A check's selector is tried on the page while recording. A check whose CSS selector matches nothing on the page the agent sees is refused then, at no cost, naming the likely cause (a link styled as a button is an
<a>): before, only the paid proof caughtp:has(button). The other half of this story, abroad check:note for a check held by a whole card, shipped in 1.3.0. (SAF-171)Exact
Thenphrases need no agent. In an API scenario,Then the response should answer 404,Then the /status field is activeandThen the response matches the Customer schemaare recorded as those checks without the agent, which records only the steps around them. Any other wording goes to the agent as before, never guessed. An option, not a required syntax. (SAF-162)Live progress, and a time budget per agent session. While an agent records, the run prints on stderr each step it records and each proof as it ends (
· Checkout: recording step 41 of 92: ...,· Checkout: proof 1 passed): a 92-step recording ran 340 seconds with no word until it finished.maxSessionMinutes(--max-session-minutes) ends one session that has gone on that long: its tools refuse, and it ends unfinished with the agent's account of where it got to. (SAF-194)Row proofs show their time and each row's latest result.
saffron diffand the review page say when an Outline's rows were proved, and beside each row its latest result in the last run (latest run: red at ..., orno run since): a proposal's passing row proofs were read as readiness while later runs of those rows timed out. (SAF-192)A failed response is named beside the step it broke. A step that fails after one of the app's own requests reported failure (an error status, or a JSON body with
success: false,ok: falseor anerror) says so in its error, which the console, the report and the heal prompt all show: an HTTP 200 holding "overbooked" before a timeout reaching checkout no longer reads like a locator problem. (SAF-193)Outline rows, from the 1.0 review. (SAF-128)
- A list check whose values come from
{data:...}or{env:...}cells ("Alice, Bobby" from{data:guests.a}and{data:guests.b}) is tied to its row, as one of plain cells is, so another row fails it with the reason and no agent session: it was written as the row's<column>, and every other row failed it and paid the agent again on every run. - The guide says what a row-bound recording keeps of a short secret read through an Examples cell, and why a PIN belongs elsewhere.
- Checked and kept with tests: a value inside a word of what a
Background step types (
testinqa-tester@example.com) binds no row; a date cell typed as ISO, a value seen only as a slug or in another case, and a column naming a selector's role refuse the other rows instead of recording again; a URL-encoded value is the row's<column>; a{data:...}value naming an unset variable stops the run before it starts; a short secret glued to other text is masked.
- A list check whose values come from
Visible text, from the 1.0 review. (SAF-129)
- A check whose value is written on several lines (a two-line doc
string) holds where the page draws those lines, split by a
<br>or by blocks: it never matched "Line one
Line two", which read "Line oneLine two". A value on one line still reads the text as one run, so two blocks drawn apart never make it. - A list check (
expectEach) no longer passes on a hidden option, or hidden text, that differs from a shown one only in case: the page's lines in another case count only where CSStext-transformchanges them. - A check on a large container is faster: 22 ms instead of 56 ms on a 4,000-row section. 0.9.9 takes 4 ms there, as it did not read what is drawn; the rest is the cost of reading only drawn text.
- Kept as it is: an element without text whose first match is hidden gives way to a displayed one, as a hidden mobile copy does to the desktop one; a check that means one row's badge targets that row.
- A check whose value is written on several lines (a two-line doc
string) holds where the page draws those lines, split by a
CLI, reports and the trace viewer, from the 1.0 review. (SAF-130)
saffron diffnames the text a role with no name is found by:expect listitem with text "Milk" to be visible, not every listitem.- The trace viewer titles another browser session's step by the keyword
it was written with (
Given,And), not by its kind, and a session opens on a step of its own rather than on another session's first one. - The results page embeds at most 20 MB of screenshots, failures first; past that it names each file instead (300 failures made a 78 MB page).
- A feature reached through a linked folder's alias, or named in another case on a case-insensitive disk, is the suite's feature, with its recordings, not a new one.
- The MCP tools keep answering while
saffron.config.jsoncannot be read, on the config they last read, and say what is wrong with it. init --examplesleaves an existingsaffron.config.jsonas it is: the examples'retries,strictandtraceno longer change the project's own runs, and a missingbaseURLis named in a note.- A bundle made in a git checkout with no remote is refused for that, not as one made outside a checkout.
- SIGTERM stops a run as Ctrl-C does (teardown, the report of what finished) and exits 143, instead of a red line and exit 2.
- A red scenario that replayed no step is labelled
not runin the run output, notcache. --workers 0, or a typo in--agent-workersor in the config'sworkersoragentWorkers, exits 2 instead of running one worker.- Fewer temporary folders are left behind by the test suite.
- Kept as they are:
saffron statusalready marks an Examples row a row-bound recording refuses (✗, with the reason), and the chronic hint counts only reds that cost an agent session, which a refused row never does.saffron report --no-openalready writes the report without opening a browser (1.3.0).
Language server and editor parity, from the 1.0 review. (SAF-131)
- The language server reads data Examples under every Examples keyword
of the file's language (
Scenarios:,Exemples:), as the parser does, instead of flagging them; a bare{data:file}in a step is no reference to the runner, so it gets no hover and no error; and the# language:header is read with Gherkin's own pattern. - Its completion rates a step after
As "<who>"(a session an Examples column names) unrecorded, as step reuse never seeds it.
- The language server reads data Examples under every Examples keyword
of the file's language (
A text check that fails on a value the page holds but does not show says where it sits:
Timed out waiting for text "Total: 42 NOK": the page holds it, but not shown (inside a closed <details>), or in an element withdisplay: none, thehiddenattribute,visibility: hiddenorcontent-visibility: hidden, naming the element. A total that is gone and one behind a panel that did not open read apart. Text the page does not hold reads as before. (SAF-195)A text check on a large container is fast again: 3.5 ms on a 4,000-row container (0.9.9 took 2.4 ms, reading hidden text too). The value is looked for in the container's text first, and only the text it covers is asked whether it is drawn; anything that question cannot settle (a textarea, a dropdown, a shadow host) is read the full way, so the check passes on exactly what it passed on before. (SAF-129)
A range of days the page shows with its year (
20–22 Oct 2026,Oct 30 - Nov 1, 2026,31 Dec 2026–2 Jan 2027, German20.-22.10.2026), up to 62 days long, is recorded as a range token ({date+7–+9:D MMM YYYY}), as a single day already was. A step that names a stay by intent ("check-in in 7 days for 2 nights") now records a check that follows the clock instead of a date frozen on the day of recording. The agent is no longer told to match dates with a pattern, which passes on any day. (SAF-124)The agent skill and the block
saffron initwrites intoAGENTS.mdandCLAUDE.mdsay when a small supporting script is the right tool (only for a value Saffron cannot express, asked for first) and how to keep it trustworthy: expected values never taken from the output being checked, run as asetupcommand writing git-ignoreddata/files, assumptions stated, and a report of what could replace it. Runsaffron initagain to update the block. (SAF-125)A video of each scenario's run:
"video": "retain-on-failure"keeps the video of every scenario that is not green,"on"every one, invideo.webmbeside its screenshots; also--videofor one run. Off by default. It works in Chromium, Firefox and WebKit, and a heal or a recording is in the video of the run it happened in; proof replays are not filmed. A browser that cannot record (no ffmpeg) stops the run once, at the start. A summary export carries no video. A video shows what is typed, secrets included, so keep it off for scenarios that type them. (SAF-51)The limits of the CI runs viewed into
.saffron/ci/are settings:"ci": {"cacheRuns": 20, "cacheMB": 4096}. The defaults stay 10 runs and 1 GB. (SAF-121)Each feature edit a proposal suggests names the file and line it rewrites, in
saffron diff(fileandlinein--json), the review page and the run report. A StepSet's step names the StepSet's definition, where accept rewrites it for every scenario using the set, and an edit to a Background step is marked skipped, as accept leaves it to you. (SAF-31)saffron status --jsonlists each pending proposal's feature edits (featureEdits, asdiff --jsongives them, with file and line), and the VS Code and JetBrains panels show them in a proposal's details without another command. (SAF-30)Each step a proposal changes is marked with its class:
repair(only how the step finds the same element again or waits for it: a fallback selector, a timeout, a wait),review(what it does changed, an element named another way, or one of Saffron's notes names it) or, for a suggested feature edit,protected(the wording, which only you apply). It shows insaffron diff(adaptationin--json, with counts insummary), the review page and the run report, and the guide has a section on the three classes. (SAF-32)From a review of this release: a step whose target is named by a selector inside an iframe is
reviewwhen the selector changes (the frame names the iframe, not the element); the heal branch's pull request body, the run report and the editors' trace panel class each step assaffron diffdoes, reading the proposal's notes, and name the file and line of each feature edit; a video with no viewport setting is filmed at the page's own 1280x720, not Playwright's 800x800 fit; a range that writes the month twice (Oct 20 - Oct 22, 2026), one ending exactly 31 days back, or one from before today to after it (Room 7 - 22 Oct 2026is a room) is kept whole, never with only its end rewritten.
Editors (VS Code extension 0.7.7, JetBrains plugin 0.7.8)
- Both editors list the feature file edits a proposal suggests, each with
the file and line
saffron accept --with-feature-editrewrites and why accept would skip one: VS Code in the proposal tooltip, JetBrains in the Proposals tab details, read fromsaffron status --json. (SAF-30) - VS Code: a
# language:header Gherkin rejects is a plain comment; a file a link inside the test folder leads back to is listed once; a StepSet under a nestedsaffron.config.jsonin the test folder is defined. (SAF-131) - JetBrains: the Files tab fails on a folder it cannot read, as the runner does, instead of hiding it; right-click Run reads the IDE's view of the files, not the disk through links, each time the menu opens; a Windows junction is walked as a link. (SAF-131)
- Both editors open a report path through a link outside the test folder only when it really leads into the project, as the runner follows links inside the test folder only. (SAF-131)
Runner
Upgrading from 1.2.2:
A recording that keeps part of a captured text (
pattern) is version 13, which 1.2.2 and earlier refuse ("Update saffron-ai"). Upgrade every machine and CI job that replays the suite together.accept --allnow also holds proposals with arow not proved:,date not following the clock:,write not finished:orcheck changed the page:note: accept them by file once reviewed. A teammate still on 1.2.2 runningaccept --alltakes them, so upgrade before bulk-accepting. An Outline recording that fails a row it serves is filed unverified, its proof error naming the row, so every runner skips it.Now refused: an Examples table that names a column twice; a text check a step words as a comparison ("starts with") but records as containment; under a
Then, an action that changes the app (it is not recorded, and the proposal is held); a captured credential (a login's token) typed through the browser.transport.api.headersgo only to an API under test; an unset{env:NAME}in them, or a value that is not text, stops the request instead of sending it.The run history (
.saffron/history.jsonl) keeps its newest half once it passes 64 MB.saffron ejectrefuses a capture that keeps part of its text (pattern), which an exported test does not do yet.--plancounts an Outline as one recording. New output fields:estimate(--plan --json),failedRows(status --json, the MCPverifytool, the CI run view),rows(diff --json, the report JSON),invocation(the report),checkedAt(a recording), andproof-N-requests.jsonevidence files.Recordings checked by machine, not read by people.
- An Outline's recording is proved on every other row it serves, at
zero AI, in a fresh execution each, before it is filed. One that fails
on a row is filed unverified, its proof error naming the row, with a
row not proved:note; one that cannot serve a row (bound to its own row's values) gets the same note, as accepted that row stays red until it is recorded again. Both are held fromaccept --all. Rows whose@requires:capability this environment lacks are not proved, and the review counts them apart: "proved on 2 of 2 rows (1 refused)". (SAF-159) - A new recording is checked against a moved clock: every
{date}token resolved for tomorrow, the month's last day and the next month's first, and the year's last day and the next year's first, and a weekday, month or year left as text beside it must still match (Thursday, {date+1:MMMM D, YYYY}does not; an amount such as2026.00is no year). A whole date near today left as text is named too, in the locale's own spellings and in the other order where only one reading is a day ("13/11/2026"), unless the scenario writes it or it is part of an address (/blog/2026/10/30/). Either gets adate not following the clock:note, held fromaccept --all. (SAF-158, SAF-165) - A proof that passed while a write to the app's own site was cut off by
its frame moving on gets a
write not finished:note, held fromaccept --all; a replay that fails after such a write names it. The app's own writes are those to the base URL, an API the run tests, or the page's own site; an iframe loading, a third party's beacon, a download or a 204 count for nothing, nor a read sent as a POST (a GraphQL query, a search), judged by the rule aThen's requests are; a navigation that fails on the network counts. (SAF-170) - A text check that passed on an element showing far more than the value
(a card) gets a
broad check:note. (SAF-171) - A page check is judged against the last step that acted on the page: an API request in between cannot change the page, and named an insensitive check that was not.
- A check of an Examples cell that is a day written out (
2026-01-05,05-01-2026, "Jan 5", "5th January 2026"), shown in another format ("05.01.2026"), binds the recording to its row, as a{date}cell does: every row checked row 1's day. A cell such as "Order 1234" is no day. rowProofsin saffron.config.json caps how many other rows a recording is proved on; a row past it is "not run", and replays the proposal on its own row later in the run.- A contract re-check (only a pinned API contract changed) files the
recording with the Examples row, bindings, data and recording time it
was made with, whichever row ran the check, and records when it was
proved again as
checkedAt. A re-check that would change any action is red, never filed. (SAF-157) - A stale reason and a row refusal start with the file and line they
point at (
features/plans.feature:5 step 2 reads ...), and a held note names its step's file and line after its kind. (SAF-160)
- An Outline's recording is proved on every other row it serves, at
zero AI, in a fresh execution each, before it is filed. One that fails
on a row is filed unverified, its proof error naming the row, with a
What a recording keeps.
- The agent types the scenario's references (an Examples
<column>, a{data:...}value, a{captured:name}an earlier step kept) as written: the browser gets the value and the recording keeps what the agent wrote, as with{env:...}. Two references that read the same in one call are left to the recording's own row matching. A captured credential is never typed through the browser: the agent would read it back, and the page could send it anywhere. (SAF-161, SAF-166) - A value captured while recording can be typed by a later step
(
saffron_capture, then{captured:name}). A capture can keep only what a pattern picks (one group, named or not; it may hold{captured:...},{date...}and<column>tokens), so a booking number beside its Copy label is the number alone; replay fails naming the text when the pattern picks no value or two, or names one that backtracks ((a+)+), which is stopped instead of freezing the run, as amatchescheck's pattern is, and adifferscheck against it compares the same part of the text. Such a recording is version 13. A value the proof read with more around it (a label) gets acapture differs:note. (SAF-166) - A day the scenario writes with its month's name and no year
(
{date+9:D MMM}) goes back to its token in every step, as a whole date does, and so do the spellings "Nov 6 2026", "6. November 2026", "2026/11/06" and, in the United States, "11/6/2026"; a path's segments (/blog/2026/10/30/) stay as they are. (SAF-165) - A
Thenchecks; it never changes the app. An entry, a key press, a pick, a tick, an upload, a drag, a navigation, a tab opened at an address or a request that writes is not recorded under it: a write request is refused before it is sent, and the proposal gets acheck changed the page:note, held fromaccept --all. A read the API offers only as a POST may be sent: a GraphQL query to a/graphqlendpoint that holds no mutation (comments and strings aside), or a request to an endpoint whose last segment issearchorquery. A click or an answered dialog under aThenis recorded, as it may only open what is checked, and held the same way, as nothing the run sees can tell an opening click from one that cancels the order the check reads: accept it by file once you have seen it only opens. (SAF-167) - A step performed again after an attempt that did not work drops the
earlier attempt (
saffron_step restart). (SAF-167) - A text check the step names as a comparison (
starts with "Oct",ends with "-2026", "the same day as") is refused as plain containment, naming the check that records it, in straight or curly quotes or as a capitalized word (starts with Oct), and a pattern recorded for it needs its anchor, in any group ((?<id>^ORD\d+),(?i:^ORD)) or as a lookahead ((?=$)). A number with a leading zero ("0042") is an identifier, never compared as an amount. (SAF-169) - A choice opened through an element named with the row's
<column>gets achoice opener:note, and the agent is told to open a choice by its own label, never by the value it shows. (SAF-168)
- The agent types the scenario's references (an Examples
Review: every view says the same.
- The review shows an Outline recording's proof row by row. One that
failed a row is never shown as verified: in
saffron run, its totals and cost split, the report, history and the dashboard,saffron diff,status --jsonand the MCP tools (failedRows), the CI run view, the pull request body andsaffron trace, which asks before accepting it. A later run that replays it keeps what its own proof found. (SAF-172) - A held note stands beside the step it names in
saffron diffand the report's review tab, its file and line included, and the pull request body names every noteaccept --allholds. - A repair that fixes the steps before a
Thenwhile theThenstill fails as written, or that changes nothing and still does not replay, is red, never a proposal healed again on every run.
- The review shows an Outline recording's proof row by row. One that
failed a row is never shown as verified: in
Replay.
ControlOrMeta, in a key press or a click's modifiers, replays as the browser's own platform has it, so a remote Linux browser driven from a Mac pastes too. (SAF-175)- A step on a page busy in its own script times out as it should.
- A popup no longer crashes the run: its first navigation has no frame yet, which the network listener now allows for.
- A refinement after a failed proof gets the values the proof captured.
API tests.
- The transport's API headers go only to an API under test, never to
another origin a request names. Each
{env:NAME}is read when the request is sent, and one that is not set is refused instead of sent empty. A value read from{env:...}, or written whole for a header whose name ends in a credential's word, a version aside (Authorization,X-Api-Key,X-Session-Id,X-Hub-Signature-256,X-API), is masked wherever it would show, including a response the agent reads. A browser scenario's API requests carry the headers too, masked in its trace, leaving out one whose{env:...}the job does not set. - A credential inside an extracted object (
extract "/data"holding a token) makes the capture a credential, and the token is masked on its own, in a step that fails too. A name counts by its last word (accessToken,authorization,pwd,otp,api_key,refresh_token), a word that only says how it is held aside (authCode,apiKeyValue,privateKeyPem), never one that only mentions it (sessionTitle,pinColor,total_tokens): one rule for captures, fields and headers, in replay and in an exported test alike. A value of 8 characters or more counts, a password at any length, or a code of 4 to 8 digits underotporpin; a short word ("auth": "yes") is no secret, and masked it garbled every text it appeared in. - A secret of any length is masked: a value of some 7,000 characters (a large token or cookie) made masking throw.
- A proof's API transcript is kept apart from the run's
(
proof-1-requests.json), for API and browser scenarios alike. The report,saffron traceand the CI run view show the run's own requests first, and each proof's under its own name and outcome. - Copies sent at once match their expected statuses in any order.
- The transport's API headers go only to an API under test, never to
another origin a request names. Each
CLI, plan and reports.
- A run of named scenarios folded into the last report carries its own
command, and says what it ran as
invocation. (SAF-164) --planestimates the browser time of its replays and recordings from past runs, counts an Outline as one recording with its row proofs, and counts no recording time with--no-agent. It counts a new step's scenarios per file, and mentionsresetas it is set. (SAF-174)- A suggested rename to an existing wording needs the two wordings to
share a word or a quoted argument, which it names, in a recording and
in
statusalike. (SAF-173) - An Examples table that names a column twice is refused with its file
and line, and a file whose only Outline has no Examples rows is told
it needs one. While one file does not parse, the step vocabulary (the
LSP, MCP
search_steps,saffron vocab) still offers every other file's steps, and names the file that does not. report --mergerefuses a shard whose rows or proof transcripts it cannot read, a history line that is not a run is skipped, and the dashboard calls a scenario flaky only when it flips in one environment.- The run history is read from its end, so only the runs a view needs are read, and it keeps its newest half once it passes 64 MB: months of CI runs made every command fail. A history that cannot be read costs a run its trends, never its report.
- A recording with many dates is finished several times faster.
report --no-open, andreport --run --no-open, write the report without opening a browser.saffron run --no-agentnames the pending proposal that does not replay on a row, instead of saying no recording exists, and the run line no longer hides a path behind an identifier named like a secret (..._SESSION_ID). An invalidrowProofsstops--plantoo.- The MCP
verifytool takesbaseRefonly as a git ref, and says so plainly otherwise;saffron lsp --stdiois accepted;saffron prunesays its count holds recordings, proposals and evidence folders.
- A run of named scenarios folded into the last report carries its own
command, and says what it ran as
Data tables for API steps. An API request's values in a data table under its step (an API key, a correlation id, query parameters, body fields) record as their
<table:key>wherever the request sends them, path, headers, body or form, so editing a cell replays at zero AI. The bundled skill andsaffron authornow tell the agent to keep each step one sentence and put more than two values in such a table, not on one long line.
Editors (VS Code extension 0.7.6, JetBrains plugin 0.7.7)
- Proposals with every new note are held, the Accept All tooltips name them all, and an Outline proposal that failed a row shows as failing it, in the proposal list and the CI run view.
- A
.saffronfile'sFixture:line is coloured as a block keyword, asScenario:is, in the editors and in the TextMate bundle the runner ships.
Runner
- Proposals are reviewed in the run report. A scenario with a proposal
has a
run | reviewtab pair. The review tab holds what used to sit among the run's panels: every step beside what it records today and what is proposed (unchanged steps dimmed), the notes on a step, the agent's account, the page before the agent took over, the proof and stability proofs, and the commands to accept or reject, each with a copy button. The header has a "N proposals to review" chip that opens the first one. The page before the agent took over is embedded once, as before.saffron diff --htmlis unchanged. The report stays read only. - A new dashboard, in both editors and
saffron dashboard. The page opens on the suite's health and a history grid: each scenario that is failing, flaky or healed again and again gets one cell per run (last 20), so a new break, a flaky test and a drifting page read at a glance. Click a row for its error, and a scenario's name to open it in the editor. A scenario that fits several chips (failing and flaky, say) is listed once, under the most urgent group, and is found under each chip; a group's count follows the chip, and a chip with nothing under it cannot be pressed. The grid's columns are the last 20 runs a listed scenario ran in, so runs from an editor's Run button of other scenarios no longer stretch it with empty columns. Slowest, tags and coverage gaps sit beside it. The pass-rate and execution-time bar charts are gone. The editors need no update: they show the runner's page. - Found in a wider test pass.
- A day the page spells with its weekday ("Thursday, October 29,
2026") in a scenario that writes that day as its own token
(
{date+1}) is recorded as one token. It was recorded as "Thursday, {date+1:MMMM D, YYYY}", which the next day's page no longer shows. saffron run --plancounts stability proofs among the proof replays.- A drag the repair no longer makes is a behavior change only, not also a "requirement not kept" note.
- A day the page spells with its weekday ("Thursday, October 29,
2026") in a scenario that writes that day as its own token
(
- After a review of the unreleased work.
verifysays when the run produced no results (runProblem, and an MCP error), instead of answering with an empty pass, and reads only the rows its own run wrote.- A recording deleted outright counts as losing every check it made, in
saffron coverage --baseand inverify. - Every copy of a request sent at once has its credentials masked in the transcript, not only the copy the step's checks read.
maxCostUsdwarns that Codex, Antigravity and Cursor report no price, so it cannot stop their sessions.- A transport module's
browser()andapi()hooks are asked once per run, however many workers start. - The command a run records hides a URL's password and writes an
environment variable's value as its
{env:NAME}. resetruns before a recording only when an agent session can follow, including when the agent's provider cannot start (a missing SDK).verifyhides every environment value in the run's output it quotes, so a secret a failed setup printed never reaches the agent.- The recorded command masks only secret-like variables (a path under
HOMEstays as typed) and keeps a glob quoted. - A transport hook that failed is asked again by the next caller.
- The
maxCostUsdwarning for an unpriced provider is said once, at its first agent session, not in every run.
- A pluggable transport for isolated environments.
transportin saffron.config.json connects to a browser already running elsewhere (a Chromium bycdpEndpoint, which records and replays, or a Playwright server bywsEndpoint, which replays), sends API requests through a proxy with extra headers and self-signed certificates accepted, and takes a project module whosebrowser()andapi()hooks decide both at run time, so a Docker setup needs no custom integration. - Verification for coding agents over MCP. The new
verifytool replays the scenarios an agent names, or the whole suite, at zero AI (saffron run --no-agent), and answers in a versioned shape: the commit and environment, each check's verdict and evidence, what could not be verified, proposals waiting for a person, and the spec's changes against the reviewed coverage map committed at a git ref, so a weakened or removed check shows even when everything is green. - Requests sent at once, and prerequisites an environment lacks. An
API step can send a request several times at once (
concurrent, with thestatusesthe copies answer in any order), so a double submission that books twice is a red; the agent records it for steps such as "I submit the booking twice at once". Such recordings are version 12. A scenario tagged@requires:<name>is not run wherecapabilitiesin saffron.config.json sets that name to false: it is red, blocked by the environment, naming the missing prerequisite, instead of being healed around it. - A run report built for triage. The HTML report lists every
scenario in one scrollable list, failures first, then proposals, then
passes, with a search box and a filter per status; the scenario chosen
shows beside it. Failures with the same error, and proposals with the
same change, are linked to each other, with one command for them all
(
saffron runfor the failures,saffron acceptwith their files for the proposals). A run map below shows one cell per scenario by feature file, coloured by status, AI cost or duration, so a run of 200 scenarios stays one page; a cell opens its scenario in an inspector under the map (its steps as cards, the failure or the change, and who shares it), leaving the list's choice as it was. Without scripts the page still reads: every scenario is shown in turn. Under the map, widgets show setup and teardown (a failed command first and flagged in the header, the ones that went well folded), the workers' timing (saying when they did very different amounts of work) and the pass rate of the last runs, the newest marked. A duration over a minute reads as minutes and seconds, and scenarios sharing a cause get a button for the command that deals with them all, the command itself still on the page. - A compatibility page per release. Compatibility lists the Node.js, operating systems, Playwright and browsers, each agent's SDK (the range accepted and the version tested) and the recording versions this release reads and writes, generated from the release's own code and lockfile. The agent skill names the saffron-ai version it describes.
- A recording-quality benchmark.
npm run benchmarkreplays the example suite's recordings several times on the demo app (flaky scenarios), and against deliberately broken versions of it (false greens);--recordalso records the suite afresh with the agent and measures the share verified on the first attempt, the share held for review, and the time and cost. The results are published in the docs (benchmark): today 66 of 66 replays green and no false green on 8 broken apps. - A coverage map per pull request.
saffron coveragemaps each requirement ID (a tag matchingrequirementTag, such as@REQ-12) to its scenarios, the checks their recordings make and the last run's verdict.--writesaves it as.saffron/coverage.jsonto commit, and--base origin/maincompares with the target branch's map: scenarios added and removed, requirements no longer covered and checks a recording no longer makes, with--markdownfor the pull request and--fail-on-lossto gate on it. - Side effects are counted, and can be reset.
saffron run --planmarks the scenarios that change the app's data (a write request in their recording, a unique value, or an@side-effectstag) and how many times the run executes each: a recording runs once, its proof, every stability proof and every sensitivity check again. The newresetsetting runs your commands before each of those fresh executions (never before an ordinary replay), withSAFFRON_SCENARIO,SAFFRON_FEATUREandSAFFRON_RESET_FOR. A failed reset stops that recording, or fails that proof, and the run goes on. - Stability proofs.
stabilityProofs(--stability-proofs n) replays a recording that passed its proof n more times, each in a fresh execution, before it is filed. The proposal, the run summary and the review page tell "passed one proof" from "passed the stability proofs", and a recording that fails one gets anunstable recording:note, left out ofaccept --allas the editor plugins mark it. - A recording plan and spending limits.
saffron run --planprints what the run would do and stops, with no browser or agent: what replays, what records or records again (with the reason), what is refused, how many agent sessions and proof replays that takes, and new steps several scenarios share, to record and accept first.--jsonand the MCP toolrun_plangive the same as data.maxCostUsd(--max-cost) andmaxAgentMinutes(--max-agent-minutes) stop new agent sessions once a run reaches them, and the summary splits AI cost into recording, repair and failed attempts. - Every repair is checked against what its recording bound. From the
committed recording, Saffron lists each value a step enters, each request
it sends and each check it makes, and finds the repaired action that
keeps each one.
saffron diff, its JSON (contract) and the review page show the list with that evidence. A repair that no longer sends a request or makes a check gets arequirement not kept:note, andaccept --allleaves it out for review, as the editor plugins mark it. - The run summary says which run it counts. It names the invocation
(
this run (\saffron run ...`, 12 scenarios, 10:04)), splits pending review by proof passed, failed or not run, and splits failures into the app's and those the recorder, the agent's provider or the environment caused (blocked` in the report JSON). A run of named scenarios also prints the combined report's counts. The report shows the command too. - A review page for proposals.
saffron diff --htmlwrites.saffron/reports/review.html: each Gherkin step beside the action it records today and the one proposed, the target, what the proof captured and any note about that step, so "enter an expiry date" becoming "check an expiry hint" shows in the step's own row. The last run's screenshot of the page before the agent took over is included.saffron diffand--jsonshow those notes under their step too. - Dependencies. saffron-ai now needs
zod4.6.7 or later, so upgrading it in a project with a lockfile also brings the fixedzod. The other packagesnpm auditmay report after an upgrade (body-parser,qs,hono,ip-address,fast-uri) come through Saffron's dependencies and are already allowed at their fixed versions:npm audit fixupdates them.
Editors (VS Code extension 0.7.5, JetBrains plugin 0.7.6)
- Two more notes hold a proposal for review. A proposal with a
"requirement not kept:" note (a repair that no longer sends a request or
makes a check its recording did) or an "unstable recording:" note (a
recording that failed an extra stability proof) is held from Accept
All, as
saffron accept --allholds it. The new dashboard needs no editor update: both show the runner's page.
- Proposals are reviewed in the run report. A scenario with a proposal
has a
Runner
- After a review of the unreleased work.
- An Outline row's date that the page spells out (
Friday, October 30, 2026) is recorded as that row's own date token, so another row never replays on it and the same row follows the day without recording again. A day the page works out from it (a checkout two days on) follows the row too. A recording that kept the date as text records once more. Such a recording is schema version 11, which 1.2.0 refuses. Another row with a different date there fails with its reason; it needs a Scenario of its own. A day shown in a step that does not use the column (today's date in a banner) no longer ties the Outline to one row. - A date the scenario writes itself (
2026-10-30) stays that date when the page spells it otherwise ("Friday, October 30, 2026"), instead of moving with the calendar from the next day on. - A token used as a key in a whole-JSON Examples column is refused instead of being sent as a broken field name.
saffron authormakes the checks a recording makes before it starts the agent, except agy's rules for Saffron's tools, which it does not use: withantigravityApiKeyset and no key it refuses instead of signing in with the Google login, and a failed session prints its reason instead of a stack trace.- A token outside any string in a whole-JSON Examples column
(
{"qty": {captured:count}}) is a value of its own, so a secret beside it can no longer add a field. A cell that is neither JSON nor one whole token is refused instead of being filled in first. The same holds in exported tests. - Captured values in a proposal mask a credential an API handed out.
- A secret inside a whole-JSON Examples column can no longer rewrite the payload.
- "Should differ" compares references and counts (
ABC-123,1 night) as text, refusing only amounts in different currencies or a price against a label, and a brief bad reading no longer ends the check. - A heal that swaps a tick, untick, upload or drag for another input is flagged as a behavior change.
- The sensitivity check gives each check a second, skips a
Thenwith no check or in a session not yet open, and cannot abort a run. The report counts only the proposals it checked. expectDisabledis decided by the recorded element, not a fallback.- Slashed dates are read in the locale's order, a UK date with its weekday becomes a token, and a month the scenario names is not a volatile locator.
- An Outline row's date that the page spells out (
- The npm page says what Saffron is. The package description no longer calls it an AI-fallback test runner, its keywords cover the terms people search for, it names its author, and the README names Antigravity among the agents.
prune --jsonno longer calls a broken file's recordings orphans. While a feature file fails to parse, prune already deleted nothing, but its JSON listed the hidden scenarios' live recordings as orphans. Now they are inorphanScan.ownershipUnknown, beside the parseerrors; other files' orphans stay inorphans, and the JSON has the same shape on a healthy project.saffron status --jsonsplits them the same way, where it listed no orphans at all.- Volatile locators are caught while recording. A date the agent
picked that the app spells out (
October 30, 2026,30.10.2026) is recorded as a{date}token, as ISO dates already were, and a day the scenario's own token wrote keeps that token. A generated id in a selector, frame or test id, a UUID in a name, or a month heading is refused once atsaffron_doneso the agent finds a stable target; what is left gets avolatile locator:note, andaccept --allleaves that proposal to be accepted by file. - An Examples column can hold a whole JSON payload. A request whose body, or a part of it, is the JSON a column the step names holds is recorded as that column, so each row sends its own payload. Before, the payload was kept as fixed fields and every other row was refused. The proposal names the body each row sends, and a cell that is not JSON fails its row by name. Such a recording is schema version 10.
- Comparisons compare like with like. A "should differ" check between
two amounts compares the numbers, so the same price drawn in another
format has not changed. Two values of different kinds fail at once and
say why:
SEK 1,295againstNOK 1,347differ by currency alone, and a whole button label against a price by its label.saffron diffand the report show what each capture read in the proof replay beside its step. A step that says an element is selectable, available or disabled records an enabled or disabled check (expectEnabled,expectDisabled), where it recorded visibility only; such a recording is schema version 9. - Recorded checks must be able to fail. A proof shows a recording
replays, not that its checks notice anything. After a proof passes,
Saffron replays the scenario once more with no AI and runs each
Thenon the page as it was just before the action step it follows. A check that passes there too gets aninsensitive check:note (picking the day that was already picked, a price compared with a value it always had), andaccept --allleaves that proposal to be accepted by file. The report shows "checks proven sensitive in N of M" beside the proofs that passed. - Heals keep what the scenario tests. A heal serves the scenario's
intent first, then the step's, and the step's wording last: it may change
how a step is done, never what the scenario exercises. The agent is told
never to replace typed data with a check of a hint, or a real payment or
sign-in route with a simulator's button, even when the final
Thenpasses, and to fail and name a missing prerequisite instead of working around it. Saffron checks this itself: a healed step that no longer enters, picks or uploads what the committed recording did gets abehavior change:note naming what it checks instead, andaccept --allleaves that proposal to be accepted by file. A value moved to another control (a text field that became a dropdown) is still entered and raises no note. - iframes are recorded with their whole scope. An element in an iframe
inside another iframe keeps both, and one of several alike iframes
(previews, cards) keeps its position, so replay acts in the iframe the
agent used. Before, an iframe picked with
first()ornth()lost its position and replay could act in another one, and an element in a nested iframe recorded no action at all. Code the recorder cannot read, such as an element reached through a variable or an iframe picked by a filter, now fails the recording and names the interaction instead of being left out.saffron_asserttakes the same frame forms. A recording with a nested or positioned iframe is written as schema version 8, which saffron-ai 1.2.0 and earlier ask to be updated for; others keep their version. - Codex and Antigravity on an API key. A machine with no ChatGPT or
Google login, a CI job say, records and heals with Codex on
CODEX_API_KEY(orOPENAI_API_KEY) and with Antigravity onGEMINI_API_KEY(orGOOGLE_API_KEY), billed to that API account. A login on the machine wins over a key, so a key set for other tools never moves your runs off your plan. Whether there is a login is asked only when the first session starts, and only when a key is set, without holding up other sessions; only the vendor saying you are not signed in counts, so a timeout or a network error keeps the login. The run's first line says how sessions will sign in, and the first session says which one it used ("Codex: signed in with API key (CODEX_API_KEY)"); a key stored bycodex login --with-api-keyreads as one, not as a ChatGPT login.saffron login codexandsaffron login antigravitysay which sign-in sessions will use. An expired login still counts as a login: the session it fails says the login won over the key, and how to sign in again or use the key. Codex'sforced_login_methoddecides alone: a key never counts as a sign-in under"chatgpt","api"with no key in the environment or stored bycodex login --with-api-key(a stored ChatGPT login does not count) is reported as a missing key, and a refused session names the setting. Antigravity's equivalent is"antigravityApiKey": trueinsaffron.config.json: the key is used even beside an agy login, by every command, and a missing key is reported and starts no session. A key reaches the vendor alone: an error that quotes it, whole or cut short, shows its variable's name, and it is never written into a recording or report. Codex on a key is verified live; Antigravity on a key is not yet. - A numeric credential is never kept in a recording. A login's token
that is only digits (a one-time code, a PIN) is written as its
{captured:name}in a form field, a multipart part or a JSON number wherever it stands, whatever the field is called. Before, a field whose name did not name an id kept the value. - An API is named by the words that call it. The check that the APIs a
scenario needs are up reads a named API from its recording, or from steps
that call it one ("the orders API", "the orders service"): "I list orders
through the default API" no longer waits on a service named
orders. - A changed contract the API breaks is red. When a recording is checked
again against a changed API contract and the API no longer answers as the
new contract says, the scenario is red and the run exits 1, as before
1.2.0, with no proposal filed (nothing was recorded again). In 1.2.0 it
was filed as an unverified proposal and the run passed. One that passes is
filed for review as before, and its run line says it replayed from the
recording, not that an agent recorded it.
saffron statussays such a recording is checked again on the next run instead of "will re-record", and the Saffron skill describes it. - A malformed
{unique:...}stops the run before it starts. An unclosed token, an empty or invalid name, or one an Examples row fills in with an empty cell ({unique:<id>}) is named and the run exits 2, as a bad{data:...}does, instead of costing an agent session per row. - Failure headings name the row's values.
Check "the customer is on the "Gold" plan" failed, as the detail below it already did. - A form field that is only a number is written as a captured value's
token only where its name names an id or the capture, as in a JSON body:
quantity=100beside order 100 stays 100. - A status code with its own reason names that status whatever follows: "expect 409 Conflict and a retry hint".
Editors (VS Code extension 0.7.4, JetBrains plugin 0.7.5)
- Proposals Accept All skips are marked. A proposal with a "behavior change:", "insensitive check:" or "volatile locator:" note, or one imported from a CI run or not verified, is marked as held (VS Code: "held for review"; JetBrains: "held from Accept All").
- Orphans while a file fails to parse. The Orphans view lists the other files' orphans and groups the broken file's recordings under "Ownership unknown until it parses"; prune waits until it parses. Needs saffron-ai 1.2.1.
- No more endless Loading. A status or diff that never answers stops after 60 seconds with its error, and later refreshes run again.
- After a review of the unreleased work.
The same plain-language scenarios now test an HTTP API, with no browser and the workflow you know: the agent records each scenario once, every later run replays it at zero tokens, a
Thenis never healed, and every proposal is proved by a zero-AI replay before you review it. API test automation andsaffron ejectare both beta: they work and are covered by tests, and they are still being shaped by use, so their recordings and output may change between releases. See "API tests" in the guide.Upgrading from 1.1.0. Nothing to change in a browser project. What can be different after
npm update:- A recording that holds a request is at schema version 6 or 7. saffron-ai 1.1.0 and earlier ask to be updated for it rather than calling it malformed.
@cursor/sdkand@openai/codex-sdkare no longer peer dependencies. npm never installed them, so a Codex or Cursor project installs its SDK itself as before, and Saffron now checks its version: one missing, too old or of another major stops the provider before it starts, naming the command that installs a supported one (npm install --save-dev "@cursor/sdk@^1.0.32").- A few selector forms an Outline row could pass on another row's data with are now refused, so they run red and record again once (see "Outline rows" below).
- A failing check now reads
Check "..." failed: <why>, and a stale recording says what changed, so a script that matched the old wording needs the new one.
Runner
API tests (beta). Tag a feature
@apiand its scenarios test an HTTP API. The recording keeps each request as sent (method, path, headers, a JSON body typed as the API expects, a form or a multipart body with a file of the project) and the response fields later steps need, written as{captured:name}. AThenchecks the latest response: its status, its JSON fields, a header, how long it took, or its body against the pinned OpenAPI contract, and each check is verified before it is recorded. A write (POST,PUT,PATCH,DELETE) is never retried, a proof replay makes records of its own through{unique:...}values, and a run of@apiscenarios alone needs no browser installed.A pinned contract, and several APIs.
saffron api pin <url or file>pins an OpenAPI 3 contract (from a document URL, a Swagger UI page or a file, JSON or YAML, following up to five redirects, each checked as the first, a page's links read against where it landed) asapi/openapi.json, and a recording checked against it goes stale when it changes.api.servicesnames more APIs beside the default one, each with its own base URL and contract; a step names the API it calls and the recording keeps the name, never the host, so--api-urlpoints a run elsewhere. While recording, a write goes only to a path the contract lists or the step names, and the agent hears when a body does not match the contract's request schema.Steps that wait, follow and check more. A request polls with
untilfor a step that waits for a state (a read only, every 100 ms at the fastest, 10 minutes at the longest), keeps a header such asLocation, and takes its own timeout. Field checks find an entry of a list wherever it stands (item), hold for every entry (every), and checkcount,differs,pattern,minandmax. Every duration may be a text with a unit and the step's tokens ("<seconds> s"), so each Outline row keeps its own. The same checks work on the browser's response checks.Fixtures and named operations (beta). A
.saffronfeature tagged@apimay open with oneFixture:block, steps run once per run before the feature's scenarios (a login, a seeded customer), recorded, proved and accepted like a scenario. The values it keeps reach every scenario of the feature as read-only{captured:name}seeds. A step set that sends a request and keeps a value is a reusable named operation.The API from a browser scenario. A browser step can call the API through its session's cookies, and later browser steps can type or check what it kept.
Credentials the API hands out stay out of everything. A login's token, a session id or an API key, read from the words of the capture's name or field, is masked as
{captured:name}in the transcript, the report, the trace, every error, a failed step's page URL and the agent's own view, in every spelling JSON or a URL gives it, and is sent only to an API under test. A browser step uses one only on the app's own origin: never in a page or frame of another origin or a sandboxed one, never in a new tab elsewhere, a dialog's answer or a key press with no target. While recording, the agent sends requests only to the APIs under test, and one it sends again is written as its token in a header, a JSON body, a form field or a multipart part; a request never goes to a link-local address; and a multipart file is never a hidden file such as.env, nor a link or hard link to one, and goes only to an API under test.{env:...}secrets keep every rule they had.Transcripts, reports and traces. An API scenario leaves the requests it sent, and the checks each step made beside them, as
requests.json. The report lists them per step and opens each onto its headers and bodies;saffron tracereplays them, the selected request and response shown where a browser replay shows the page; nothing is drawn while a scenario runs. Each step of a trace is titled with its own keyword (Given,And), and the report names an Outline row by its values beside its number.api.transcript("on-failure","on"or"off") sets how much is kept.Reviewing a recording.
saffron diffand proposals show what a request sends, what it keeps and from where, and what each check expects, so a recording can be reviewed from them. A stale recording says what changed ("step 2 reads ... now", "the API contract ... changed"), in the run's error and insaffron status. A status counts as expected only where the step names it ("should answer 404", "returns 201 with the new customer", "is not found"), never a quantity ("a crate with 500 units", "a product with 500 in stock", "with 500, then").Outline rows stay on their own data. A row value in a JSON or header
patternis escaped and kept whole, so^Order <id>for 1 never passes on "Order 12". In selectors, a row value inside:has(...)or a nested XPath predicate is checked on the element that predicate selects, text under two negations counts as present, and a row value checked before a sibling combinator (+,~) is refused, since the check would land on the sibling. An untypeddiffersreads the number 5 as the text "5".Export to Playwright Test (beta) covers API steps.
saffron ejectwrites requests and their checks as calls on the exported helper, every kind of JSON field check included (the helper carries Saffron's own checker), with extracted values, typed bodies, forms, uploads, polling andapi.services. A schema check is exported with the contract's schemas resolved into the spec and validated with ajv, as replay validates it; a project with one installsajv@^8 ajv-formats@^3, as eject says. It guards and masks credentials as replay does, and writes astest.fixme, with the reason, a scenario that reads a fixture and a browser step that carries a captured credential.Step reuse with the API. A value the agent kept while recording reaches the reused steps after it, so a new scenario built mostly from known steps records without going back to the agent. A reused step whose recording no longer matched is reported as re-recorded only when the agent did record it again.
No agent session that could not help. When the app or API under test does not answer at all, a scenario that would record or heal is red with "no agent session was started" and why, instead of spending tokens on a server that is not there (browser scenarios too). Only the APIs a scenario names are checked, so an unrelated one being down blocks nothing; any port is checked; behind a proxy no check is made. An API
Thenthat fails after every request answered as recorded is red at once, since aThenis never healed. A changed contract is checked again by a zero-AI replay of the same recording and filed for review, instead of being recorded again. A run with--no-agentsays so where it needed the agent.Reviewing and reading results.
saffron difftakes a feature file for its pending proposals, and the run's hint says to review with it. A failure and a transcript check name an Outline row's values instead of its<column>, and a null check reads "equal to null". Chronic advice counts only reds that cost a session, for scenarios the run ran, and the "vs previous run" line names scenarios that turned green. The agent counts steps from 1 in what a person reads, as the runner does.saffron initignores.env. The.gitignoreentries it writes include.env, where{env:...}secrets are kept.Agents know about API tests.
saffron mcphas anapi_contracttool that lists every API's operations, the Saffron skill covers@apiscenarios (references/api.md), and the instructionssaffron initwrites and the Claude Code and Codex plugins describe them.Internal. The runner is split along the line a second kind of test crosses: the orchestrator owns what every medium shares (freshness, proposals, proofs, heals and the verdict) and drives the browser and the HTTP medium through one contract. Nothing changes for a project.
Editors (VS Code extension 0.7.3, JetBrains plugin 0.7.4)
- An API scenario's requests in Last Run. Last Run never opens an API scenario's request transcript as a screenshot: only failure and before-heal pictures open as one, as the runner picks them. Such a row offers Open Replay, which steps through the requests it sent. Needs saffron-ai 1.2.0.
Upgrading from 1.0.1. Nothing to change in a project. What can be different after
npm update:Row value boundaries are stricter: a regex atom that may match nothing or anything (
\W*,.+,[^,]+), a lookaround, or a group edge or alternation next to a row value (Order <id>(?:st|)) no longer drops the value's boundary, and row values are matched in every selector-chain container. A row value at the edge of a role's name, a label, a placeholder, an attribute (alt,title,[aria-label*=...], XPathcontains(@href, ...)),:text(...),:text-matches(...), an innerhaslocator or a frame's selector is whole too, as in text: row1never findsOrder 12. Beside a digit, a decimal or thousands part counts as part of the number (1is not in1.5or1,000) unless the pattern writes it (<n>\.\d{2}), and a pattern with a quantifier right after a row value (<id>+) is refused. Text a selector says must be absent (:not(...), XPathnot(...),has-not-text) is no longer looked for as present. A recording that passed in 1.0.1 because of that can turn red afternpm update; it never turns a failing check green. Re-record what fails (saffron run --rerecord <file>:<line>).Recordings that use JSON response field checks, or single-quoted row values in any part of a selector chain (an XPath part first too), are format version 5. Saffron 1.0.1 refuses them by name, so a team updates together.
An Outline with
As "<who>"steps recorded before 1.0.1, where a step types a value of that person's own, records again once on another person's row (red under--no-agent, saying why). The new recording then refuses the other rows by name, as 1.0.1 recordings of that shape do: give each person's values a column, or each person a Scenario.saffron runexits 2 when the files it is given hold no scenario (only step sets, or nothing yet), where it said "0 passed" and exited 0.Amount checks read a minus spaced off at the start (
- €5.00) or after a currency (CHF-5.00) as negative, and a hyphen after another word (ID-123) as no sign.Agent-blind secrets: the recording agent no longer sees
{env:...}values. It is given the token and types the token; Saffron's browser tools put the value into the page, and every tool result (snapshots, echoed code, console, network,browser_evaluate) and every Saffron reply comes back with the value as its token, as do the prompt's base URL and a heal prompt's replay failure and page address. That covers a snapshot line Playwright single-quotes, a value Playwright's call log cuts with "…", and a downloaded file quoted in a reply. A token counts only where the browser types, picks, opens or uploads it ({env:...}inbrowser_evaluatecode is refused), and a word the scenario or a data value it uses writes with a short value glued in (qa{env:PIN}) is shown back as written. The recording reads the browser's own results, so a value quoted in Playwright's echoed code is masked to its token there too. This holds for Claude, Codex, Antigravity and Cursor alike. The guide has a diagram of the flow and lists what remains: values under three characters, a page that shows the value transformed or only part of it, code that reads a field the value was typed into, same-text matches masked along with the value, and a check on the value shown in another case.saffron ejectwrites the committed recordings as Playwright Test specs you own from then on: one spec per feature file, each Gherkin step atest.step, a helper file beside them, andplaywright.config.tswhen there is none. An exported test is never looser than replay: it may fail where replay would pass, never the other way, and what has no faithful form yet (amount checks, patterns or selectors holding a row value, JSON response checks) is written astest.fixmewith the reason.{env:...}values are read when the test runs and never written into a file. The exported checks read an element's text exactly as replay does, so CSStext-transformor text split over blocks cannot pass there and fail in replay. Two feature files that would share a spec file stop the command before anything is written.UI tests can check JSON response fields by path, including nested fields and array entries. Equality checks preserve string, number, boolean and null types; presence checks distinguish a missing field from null. Failures name the field and expected and actual values, with secrets masked. These checks replay without AI and Then assertions stay strict. Recordings using them require format version 5, so older runners refuse them with update guidance instead of skipping the JSON checks. A field's path is kept as recorded: a row value that sits in it by chance (
/errors/0) never makes another row check another field.Match row values in every selector-chain container, including single-quoted text. Such a recording is format version 5, since 1.0.1 would fill a single-quoted value in without a row boundary.
A row value keeps its boundary in a pattern next to an atom that may match nothing or anything (
\W*,\d*,.+,\S+,[^,]+,\P{L}+), a lookaround ((?!-),(?<!-)), or a group edge or alternation ((x|<id>)):Order <id>.+fails row "1" on "Order 12 shipped".An XPath selector (
xpath=//button) replays: it was read as a regex with the flags "button" and threw.An Outline with
As "<who>"steps recorded before 1.0.1 records again once on a row of another person, instead of replaying the first person's values; the new recording refuses the other rows by name.Amounts: a minus spaced off at the start (
- €5.00,- 5,00 €) or glued after a currency (CHF-5.00, as Swiss formats write it) is negative, and a hyphen after another word (ID-123) is no sign.A run of feature files with no scenario in them exits 2 and says so, instead of "0 passed". A step set file that does not parse is named in the error. Feature edits apply to files written with CRLF and keep their line endings. A recording is written beside and renamed over, so a stopped run never leaves half of one.
Mask case-changed encoded secrets in capture names, reports and traces.
Read labelled counts such as
Guests (2 pax)as positive, also where the label is a currency code in lower case (Pens (2 pen)). Accounting parentheses still denote negative amounts with currency notation.
Upgrading from 1.0.0: some recordings can turn red. Checks are stricter, so a recording 1.0.0 passed can fail after
npm update, with nothing changed in the application. It never turns a failing check green. What changed:- A role's name matches exactly (a partial name needs an explicit name regex), attribute patterns are case-sensitive, and an input-value check needs a visible input.
- A text or input-value check records an explicit expected value. Expected text, a text pattern, a download check, an expected URL, or a target's text, label or placeholder that a blank Examples or data value leaves partly empty fails instead of matching more than it means.
- A response check counts only requests started since the latest browser action, so an earlier successful save no longer satisfies it.
- An Outline recording that cannot follow another row (short values,
dates without a year, a role taken from a row, an unscoped action,
check or capture in a section a column names, a value typed in a
column's browser session, a step naming a column that recorded no action
writing it, a state check naming a column whose value it does not show,
as in
Then <user> sees the dashboard) refuses that row with guidance instead of passing on row 1's data. Login Outlines of that shape now pass only on the recorded row: write the user in the text the check expects (Then the dashboard greets "<user>") and record again.
Re-record what fails (
saffron run --rerecord <file>:<line>) or correct the step the message names. Recordings 1.0.1 writes with its new features (localized dates, several downloads per click, the new row bindings) are refused by 1.0.0 by name, so a team updates together.Agent plugins
Declare documentation, privacy, license terms and support links for the Claude directory listing, with explicit README links as a fallback.
Prepare the agent plugin listings with the Saffron icon and documented data handling. The Codex listing adds a shorter subtitle, local execution requirements, prerequisites and links to support, data handling and the software license.
Runner
A row refused because its check only shows that something is visible or in a state (
Then <user> sees the dashboard) now says so, and suggests writing the value in the text the check expects (Then the dashboard greets "<user>").A fallback selector's text (
:has-text("Order <id>"),text=Order <id>) finds row 1 only as a whole number or word, not inside "Order 12", and a blank row value at its edge is refused, as in other checks.A pattern that joins letters or digits to a row value itself (
^<prefix>\d+$) keeps them joined at replay.A secret shown lower-cased and percent-encoded in a URL is masked, and so is a number secret shown with spaces or dashes between its digits.
A day a step writes as plain
{date+1}and the app shows in its own format (21/10/2026,Oct 21, 2026) is recorded as the token in that format, so the check moves with the calendar.A quoted CSV field keeps its own spaces. A double-quoted
.envvalue takes\"and\\escapes. A count in parentheses ("Cart (3)") is no longer read as a negative amount.Mask mixed-case percent escapes in capture names, reports and traces through the shared secret masker.
Keep state checks bound to their recorded row even when the assertion step also presses a key or performs another browser action.
Match row values with Unicode-aware regex boundaries, including selector fallbacks and downloads. Non-ASCII suffixes cannot satisfy a shorter value.
Refuse tokens inside regex repeat counts instead of silently changing the assertion. Use a separate Scenario with a literal repeat count.
Mask URL-encoded secrets with lowercase or mixed-case percent escapes, including escaped patterns, before saving recordings.
Bind state checks to their recorded Outline row when the check does not follow its column. A column at the start of the sentence no longer bypasses this safeguard.
Apply row-value boundaries to selector regex fallbacks and download checks. Preserve character-class syntax when resolving regex tokens.
Classify Given and inherited setup steps using the feature's language, including Latvian keywords shared with other languages.
Rendered-text capture tests allow time for capture stability and browser round trips on CI. Hidden and empty text must still fail within the same short test timeout.
Runner CI installs WebKit alongside Chromium and Firefox so the existing tests for all three browser engines can run on GitHub Actions.
Review fixes: a provider or agent error is masked for secrets in full before it is cut to its first line, so a secret spanning lines no longer shows in the console or a later scenario's error. A typed value submitted with Enter presses it on the field typed into, not whatever has focus after an input handler ran. A date token with a locale tag Node.js rejects (
en-x) is marked in the editor instead of stopping the diagnostics. A report row'sranAtis validated like the run's times, and the HTML page is rendered before either report file is replaced. A quoted value containing a word like "clear" no longer hides a lost typed value. An amount with its minus before the currency (-$5.00) reads as negative, so doubling it is never checked against $10.00. A capture named like a token (price {date}) keeps its name in an amount check at replay. A click that starts several downloads stores the count as a number of its own, which no Outline pass rewrites, and the recording is written as version 4, so an earlier runner refuses it instead of checking the first file. Recording judges a download check by replay's blank rule and resolves{unique:...}inside Examples values, as replay does.Fourth review round. Outlines: a step that names a column but recorded nothing that writes it (the page already showed "myself", or a box was ticked) binds the recording to its row, so the "someone else" row is refused instead of replaying row 1's page; a recording made before this is recorded again once on a row that differs, and the agent is asked to pick the row's value even where the page already shows it. A blank first row no longer refuses every filled row: a filled row records once more, and that recording serves the blank rows. A typed value equal to the section's name no longer passes for scoping where the step says it is a section (
in the "Double" room card); captures and an assertion step's checks must name the section too, andthe country selector in "<section>"and a hyphenated container word (time-selection dialog) read as sections. A dialog's button the scoped action opened, scoped to the dialog, and a wait, need not name the section. Literal braces in a typed value (a JSON address) are text in a person's browser session, and a short{data:...}value the step writes is the step's own. A date cell written with its month's name ({date+1:MMM D}) follows the row in any step, instead of being recorded again every day. Recording: performing several flagged actions again replaces each in its own place, one by one, and one not performed again stays where it was. A name Playwright's MCP shortened ('26. Oktober'for "26. Oktober 2026",'Room'for "Room 101") is recorded in full from the snapshot the agent clicked in, as replay matches names exactly. Through the trace bridge, no page fingerprint is taken before a browser action, which had made every click by snapshot ref fail. Replay: a captured value is taken once the page has stopped changing it for half a second (sooner for a short timeout), so a total drawn again after the last change is not read stale. Proposals: a proposal git checked out with CRLF line ends is still the one this machine filed. Review: a proposal whose only change is beside the steps (a row binding, the row it was recorded on, a data value) says so insaffron diffand the report.saffron statusjudges an Outline row the recording refuses as refused, not stale. A CI run's results say when the run was interrupted. Step reuse records afresh instead of keeping a capture under another value's name, and a fix carried to other scenarios never copies a<column>into one without it. The editor checks date tokens in doc strings too, never on a comment line.Fifth review round. A step naming a column is judged by its own actions: another step writing the column, or another column sharing its value, no longer lets an empty step pass on other rows. Step reuse never puts such a step back for a row whose value differs, so the row records it afresh. A check on a bare role (
the error message in "<language>") takes no section from the quote, and a wait in a section step never binds the row.saffron statusmarks a refused row with ✗ and says why, instead of calling its recording unreadable and counting it as replaying.Sixth review round. Recording: a full button name is read only from the page snapshot a browser call shows, never from a response body or the console. Corrections a step performs again, one by one or in one call, each take the place of the flagged action of their kind and element, so a fill stays before the Save after it; one that reopens a dialog replaces the click that first opened it instead of clicking twice. A check spelling its own value (
Then ... "Guest: Tor") keeps it, and no longer reads another step's table cell. A value typed from{data:users.<who>.email}is recorded as that reference, so an edited data file reaches replay.the "<room>" room card should show ...is a section without "in". Replay: a capture waits for the page's fetch and XHR requests to be answered (up to 3 seconds) before reading.Seventh review round. One value two references in a step read (shipping and billing both "Home") stays as typed and binds the row, instead of becoming one of them; in different steps, each keeps its own reference. An alert or other bare-role check inside a section the step names explicitly must be scoped there. Step reuse keeps out a
Whenstep that checks something but sets nothing of its column. A dialog reopened while correcting a step replaces the click that first opened it, across the waits between them, and a wait performed again is filed once. Only the very same action counts as reopening (the same target in the same scope, with the same click options): a click focusing a field never replaces its fill, nor a click on a checkbox of the same name. A short shared value ("NY"), a blank one, or one written escaped in a pattern or selector, binds the row like any other. A{data:...<column>...}value blank on the recorded row records once more on a filled row, as a blank cell does, and one kept in another spelling (ship-home) binds the row. A recording made before these bindings that still holds such a value literally records once more on a row reading another. A later check still holding a value another step's reference read (a summary showing "Home") binds the row too, and a step that reads a value and also spells it as its own constant (Shipping "{data:...}" and Billing "Home") binds the row, as which field was which cannot be told. A short value ("M", "NY") counts only as a whole word, never inside "My cart", and one inside the step's own text ("Shipping to NY") becomes its reference. A recording made before this that typed the reference for the step's own constant records once more. A value the step spells inside a longer word ("Johnson" beside a first name "John") binds the row too, and step reuse never puts back a check still holding a value a reference read on a row that differs.Twelfth review round. Recording: a data value never rewrites a
<column>placeholder or a{data:...}reference's path ("email" in<email>), and a tag in a selector (button) is never data. A value a step spells itself stays its own when another step reads it ("Johnson" beside a first name "John"). A check naming a column nothing in the recording follows (Then the banner is <state>, recorded as "not visible") binds the row. A role, an amount written another way ("12,50" checked as 12.50) or another spelling (address-home-street) of a{data:...<column>...}value binds the row too, and a password read through{data:users.<who>.password}from{env:...}follows the row instead of keeping the first row's variable. Step reuse: aWhenrecording never seeds aThenof the same words, and a check holding a value a joined key (zone-<s>) read, or expecting nothing for a blank one, is never put back, also from a recording that names no row. Replay: a{date...}inside a data value resolves to its day, as it did while recording. The review that followed: secrets are masked before any other pass, so a row, table or data value inside one never cuts it up and leaves the rest in the recording. A state check ("not visible", "disabled") naming a column binds the row even where another step sets that state, and so does a check showing a date column's day in the page's own format; a check stating its own text may name a column to describe the row. Another step's value in an action step ("Ship to NY") binds the row, a session's column (As "<who>") is never cut out of a longer word ("Hannah"), a data value holding a formatted date becomes its reference, and a constant spelled inside a longer word stays however often the step names a reference. A changed data file is noticed where a check holds the old value as a role or an amount. Step reuse refuses joined keys without a separator (zone<s>) and empty checks in any spelling (^(?:)$), and a recording whose data holds a date is version 3, which saffron-ai 1.0.0 refuses rather than typing{date+1}.Thirteenth review round. Outlines: a row value inside a run of letters ("red" in "Delivered", "Pro" in "Products") no longer binds the row; a camelCase part ("SelectedRed") still does. Rows whose data reads blank on every row no longer record each other forever. A row value holding a secret's value ("admin@shop.test" beside a variable "admin") is the row's
<column>, and a secret equal to a row's lookup key leaves the step's{data:...}reference alone. A state check ("Promo" hidden, an attribute's pattern) binds the row whatever its element is named. A dialog answered after announcing the next step is armed before its trigger, in the trigger's step. The fingerprint a row binding keeps of a secret is salted and slow, so a short one is not read back from the recording (version 3); older plain ones still match. A blank Examples cell beside a step with something to check is version 3, and step reuse refuses a one-character joined key (zone0). An older recording is recorded again once a data value it reads holds a date. Replay: a row value starting or ending an expected text, URL, response URL or text target is never found inside a longer word or number ("Order 1" never passes on "Order 12"). A pending proposal that fails to replay does so in a browser of its own, so the agent records in a clean one. CLI:saffron prune --yesnames what it could not delete and exits 1; a hook killed by a signal says so;.envreadsexportlines and drops inline comments; "$.50" is 0.50. The language server runs a scenario sharing its name with another by line, readsSAFFRON_ENVfrom the project's.env, and checks date tokens only in steps, tables and doc strings. Then: a check that an element found by its text is absent ("Promo" not visible) binds the row like any state check; a role with its text never finds a row value inside a longer one either; re-masking a secret never cuts into a<column>it sits in ("admin" in<adminEmail>); and "-$.50" and "($.50)" are negative. The review after that: a secret percent-encoded in a recorded URL (admin%40shop.test) is masked too, and a URL check also reads the page's URL decoded; another step's data value shown in another case or as a slug ("MUG") binds the row; two columns holding one{env:...}password record once more for a row typing them apart instead of being refused for good; a column opening a Then step that another step writes (Then <user> sees the dashboard) describes the row; a three-letter data value inside text ("1 x Mug") follows its reference. A response URL pattern bounds a row value wherever it stands,saffron statusjudges every Examples row and finds an install in a parent folder, a pending proposal failing for another row keeps its screenshot and trace, a quoted.envvalue followed by a comment loses its quotes, and an empty Given in any language is not sent back. The full review after that: a secret percent-encoded in a pattern, written in another case, or in anencodeURIURL is masked; a data value that is a variable follows its reference inside text too; an amount a text check shows in another format, a state check found by its text (visible or not), and a number only inside a longer one are each judged as they should be; a column opening a check is its subject only when another step writes it and the check does not state it (<state> is ...); a multi-word value in a URL (q=red+shoes) follows the row. Replay bounds a row value in every pattern (text, name, attribute, response URL and body) and refuses a blank one left inside an attribute or body pattern, reads response URLs decoded, and keeps a regex's own named groups. A fresh recording that never announced its last steps is sent back instead of filed as verified. A quoted CSV field after a space is read whole, and a row with more fields than its header is refused..env: the last of a repeated key wins, quotes match non-greedily, and inherited names are ordinary keys; neither data nor rows read inherited properties.saffron statuscounts unreadable recordings apart from stale ones,saffron traceprints report names without terminal escapes, and a run without the agent says so. A click performed twice on purpose while correcting a step is kept twice; a reopened download trigger takes its download wait with it, so the count is not doubled; and an armed action goes before its trigger even when the same call waited for something after it.An Outline whose browser session comes from a column (
As "<who>" ...) no longer lets another row's person type the recorded person's value: a step in that session typing a value the step does not write binds the recording to its row (written as version 3), so the other rows are refused with guidance instead of passing on the first person's data.Third review round. Recording: each action and check in a section a column names must be scoped to it (one scoped action no longer covers an unscoped fill beside it), a quoted value the step types is no section, and performing a flagged action again replaces only that action, in its place, with its download wait. A download is counted from the browser's own events only, never twice. An amount the step writes itself stays the step's, a column two written amounts match stays literal, a date cell's day stays literal only in the steps that name its column, tokens that always read alike (
{date+7:dddd},{date+14:dddd}) are one, and "an empty" field reads as nothing to type. Format: a recording made before a new-syntax date token was added to its scenario is recorded again once as version 3, a "container" row binding is version 3, androwsUseNewDatesis part of what a proposal binds. Replay: a download check, an expected URL, or a target's text, label or placeholder that a blank row value leaves partly empty fails instead of matching more; a newer dialog answer replaces one no dialog used; a targeted upload that starts a download is waited for; a retry sets the chosen files again; an amount check's quoted names are never resolved. Runs:--rerecordrefuses, deleting nothing, when no browser of the run can record or the agent cannot start; an offset time zone is refused; "retry in 41.8s" is a passing throttle; Claude overage use is no limit.Second review round. Recording: a section a column names that the recording ignores binds the row, so another row is refused rather than acting in row 1's section; a place or manner (
in "Oslo") is no longer taken for a section, and aGiventhe page already satisfies is not asked for an action. Amounts: accounting, trailing and en-dash negatives, decimals such as0.125, and an amount written in another format than its Examples cell follow the row; a misplaced quote in an expression is refused. Dates: a day the recorded row's cell holds stays literal, two tokens that read alike stay literal, only whole dates are made tokens outside their own step, and an Outline whose other rows use the new date syntax is written as version 3.{date-of-birth}is typed as written, offsets past a hundred years and locale-like typos are named, and a time zone must be spelled as browsers spell it. Downloads: one started in a popup tab or after its click returned is credited to that click, one reported by a later result raises the count, opening a file's URL and a chosen file that starts a download replay, and a response body is never read as actions. Runs: a throttle that clears in seconds does not stop AI sessions, "Quota exceeded" and a Claude subscription limit do, and Ctrl-C before any scenario finished keeps the last report.Amounts that add up. A step such as
Then the basket total should be "before" plus "breakfast"records anexpectAmountcheck: the target must show the amount an expression over captured values states ("before" + "breakfast","total" - "discount","night" * 2), compared to the cent. Amounts are read in either decimal style (NOK 1 234,50,$1,234.56,CHF 1'234.50); an element showing two numbers is refused rather than guessed. A total that changed by the wrong amount now fails, where a "differs" check passed. The agent records it withsaffron_assertkindamount, which refuses a name no step captures. Capture names in the expression are identifiers: no Outline, data, date or secret pass rewrites them, and a secret-bearing one is renamed together with its capture.Agents are told that other scenarios reuse only accepted recordings: the run's "proposals pending" line, the
project_statusMCP tool and the bundled skill say to accept a shared setup before recording more scenarios that use it, instead of paying for it in every one. The skill also asks for a mid-scenarioThenat each checkpoint of a route (a 3DS challenge, a redirect), the exact control a variant uses, and the section a step acts in when labels repeat.Fixes from a field report on recording a checkout suite with Codex.
saffron_donerefuses a pass once while an action step recorded no browser action: announcing a step performs nothing, and a step's action can land under the next step. A step that acts in a named container (in "Your contact information") with a target that ignores it is sent back too, since replay acts on the first match; performing it again replaces the unscoped action. The step reply and the agent's instructions say that announcing performs nothing, ask readiness steps to record the condition they wait for, and ask for a fresh snapshot when a ref no longer resolves. A second pass stands, so a step that needs nothing still finishes.The lost-value check no longer flags a step that clears a field or passes
"": a quoted field name is not a typed value. An empty fill on a step that types a value is still flagged. A blank first Examples row binds to<column>, and an expected message held in a row's{data:...}token follows each row.--rerecordrecords afresh over a pending proposal from an earlier run, not only a row-bound one, instead of replaying it at zero AI. Rows of the same run share the new recording.When an agent session fails on its provider's usage limit, no further agent session starts in that run, on any worker (the agent describing such a message in the application does not count, and the message is stored with the scenario's secrets masked): the scenarios that need one fail at once with the provider's message, and replays carry on. A session in which the agent made no tool call at all says its tools did not reach it.
The run summary counts pending proposals whose proof replay failed apart from the others, and the report splits them into proof passed, proof failed and not proved. A run of named scenarios marks each report row kept from an earlier run with when that run started; shards of one run are not marked. A pending proposal that does not replay for another Examples row names the columns that row types differently, and which of them the failing step uses, instead of guessing at the cause.
Typing with submission checks Enter in its own response window, so a successful autosave cannot hide a failed submission during healing.
Completed form fields and typing stay in the recording if a later field or submission fails, including when the agent retries only the failed part. Compound calls are validated before their first edit.
Agent shutdown allows five seconds for started calls to finish recording, then cancels stuck work and fails the incomplete recording with a clear timeout. Queued calls stay blocked and provider usage is retained.
Healing checks each field of a multi-field form against its own response window, matching replay. The full batch is validated before any field changes. Capture tracking preserves named regex groups when step parameters change and prefers a unique test ID over app-generated DOM IDs for remounts.
Healing response assertions follow the latest agent interaction in their browser session, so an earlier successful retry cannot hide a later failure. Agent shutdown waits for started browser calls to finish recording before closing their clients or finalizing the failed session; queued calls stay blocked.
Codex recording and healing enable the code-mode tool host while retaining Saffron's isolated browser tools. Tool-call limits are enforced at the gateway, including calls nested in code-mode batches, before an excess call can reach the browser. No dependency patch is needed.
Text patterns reject partly blank row, data and environment values, including nested tokens. The healing assertion guard retains captured element and frame identity, so another frame cannot stand in for a removed target.
Follow-up assertion fixes: redirects retain the original request's action window, and partly blank text is refused even when the blank comes through an Examples cell or data file. Live comparisons retain the captured target and frame when that target becomes hidden or disappears. A unique ID or test ID allows ordinary DOM replacement in the same frame; duplicate matches remain tied to the original element.
Response assertions check requests started since the latest browser action, so an earlier successful save cannot hide a later failed save, even when the earlier response arrives late. Waits and consecutive assertions keep that window open. Check a request before the next interaction or navigation.
Assertions use exact role names and case-sensitive attribute patterns; input value checks require a visible input. Use an explicit name regex for a partial role name. Text expectations containing a blank row or data value alongside other text now fail with guidance. Recording a text or input-value assertion requires an explicit expected value, including
""for an intentional empty check. Existing recordings relying on these looser checks must be corrected or re-recorded.Outline recordings refuse other rows when they retain short values in action steps, dates without a year or glued to digits, an unparameterized blank cell, or an element role taken from the recorded row. A whole
{env:...}or{data:...}field equal to a row's value is treated as ambiguous even when the step names that token. These cases fail with guidance instead of passing on another row's data or repeatedly recording.Step reuse refuses changes to quoted arguments containing several placeholders, where it cannot safely separate their values. Row comparisons keep date tokens inside environment values literal, matching replay.
Setup hooks can create or refresh files used by
{data:...}steps. Data and referenced environment variables are validated after setup, and replay reads the refreshed values. Invalid data stops the run before a browser opens and still runs teardown.Ctrl-C during
saffron runwrites the report of what finished, with or without setup and teardown configured, instead of leaving the last run's report in place. The report has"interrupted": trueandnotRun, the scenarios the run selected that did not finish, which count as neither passed nor failed; the HTML report, the run's output andsaffron statussay it is partial, and its history line is marked partial. The browsers close and teardown runs first, then Saffron exits with code 130. Playwright's own Ctrl-C handler is off, so it no longer exits before teardown has run.saffron report --mergerefuses an interrupted shard, and a run of named scenarios replaces an interrupted report whole.The run's first line and the report's
provenancesay what produced it: the provider, each model asked for and whether it came from--model,--heal-modelor saffron.config.json, and the installed version of the vendor package (@openai/codex-sdk,@anthropic-ai/claude-agent-sdk,@cursor/sdk) beside saffron-ai's. A model nobody asked for is left unnamed; the models that answered stay intotals.models.Dates as the app shows them. A
{date...}format writes month and weekday names (MMMM,MMM,dddd,ddd) and two-digit years (YY), in the locale the token names ({date+7:D. MMM:de-DE}is "24. Okt.") or indateLocalefromsaffron.config.json(en-US when absent). A range is one token,{date+7 - +9:MMM D, YYYY}: what is written between the offsets is written between the days, and what both share is written once ("Oct 24 - 26, 2026", "Oct 30 - Nov 1, 2026", "Dec 31, 2026 - Jan 2, 2027"). A suite no longer needs a setup script to compute the stay it picks and the range it checks.One clock.
timezoneinsaffron.config.jsonsets the calendar date tokens count days on and the time zone every browser runs in, so the page and the tokens agree on today, around midnight and through clock changes. Every token of one replayed scenario resolves on the clock the scenario started with.The agent is given each date of a scenario resolved, on the clock its recording started with, instead of the token; the recording turns each of those days back into its token, so a date written in a step's text in any format moves with the calendar, not only an ISO one.
A
{date...}token Saffron cannot read, or with a locale this Node.js does not know, stops the run before any browser opens, naming it and the scenario; the language server marks it in.featureand.saffronfiles. The TextMate grammar colours ranges.A recording with a range, a locale, a name or a two-digit year in a date token is written as version 3 of the recording format, so saffron-ai 1.0.0, which would resolve it wrong, refuses it by name.
What a download says. A booking confirmation's "Add to Apple Calendar" link could only be checked by its
hrefand file name, never by the dates and identifiers inside the file. Now the click that downloads a file records with the download armed before it (waitForDownload, as a dialog is), and replay waits for that download, fails the click when none starts, and keeps the file in.saffron/artifacts/<feature>/<scenario>/downloads/. AThenstep such asThen the calendar file should contain "DTSTART;VALUE=DATE:{date+7:YYYYMMDD}"records a content check (expectDownload): text the scenario's latest download contains, a regular expression it matches (case-sensitive,^and$at each line), or both, with{date...},{data:...},{env:...}and Examples values resolved. Line breaks read as\n, a calendar or contact file's folded lines read whole, and only UTF-8 text files up to 5 MB are checked. The check is never healed. While recording, Saffron checks it against the file the agent downloaded (saffron_assertkinddownload) and shows the agent the start of the file when it does not hold. A click that downloads a file is no longer lost from a recording made without tracing: Playwright's MCP, attached over CDP, could not copy the file and reported the click as failed, so nothing was recorded. Saffron now keeps the file through its own connection and drops that one error. A large page no longer hides a download from the recorder either: a tool result cut to fit the agent's context keeps the sections the recorder reads (its code, tabs and Events) whole, even where the cut would split one, and the parts of a compound call share one allowance. A click that starts several downloads records how many; replay waits for that many and checks the latest, the file the recording checked. While recording, a download check resolves{env:...},{data:...}and{unique:...}values, dates on the recording's clock, and values in a pattern escaped, exactly as replay does. A recording with a download needs this release; an older runner refuses it by name.
VS Code extension 0.7.0 and JetBrains plugin 0.7.0 (with saffron-ai 1.0.0): a CI run's row says what it left to view, and a results panel shows a run as one page, with its screenshots and buttons for its traces, scenarios, report and Import (VS Code: Show Results; JetBrains: below the runs in the CI Runs tab).
Runner
- Capture names containing resolved environment secrets become stable opaque identifiers in recordings and proposals, with every comparison updated to match. Ambiguous names are refused instead of merging distinct captures.
- Live captured-value comparisons search child frames when no frame selector was recorded, matching captures; explicit frame selectors stay scoped. Comparisons select the first displayed non-empty value by locator order and keep that source while polling, rather than passing on another frame or fallback merely because its value differs. Removing a hidden duplicate cannot shift the comparison onto another price. A removed or replaced selected element fails the comparison rather than switching targets.
- Captured-value comparisons use displayed text, ignoring hidden descendants. Changing only hidden text no longer passes a price-change assertion. Captures and comparisons against a live element require a non-empty displayed value; hidden or blank elements do not prove a value changed.
- Browser sessions. A step that starts
As "buyer"runs in a browser session of its own: a browser context with its own cookies, storage, login and tabs. The session carries over to the steps after it until anotherAs "..."names one; steps before the first one run in the default session (As "default"goes back to it), and the name may come from an Examples column. A step that reads as a session step and names none (As "" ...) stops the run with its file and line;As "Jane", I log instays an ordinary step. One agent records, heals and refines across every session, its browser tools following the session of the step it is on (every provider); replay opens each session the first time a step needs it and goes back to the page a session was on. Tab indexes and network checks count within a session. Recordings store nothing new: a step's session follows from the step texts, so recordings made before stay valid, except for a step whose text already beganAs "name"(with a space after the quote): it now runs in that browser session, which starts fromstorageStateunlesssessionsgives it a login of its own. Reword such a step, or configure the session. - A login per session.
sessionsinsaffron.config.json({ "buyer": { "storageState": ".auth/buyer.json" } }) starts a session signed in as its own person; a session without an entry takesstorageState, as does the default session. Every file is checked when the run starts, and so is the shape of each entry: a misspelt key stops the run instead of starting that person as the default one, as does asessions.defaultnaming a file other thanstorageState(the default session never reads it). When the whole suite runs, a configured session no step names gets a warning, and when two or more sessions the steps name have no entry whilestorageStateis set, the run warns that they all start signed in as that one person. - A trace per session. Each session is traced on its own
(
trace.buyer.zipbesidetrace.zip,proof-1.buyer.zipbesideproof-1.zip), listed under its session's name insaffron trace, which opens on the session a failure is in and marks the other sessions' steps as theirs. The report tags the step where each session takes over and names every trace file. - Editors. Completion, hover, the similar-wording hint and the step
vocabulary (
saffron steps, the MCPsearch_steps) offer a step without itsAs "...", so picking one never moves it into someone else's session. Accepting a completion replaces the step's text, before and after the cursor, keeping the keyword and anyAs "...", and each suggestion's badge is the status of the step as it will read there. A StepSet's preview keeps each step'sAs "...". - Step reuse stays in a step's browser session, whether the step names
it or carries it over from an earlier
As "...": another person's recording may type that person's own values (an address, a card), so it never seeds a step for someone else, not even through a quoted value. - A duplicate-wording rename, or a step edit the agent suggests, never
adds, drops or changes a step's
As "...";saffron accept --with-feature-editskips an edit that would, with the reason, andsaffron diffsays so beside it. - The test folder is yours to name.
saffron initcreates the folder thefeatureskey names, andinit --examplesinstalls the example suite into it, with its data indataDir, instead offeatures/anddata/.saffron status --jsonnames the folder asfeaturesDir. - CI runs say what they left. On GitHub,
saffron runsmarks each run with its results and their size, a summary only, results expired, none uploaded, or still running, from one listing of the repository's artifacts (resultsin--json, with the words both editors show asresults.label). Azure Pipelines runs say so when opened. - A summary of every run.
saffron export --report-onlywritessaffron-summary.zip: the last run's report and screenshots, no proposals and no traces, small enough to upload on every run. Viewing a run (report --run) uses, for each shard and attempt, its full bundle when there is one and its summary otherwise (bundle.summaryOnlywhen only summaries were left); importing a run that left only summaries says there is nothing to import. A full bundle, or a later attempt, stands only for the same browser, base URL and shard: beside another browser's or environment's results the view and import refuse the mix by name instead of hiding a failure. - A CI run as one page. Viewing a run writes its results page beside
its report (
panelHtmlinreport --run --json;--panelprints it): totals, failed and pending-review scenarios as cards with their screenshots, what passed, the proposals, and buttons an editor wires to Open trace, Open scenario, Open report, Open run page and Import. - Agents other than Claude Code (Codex, Cursor, Antigravity) have their tool calls run one at a time in the order they arrive, as Claude Code runs them, so a step announced before a click is in effect when the click runs; a call the vendor gave up on while it waited never starts.
- The Saffron plugin for Codex. The plugin repo that serves Claude
Code is now a Codex marketplace too:
codex plugin marketplace add s-chathuranga-j/saffron-claude-plugin, thencodex plugin add saffron@saffron, installs the MCP server and the skill. Codex plugins have no language server slot, so diagnostics and navigation stay Claude Code only. - Node.js 22.12 or later. The command-line library Saffron depends on needs it, so an install that enforces engine versions failed on the Node 20.6 the docs promised. Node 20 is past its end of life.
- Playwright 1.64. A recording session refuses
browser_tabswithisolatedContext: such a tab would run without the scenario's sign-in and trace, and replay as a plain tab in the shared context.
Fixes:
- An Outline row never passes on another row's data. Recording and
the check that decides whether a shared recording fits a row now use
one rule for where a row value sits, so whatever the check would flag
is written as
<column>: a value glued to other letters or digits (order<id>,2 <item>s booked,<name>님), in any normalization, and a value holding markup (<script>...</script>,Alice <alice@x.com>). A value two columns share on the recorded row stays as typed (which column it was cannot be told): a row typing them apart is recorded again once, and where no row does (shipandbillboth France, both Spain), the recording is bound to its row. A value glued to other text is written as<column>only in steps that use that column. Where one recording cannot carry a row's value, it is bound to its row: a field that still holds other rows' values beside the recorded row's (Red, Green, Blue,Plan changed: Basic to Pro) stays as recorded, and so does the row's value glued inside other text in a step that does not name its column (alice2024@example.com). The recorded row, and rows with the same value, replay; every other row fails red at zero tokens, without healing, recording again or calling the agent, with a message naming the column, the step and the fix: put<column>in the step, give the text its own Examples column, or give that check its own Scenario. A spelling recording cannot write binds the recording to its row the same way, wherever the check that decides whether a recording fits a row would find it: another case (Welcome, Aliceforalice), a slug (/weather/new-york), URL-encoded (?q=C%2B%2B), words joined (data-isopenforIs Open). The other rows fail with guidance instead of recording each other again forever. So does a value of one or two characters that an assertion not naming its column checks (?page=1,Qty: 1), which would otherwise pass on 10 or 12. A date typed in ISO for a cell in another format makes the other rows record again rather than pass, and so does a row filling columns the recorded row left blank, and one whose shared value came from an environment variable that has changed since. A recording that reads a value through{data:...}or{env:...}stays current when the data changes. The recording format version no longer depends on the environment or clock. The check also holds for a row added after recording, for{data:...}and{env:...}cells, which count by the value they read, and for an environment variable changed since recording, named in the cell or in its{data:...}value (the recording keeps only a one-way hash of the value, never the value). Recording and the check read a{data:...}value that is itself{env:...}the way replay types it, data first, so such a row neither records over the others forever nor passes on their values. A value the step writes itself as the whole field, such as a login with{env:...}or through its own{data:...}reference, never binds the row; a row value glued to such a value ({env:ADMIN}@example.comtyped as the row's mail) still does. A login, a base URL ({env:BASE_URL}/cart) or text the step spells in full counts as the step's own, so a row value that happens to sit inside it never binds the row. A{data:...}reference keyed by a column ({data:users.<user>}) is recorded as written, so each row reads its own entry instead of the rows recording over each other; when two such references choose the same entry on the recorded row, a step that reads it without writing either one binds the recording to its row, and editing that entry in the data file no longer makes the recording stale. An expected response status or method, or an attribute's name, follows its column in a step that names it (the response status is <status>), where it was recorded as the first row's and a row expecting another status could pass. A pending proposal bound to its row refuses the other rows the same way, so they never record over it, and the message namessaffron run --rerecordwhen the row it was recorded on is gone. - Assertions read the first element a target matches. A first match that
is hidden (including
visibility:hidden, a closed<details>orcontent-visibility:hidden) gives way only to a shown one with the same text, so "You have unsaved changes" can no longer pass a check for a hidden "Saved". Expected text matches with whitespace collapsed first, then as written; a blank expected text passes on a target that shows none; a list check also matches the list as the page shows it (text-transform). A column inside a frame selector is escaped on replay. - Linked folders inside the test folder are followed, each folder once, so a link that loops no longer stops the run. A folder that cannot be read, directly or through a link, stops it with the error, as in 0.9.9, instead of dropping its scenarios and calling their recordings orphaned.
- An Outline row whose Examples cell is a
{data:...},{unique:...}or{date+N}token, in any date format ({date+2:DD.MM.YYYY}), replays its own value, not the first row's. - A blank expected text or pattern (a
{data:...}value or an Examples cell that is now empty) passes only on an element that shows no text, never on one showing any text. - A response's expected status or method, or an attribute's name, follows
its
{data:...}or{env:...}value: a data file changing 201 to 422 makes the recording read 422, and an older recording still holding 201 there is recorded again. A step reused for another quoted value ("201"to"422") takes the new value there too, or is not reused. - An attribute check whose pattern takes only an empty value fails when
the attribute is missing:
disabledchecked as^$no longer passes once the button is enabled, and neither does one taking any value (.*) or anything but false (^(?!false$)for "is pressed"). A pattern that takes an empty value andfalsebut nottrue, such as^(false)?$for "not invalid", still holds on a missing attribute, as in 0.9.9. A recording that relied on another pattern holding on a missing attribute now fails and needs recording again. expectDifferswithout a target keeps the capture it compares as recorded, like the capture that saves it, so other Outline rows no longer fail with "no captured value named".- A field that a blank cell or
{data:...}value leaves empty no longer makes a check hold for anything. An attribute's pattern left blank passes only on an empty or missing attribute. An expected URL, a response's URL, status, method or body, or the name or text of the element a step finds, left blank for a row fails with a message naming the field. A field that keeps other text around the blank value (Total: <amount>, a pattern\b<v>\b) checks only that text. - A list check (
Then the plan lists every {data:plans.<tier>}) reads the row's own list, and one recorded before, still reading the first row's, is recorded again. - A column a step names follows its row inside an attribute's name too
(
data-<tier>), in any case (disabledfor a row'sDisabled), so two rows no longer record each other again forever. A response's status or method, or an attribute's name, takes a{data:...}reference only when it is that whole value and the step names the reference: a 200 is not an order's quantity of 200. A captured value's name (captureTextandexpectDiffers) no longer counts as a row's value, which also had two rows recording each other again forever. - A
{data:...}reference that two references chose alike on the recorded row binds the recording to its row when a response status, method or attribute holds it too, as it does elsewhere. - A step reused for another quoted
{data:...}entry, or for an edited doc string, is not reused while the old entry's value or the old doc string is still written in other text (a test idrow-alice-smith) or the old reference is. An edited doc string still written that way makes the recording stale too. - An Outline recording made by 0.9.x, which does not name its row, is
recorded again once on a row it holds another row's value for. Under
--no-agent, as in CI, that row stays red, with a message saying the recording comes from an older saffron-ai, until it is recorded once. - A response status, method or attribute that a step not naming the
column checks as the row's value (
Then the API answers as expected, recorded as 201 on the row whose<code>is 201) binds the recording to its row for every column holding that value, so the other rows fail with guidance instead of checking 201. So does one holding what a{data:...}reference keyed by columns reads on that row ({data:users.<region>.<user>}reading 201), for the columns it reads. Both are judged on the recorded row alone, so a row added after the recording was accepted is refused too. So does an attribute's name or a list check's data path holding the row's value, in any case or as a slug: an attribute's name as a whole part (aria-expandedfor a row'sExpanded,data-is-openforIs Open, neveraria-invalidforAli), a data path inside a key too ({data:plans.basicPlan}forbasic). Where the step names the column, the value follows the row instead when it is a whole part of the name or path: an attribute's name in any case (data-<tier>), a data path spelled as the cell is ({data:plans.<tier>}). Inside a word it is never written as<column>(aria-invalidstays as it is on a row whose value isvalid). A value that only coincides (an amount of 200 beside a check of status 200, apostbeside method POST, adisabledstate beside thedisabledattribute) binds too: give the check its own Scenario, or give the expected value its own column. - The text or regex inside a fallback or frame selector that is blank for
a row (
button:has-text("<v>"), an XPathcontains(., '<v>'),text=<v>, anywhere in a>>chain) fails with a message naming the selector, instead of matching any element. An exact attribute match such asinput[value="<v>"], with or without theiorsflag, still checks for an empty value. - A failed
Thenstays red when a pending proposal from--rerecordchecks something else there; it no longer turns yellow. The same holds when the run fails before reaching theThen: a pending proposal covers it only if it checks everyThenthe assertion policy protects as the committed recording does. A heal never files an edit to the text of aThenstep. - An
{env:...}value with a'or regex characters in it (an email) is masked in a failed step's message, reports, proposals and traces, where Playwright writes it escaped. - Data-table and doc-string parameterization preserves captured-value names in comparisons. Editing a table value no longer makes replay look for a different capture that was never saved.
- Network body assertions wait for a successfully read, complete response. A pending or failed body read no longer counts as an empty body, and responses longer than 262,144 characters are checked in full instead of letting a truncated prefix pass the assertion. A body check against a response that is not JSON, text or XML (an image, a binary file) fails instead of matching an empty body.
- Recording format version 2. A recording with several people
(
As "buyer"steps), or one that uses a double, right or modified click, a multi-select or a cleared selection (all now recorded and replayed as made), is written as version 2, so an older runner refuses it instead of replaying everyone in one browser or a plain click. So is an Outline recording bound to its row or keeping a recorded row's value as typed, which only a runner that checks the Examples row replays safely, and one whose response status, method or attribute name follows a column or holds a{data:...},{env:...},{date...}or{unique:...}token, which a 0.9.x runner would read literally. Every other recording stays version 1. - Recordings 0.9.9 saved for names with no ASCII letters (
unnamed.json) are still found, a damaged older recording is reported as invalid instead of recorded again, andsaffron prunekeeps an older evidence folder its live scenario still owns until the scenario has run under the new name.saffron runsworks with aghtoo old to report when a re-run started. saffron steps,statusand the MCPsearch_stepscount a step recorded in any person's session as recorded, and divergence is counted per session;search_stepsnever shows a session'sAs "..."text. The MCP tools readsaffron.config.jsonon every call, so a changed test folder is seen without restarting the server.- A dropdown option whose text holds
](Large [L]) is recorded whole. A{date+N}Examples cell is matched only as a whole value, only in the steps that use it, in any format including two-character ones ({date+1:DD}) and inside regex and selector fields. expectEachreads the same shown text as the other text checks: a visible value is found even when a hidden option has the same text, and a hidden line break no longer splits a word in two. Only a line the page draws ends an entry (a line break in the HTML source does not), and text inside a closed<details>,hidden="until-found"orcontent-visibility: hiddenis not shown. A check aimed at a dropdown option, a<textarea>'s text, or a visible part of a hidden container passes again, and text checks are about ten times faster on deep pages. A text or visibility check reads the first copy of its target the page shows, skipping a hidden duplicate (a hidden mobile menu before the desktop one), and never another copy further down. A two-line doc string or a value with doubled spaces matches the page's text as it is shown, and text below the fold in acontent-visibility: autobox is found in WebKit too.- A
{date+N}Examples cell passed through a step's table or doc string replays each row's own date. When two Examples columns (or two keys of a table) hold the same value in the recorded row, that value stays as typed rather than being guessed into one of them: a later row with different values records again instead of passing on the wrong column. A recording keeps the Examples row it was made on (examplesin the recording), so a row whose value differs from one left as typed is recorded again rather than replayed. A date spelled with its year follows its row in later steps too; dates resolve on the clock the recording started on. Step reuse applies the same check, so a step still holding another row's value is never reused for a row that differs, and a one- or two-character value (1,2) that fills a whole field follows its row too, including as the whole of a regex (^12$). An Examples cell left blank on purpose replays blank, in the fill and in the assertion that checks it, instead of being reported as a lost value. - An Outline assertion keeps its whole recorded text with only the row's
value made a placeholder (
Order for <user>: cancelled, not just<user>), so a row cannot pass on a check cut down to its value. A value the step writes out itself (I log in as "alice") is never bound to an Examples column, a value inside a CSS selector (#row-alice) follows its row, and step reuse never carries one column's placeholder into another scenario, nor reuses a step whose recording still names the old value in another spelling (Red Shoeas/product/red-shoe). Recordings made by 0.9.9 are not rewritten; a newly recorded Outline whose rows differ in such text records each row's check as it is. - A CSV line of only commas or spaces is no row (write
"",""for a deliberate empty row). An empty"features"meansfeatures/everywhere. - A re-run GitHub run's results label describes what its latest attempt
left, shard by shard. In a sharded workflow, name each summary
saffron-summary-<shard>(the docs' snippet now says so). - Text checks pass only on text the page shows: hidden text, a hidden part
of a visible element or a hidden dropdown option no longer satisfies
"should be shown", and text in a
display: contentswrapper does. A check against a{data:...}list (expectEach) leaves out hidden options and hidden option groups too. - A filled value that looks like code (
text .click('x')) is recorded as the fill it is; two confirm dialogs in one step each keep their own handler. - Step reuse refuses a quoted-value match when the old value survives in
a test id, selector or frame (
I delete "ann"never clicksdelete-bob), and when the recording's locators name a placeholder the scenario lacks. saffron report --mergechecks every shard history line field by field, and the dashboard escapes every value it shows.- A proposal's feature edit that would leave a step empty is refused.
- Healing a mid-scenario assertion (
adaptable-mid) keeps what the step already did, such as a remembered value. - Secrets stay masked in a failed proof's tab and response URLs (report, bundle) and in both terminal lines a pending proposal prints when it no longer replays.
saffron authorwith Claude runs with no built-in tools.- A recording without
recordedAtno longer stops step reuse. - A date the scenario writes out (
I enter 2026-10-10, a table, a data value, an{env:...}variable) is entered as written; only a date the agent picked moves with the calendar. - Typed text keeps its line breaks and tabs; double, right and modified clicks replay as recorded; a multi-select keeps every option.
- A browser-matrix bundle's evidence (
.../chromium/failure.png) is accepted on view and import. features: "."finds only real files, never.saffron/,node_modulesor.git, andadmin/login.featureandshop/login.featureget a recording folder each instead of sharing one (so does a file outside thefeaturesfolder). Recordings saved in the shared folder are still found by the feature they name; files directly in the folder keep their paths.- A browser session named by an Examples column (
As "<who>") never seeds another row's step: the recording made for the buyer's row is not reused for the seller's. - Scenario names with letters outside ASCII (
Login Успешный,Café ouvert) get distinct recording files and screenshot folders; recordings saved under the earlier name are still found. - Completing a step in the middle of a line, or before its
As "...", never duplicates or drops text. saffron status --jsonsays when the orphan scan could not finish (orphanScan: { complete, errors }); both editors show it instead of "no orphaned recordings".
Agent skill and Claude Code plugin
- The skill now teaches a loop for writing a new test: survey the project, search the existing steps, place the scenario (StepSet or inline), fix the language server's diagnostics, run only that scenario, review and accept. It says to write steps as a user sees the page, never as selectors or test ids, and the syntax reference ends in one complete example project.
- The plugin's listing describes what Saffron does and its README states the license terms in plain words.
Runner
- A browser matrix in one run.
saffron run --browser chromium,firefox,webkit(or a list inbrowser) runs the suite in each browser, one after another, into one report: a row per scenario per browser, in the terminal, the report, the history and the dashboard. Chromium-family browsers go first, so Firefox and WebKit replay what they recorded. Every browser starts before anything runs. Screenshots and traces are kept per browser, andsaffron tracenames themfeature:scenario@browser. The dashboard stands a scenario at its worst browser and never calls a difference between browsers flaky. Refused with--shard. Every result now names its browser, so running one browser again replaces that browser's rows and keeps the others'.saffron status --json(and the MCPproject_statustool) names the browser of each row that needs attention. - Device and viewport profiles.
--device "iPhone 15"(configdevice) runs every scenario under a Playwright device profile, recordings and heals included;--viewport 1280x720(configviewport) sets the size alone. An unknown device names the closest ones, and a mobile profile in Firefox stops the run up front. - Parallel recording and healing.
--agent-workers N(configagentWorkers, default 1) lets up to N agent sessions run at once, each in a browser of its own. An outline's rows stay together on one worker. Each session is paid for: raise it deliberately. - Shards know their browser. A shard records the browser it ran in;
saffron report --mergerefuses shards of different browsers, a re-run attempt replaces only its own browser's shard, and each browser's merge keeps its own history line. - A bad storage state stops the run first. A
storageStatefile that is missing, unreadable or not JSON now stops the run at startup, exit 2, before--rerecorddeletes any recording. - No second replay without an agent. With
--workersand--no-agent, or in Firefox and WebKit, a scenario that went red in the parallel phase is no longer run again afterwards: nothing could change its result. - Per-worker timing. The report has a timing panel (each worker's runs, busy time and share, and the five slowest scenarios), and the terminal prints each worker's share and the three slowest scenarios. Merged shards list each machine's workers.
- Saffron's own data in the replay. Each step shows the tabs it
opened or switched to and the response each network check matched,
and the Saffron pane shows what the agent cost for the scenario. The
report's steps carry
tabsandresponsesfor tools. - Browsers in the editors' contracts.
saffron status --jsonandsaffron report --run <id> --jsongive each row itsbrowser, and the run'sbrowserswhen its rows span several.
VS Code extension 0.6.2
- After a browser-matrix run, Last Run and a CI run's results label each
row with its browser, and Open Replay and Open Trace open that
browser's execution alone (
feature:scenario@browser).
JetBrains plugin 0.6.2
- The same in the Last Run and CI Runs tabs: rows say their browser, and the replay opens that browser's execution alone.
- A browser matrix in one run.
Behaviour changes
- Tags now apply as in Cucumber: a Feature's and a Rule's tags to
every scenario in it, an Examples block's to its rows. A CI filter can
select differently than before:
not @ignorewith@ignoreon a Feature now skips that feature, and--filter @smokewith a Feature-level@smokenow takes all of its scenarios. Check your--filterexpressions when you upgrade. The dashboard's results per tag and the IDEs' Tags tabs count the inherited tags too.
Runner
- Run one scenario by its line or its name.
saffron run features/login.saffron:12runs the scenario that line is in, Cucumber-style: its Scenario line, a tag above it or any line inside it. An Examples row runs that row alone, any other line of an outline runs every row, and:12:30names two.features/login.saffron:Successful loginnames it instead, which stays true while the file is edited, and:Roles (example 2)one Examples row, assaffron tracelists it. A line no scenario owns (the feature's header, a Background, a StepSet, a Rule's header, past the end), a name no scenario has, or an outline whose Examples have no rows yet stops the run with exit 2 and says what is there. Works with--filter,--shardand--rerecord. - Tags on a Feature, a Rule or an Examples block apply to what they
hold, as in Cucumber:
--filter @smoketakes a feature tagged@smoke, andnot @wipleaves out a Rule's scenarios, or an Examples block's rows, tagged@wip. Reports, history and the IDEs' tag lists carry them; a scenario lists every row's tags. - A scenario whose own name ends in "(example N)" keeps its results. Report and history rows now carry the scenario's name; an older row is read as a scenario of that exact name first, and as an outline's row only when no such scenario exists, in status, the dashboard and CI results alike.
- A run that writes an outline's recording runs every row of it
(
--rerecord, or an outline not recorded yet or whose recording is stale), also rows an Examples line or a--filteron Examples tags left out: the rows share one recording. A row asked for records first and the others replay the new recording in the same run, so one that fits only that row comes back red, not as a proposal. - The dashboard keeps each Examples row's latest result from whichever run ran it: a run that left a row out (a tag filter, a named run) no longer clears that row's failure; a row removed from the table stops counting.
- A run of named scenarios is not a suite run. Its results replace those scenarios' own in the last report, beside the rest of the last run, and the report takes the run's own times and hook results. Its history line is marked partial: the dashboard counts it for those scenarios (their latest result, never run, recent failures), while trends, the pass-rate series, the last run's figures and the next run's comparison count runs of the suite only. A last report that was imported from CI, for another base URL or billed another way (subscription or API key) is replaced, and the run says so. Sharded, such a run merges the same way.
- Two scenarios of one name in one file stop the run, as any two
scenarios sharing a recording do: one per Rule was allowed, and each
overwrote the other's recording. The check now reads the configured
featuresfolder, not onlyfeatures/, and always the files the run names. - One pass rate everywhere. The run report, the trends and
saffron status --json(a newpassRateper run) leave out a heal whose proof replay failed, as the dashboard already did: the same run no longer reads 100% in the report and 0% on the dashboard. - History written on Windows counts on the dashboard. A history line
naming
features\login.saffronis the same feature asfeatures/login.saffron, so its scenarios keep their latest result and recent failures instead of reading as never run. - A refused run deletes no recording.
--rerecordremoves recordings only when the scenarios are about to run: a run stopped first (a cache collision, an{env:...}or{data:...}reference, an invalid--trace,--browseror--assertion-policy, a failed setup hook, a missing browser) keeps them. - A path that does not exist stops the run.
saffron run features/chekout.saffron(a typo) exits 2 naming it, instead of running the other paths alone. - Run lenses from the language server, for an editor that asks for
them (
initializationOptions.runLenses).saffron lspputs Run, Replay only and Headed above each scenario, in the file's own Gherkin language, each carrying the scenario'sfile:nameand the project to run it in. A.featurefile outside a Saffron project gets none, and neither does a file outside the editor's workspace, so a lens never runs anywhere the user did not open (such a file keeps its own project's completion and diagnostics). A scenario named like lines (42,10:30) is targeted by its line. The JetBrains plugin also refuses a lens whose folder is outside the IDE project. saffron status --jsongives each scenario's line from the parsed file: right in another Gherkin language, and never aScenario:line inside a doc string.- Open Trace on a CI run works from the editors again.
saffron trace --run <id>takes--max-download <MB>, as the JetBrains plugin passes it, and refuses a larger download exactly asreport --rundoes: exit 2, and with--jsonthe samekind: "too-large"andsizeBytes. - Two editors can load one CI run at once. Each request downloads
into its own temporary folder beside
.saffron/ci/<provider>-<id>/, so one never deletes another's download. When another request finished first, Saffron uses its folder, checked as a cached one is, and removes its own. A finished download removes only earlier attempts of the run, never a later attempt or another request's download in progress; what a killed download left (0.9.7's.partial, or a folder untouched for a day) goes. --max-downloadcovers everything downloaded. Saffron reads every page of a GitHub run's artifacts, not just the first 100, and downloads exactly thesaffron-bundle*artifacts it sized, one by one by name, on GitHub and Azure. An artifact whose name it does not accept is neither counted nor downloaded.- Doc strings in
.saffronfiles stay as written. A line such asStepSet: sampleorStepSet sampleinside a"""or```doc string is literal test data now, no longer read as a step set definition or call.
VS Code extension 0.6.1
- Health's pass rate of recent runs is the runner's (saffron-ai 0.9.8 or later): a heal whose proof failed does not pass.
- Open Report and Open Trace on a CI run keep
saffron.ciMaxDownloadMBwhen the run's results are no longer on disk (trace with saffron-ai 0.9.8 or later). - Run, Replay only and Headed above each scenario. A click runs that
scenario alone, by name (
saffron run <file>:<name>), as a task in the project that owns the file. The file is saved first, and the panel's replay-only and headed toggles apply as they do to every run. Lenses and the side panel find scenarios in the file's own Gherkin language (# language:), nameless ones too. Needs saffron-ai 0.9.8 or later; with an older one in the project a click says so. - A CI run's yellow results read "pending review", not "healed". Yellow also covers first recordings and proposals not yet verified.
- Each yellow or red CI scenario says whether the run verified it: "unverified" or "verified", and nothing when the run does not say.
JetBrains plugin 0.6.1
- Health's pass rate of recent runs is the runner's (saffron-ai 0.9.8 or later): a heal whose proof failed does not pass.
- Run, Replay only and Headed above each scenario, with LSP4IJ installed. A click saves the files and runs that scenario alone, by name, through a temporary run configuration made for it, so the run widget runs it again. A saved configuration of the same name is left as it is, and nothing from the Saffron template carries in, and a scenario of the same name in another file gets its own. The lenses work while the IDE indexes, and say when the project's saffron-ai is too old.
- Right-click Run inside a scenario runs that scenario alone, with or without LSP4IJ, never reusing a lens's Replay only or Headed configuration; outside one, or in a file in another Gherkin language, it runs the whole file, as before. Both need saffron-ai 0.9.8 or later.
- Open Trace works with every runner and keeps the download limit.
With saffron-ai 0.9.8 it passes
--max-downloadtosaffron trace, which downloads a run no longer on disk; with 0.9.7, which rejects the option, it leaves it out. - Open Trace finds a Scenario Outline's example row ("Login (example 1)") by the name the run shows.
- CI yellow results read "pending review", not "healed", and each scenario says "unverified" or "verified" when the run reports it.
- Tags now apply as in Cucumber: a Feature's and a Rule's tags to
every scenario in it, an Examples block's to its rows. A CI filter can
select differently than before:
Runner
- See what a CI run found before you import it.
saffron report --run <id>downloads the run's Saffron bundles once into.saffron/ci/<provider>-<id>/attempt-<n>/, checks them assaffron importdoes (repository, digests, sizes, format, the run, attempts and shards), judges each proposal against this checkout without filing it, and opens the run's report page. Your.saffron/reports/,.saffron/proposals/,.saffron/artifacts/and import ledger are left as they were.--jsongives the run, its totals, each scenario with its failing step, first error line, screenshot and trace, each proposal as importable or not with the reason, and warnings: what the VS Code and JetBrains CI runs views show under a run. - A run without results says why.
running(not finished),expired(the service deleted the artifact) ornone(nothing uploaded; the guide's workflow exports only on failure, andif: always()on the export and upload steps keeps every run's results), asreasonKindandreason, exit 0. - Large runs ask first.
--max-download <MB>refuses a run whose bundles add up to more, with{"error", "kind": "too-large", "sizeBytes"}and exit 2, so an editor can ask and run again. - Replay a CI run, view only.
saffron trace --run <id> [scenario]replays the run's traces from.saffron/ci/, taking the same scenario forms assaffron trace(<feature>:<scenario>included). The page has no Accept or Reject and its server no review routes: nothing was filed. - View, then import, downloads once.
saffron import --run <id>reuses the viewed download of the run's latest attempt. - A cache that looks after itself.
.saffron/ci/keeps the newest 10 runs and at most 1 GB, removing the oldest first; a download goes through a.partialfolder and is renamed whole, so a cancelled one leaves nothing behind. The folder writes its own.gitignore(*), so it never shows ingit status, even in a project set up before it existed, andsaffron initadds.saffron/ci/to the entries it manages.
VS Code extension 0.6.0
- Expand a CI run to see what it found, without importing it. The run lists its totals, each failed and healed scenario with its failing step and first error line (the tooltip has what the agent changed), and "N proposals: M importable" with the reason each one would not import. Your reports, proposals and evidence are left alone. Needs saffron-ai 0.9.7 or later; an older runner says so.
- Open Report and Open Trace on a viewed run. Open Report shows the run's report in your browser; Open Trace replays a traced scenario, view only. Import stays the only action that files proposals.
- Large results ask first. Results above the new
saffron.ciMaxDownloadMBsetting (default 100, 0 for no limit) ask "This run's results are N MB. Download?". Loading can be cancelled, which stops the whole download, and never runs while the CI runs view is hidden. - A run with no results says why: still running, its artifacts expired, or it uploaded none.
JetBrains plugin 0.6.0
- CI Runs shows what a run found without importing it. Select a run to see its totals, each failed and healed scenario with its failing step and first error line (the narrative in the tooltip), and "N proposals: M importable" with the reason for each one that is not. It loads in a background task you can cancel, only while the tab is on screen. Needs saffron-ai 0.9.7 or later; an older runner says so.
- Open Report and Open Trace on a viewed run. Open Trace (or a double-click) replays a failed or healed scenario's trace, view only. Import Run stays the only action that brings proposals into the checkout.
- Large results ask first, above the limit in Settings, Tools, Saffron (default 100 MB, 0 for no limit), the same as the VS Code setting.
- A run with no results says why: still running, expired, or none uploaded.
- See what a CI run found before you import it.
Runner
- Three dashboard figures no longer flatter the suite. Recorded
counted a stale recording, one whose scenario was edited since and
needs the agent again, so the dashboard could say 100% while
saffron statussaid otherwise; it now counts current recordings only, and at 100% says "every scenario has a current recording". Passing and the last run's pass rate counted a heal whose zero-AI proof had failed; such a heal is now "healed, unverified", out of the pass rate, with its own stripe in the health bar, a count on the tiles and a mark in the healed-again list. Flaky counted a scenario that was red on one base URL and green on another as flipping; flips are now counted between consecutive runs on the same base URL, and a first recording that failed is left out.status --jsongainsquality.suite.unverified,quality.lastRun.unverified, andunverifiedon tags and healing entries. Both IDE dashboards show the runner's page, so they follow without a plugin update. - An upload whose file does not exist says so. Under a project path reached through a link (a temp folder on macOS), the message was "outside the project". The repository's own example now names its upload file relative to the project, as replay has resolved it since 0.9.4.
- Uploads are single files. A folder passed to a file input was let
through, and a folder upload sends everything inside it, dotfiles such
as
.envincluded. The agent's upload and replay both refuse a folder now. saffron report --mergenever hides another environment's result. A re-run attempt replaced the earlier attempt of the same shard number before checking that both ran the same suite against the same base URL, so a green production attempt could silently drop a red staging one. An attempt now replaces only the same shard of the same suite, split and base URL; anything else is refused, naming the base URLs.saffron export --branchworks with"features": ".". With the scenarios at the project root, the shared StepSets were left out of the check against the commit the branch starts on, so a correctly bound proposal was refused as using a StepSet "not defined anywhere".- A replay that finds its recording stale no longer leaves a listener on the browser context.
VS Code extension 0.5.1
- The dashboard tab shows the Saffron mark in dark themes. It used the activity bar icon, one dark shape that VS Code tints there and a tab draws as is, so it was dark on a dark tab.
- Cancelling an import stops every process it started, on macOS and
Linux too. The CLI runs in a process group of its own, so a
ghorazdownload that outlives it is stopped as well, with a forced stop for one that ignores the first request. Output over 16 MB ends the command with a clear message.
- Three dashboard figures no longer flatter the suite. Recorded
counted a stale recording, one whose scenario was edited since and
needs the agent again, so the dashboard could say 100% while
Runner
- A recording can only use the environment variables its scenario
names. Replay fails any other
{env:...}by name, so a recording someone commits, or a CI bundle you import, cannot read the rest of your environment. The run's preflight checks the same set, masking covers exactly what can resolve, and a{env:...}or{data:...}the agent typed as plain text is recorded escaped and never resolves. An outline whose Examples rows hold different{env:...}values now replays each row with its own variable. - Replay opens only http(s) pages and uploads only project files.
Recorded navigation and new tabs refuse
data:URLs, otherabout:pages and other schemes, at replay and during recording. Upload paths resolve against the project root, wherever saffron was started from. - Proposals that came from elsewhere are not run with your secrets.
A run replays a pending proposal at zero AI only when this machine
filed it. One imported from a CI run, or checked out from a heal
branch, is never replayed, healed over or recorded over: the scenario
reports yellow, pending review, with how to review it and the CI run
it came from.
--strictfails on it like on any yellow;saffron run --rerecordrecords over it. A CI build of a heal branch therefore reports yellow until the proposals are accepted, and does not heal again. After upgrading, a proposal filed by an earlier version is reported as pending review on its first run: accept or reject it. - Heal once. When a committed recording fails and a heal filed on
this machine is still pending, the run replays that heal before
calling the agent. If it passes, the scenario is yellow, covered by
the pending proposal, so the next run and the outline's other Examples
rows do not pay for another heal. This works under
--no-agenttoo. saffron import --runasks the CI service who started the run. A run started by a fork's pull request is refused unless you pass--allow-fork, and a downloaded bundle that names a different run from the one you asked for is refused.saffron accept --alltakes only what you reviewed one by one. It skips proposals imported from a CI run unless you pass--include-imported, and treats a proposal no proof replay checked as unverified (--include-unverifiedto take it). Proposal files you name are accepted as before. The Accept All buttons in both IDEs run this command, so they are covered too.- Re-runs and shards. Import takes the latest attempt of a whole run
exported more than once, reports a conflict when shards of one run
filed different proposals for the same scenario, and refuses two
bundles holding the same shard.
--shard-by scenariokeeps all rows of a Scenario Outline on one shard. Shard reports record the CI attempt, andsaffron report --mergemerges the latest attempt of a re-run shard. - Feature edits are shown before they are applied.
saffron diff(text and--json) and the draft pull request list the feature file editssaffron accept --with-feature-editwould apply, and mark an edit to a step set's step, which changes every scenario using it. Feature edits can no longer hold line breaks or control characters, accept skips one that would make the scenario name a new environment variable, and every reader refuses a recording that names a feature file outside the project. - Text from a proposal, report or trace is printed as text. The HTML
report and the
saffron tracereplay page escape every value, the replay page runs under a Content-Security-Policy with sandboxed snapshot frames, andsaffron diff,saffron acceptand the pending listing strip terminal control sequences.saffron importrefuses a bundle whose zip holds a file twice, or whose report has a step status or counts no run writes. - Reports name proposal files relative to the project, so a report read on another checkout points at the right file. Older reports still work.
- Messages use plain punctuation, the missing
{env:VAR}error included. - The dashboard's Failing count no longer forgets an old failure. It counted only failures from the last 10 runs, so a scenario that failed and was then left out of ten runs read "all clear" while the health bar still showed it red. It now counts each scenario's latest result, as the health bar does.
JetBrains plugin 0.5.1
- The Dashboard never shows another project's page. A page still loading when you switched projects (or when the tab was hidden) could land afterwards and stay on screen; it is now dropped. Clicking a scenario opens it in the project the page was made for.
- A recording can only use the environment variables its scenario
names. Replay fails any other
Runner
- The agent skill knows the new features. It shows tag expressions
in
--filter,saffron trace, and a CI job that exports its heals as a bundle (or a draft pull request) forsaffron runsandsaffron import --run. The AGENTS.md / CLAUDE.md blocksaffron initwrites says how a CI heal arrives, and theproject_statusMCP tool says which run an imported proposal came from. Runnpx saffron initagain to update a project's copy. saffron dashboard: the suite's quality at a glance. A page in the run report's design, read from.saffron/history.jsonl: tests, the share passing (each scenario's latest result), failing scenarios and for how many runs in a row, flaky ones (passed and failed by turns over the last 20 runs), ones the agent healed again and again, never run and left out of the last run, the slowest, pass rate and execution time per run, and results per tag. It writes.saffron/reports/dashboard.htmland opens it;--stdoutprints the page for the IDE panels. The same numbers are insaffron status --jsonasquality.
VS Code extension 0.5.0
- The dashboard is the suite's quality page, the one
saffron dashboardrenders, instead of the run report. Click a scenario's name to open it. Needs saffron-ai 0.9.4 or later. - Last run: Open Replay and Open Report in the title bar. Open Replay opens every scenario the run traced, so it works when everything passed and no row is listed; Open Report opens the HTML report in the browser.
JetBrains plugin 0.5.0
- The Dashboard is the suite's quality page, the same one as in VS Code, instead of the embedded run report. Click a scenario's name to open it. Needs saffron-ai 0.9.4 or later.
- Last Run: Open Report opens the run's HTML report in the browser.
- Last Run: Open Replay works when everything passed. With no row to select it did nothing; it now opens every scenario the run traced, or says how to keep traces when the run kept none.
- The agent skill knows the new features. It shows tag expressions
in
Runner
- A heal made in CI can be reviewed and accepted on your machine.
saffron exportpacks the pending proposals, the run's report and the screenshots and traces it names into one zip, with where the run happened (repository, branch, commit, CI run, shard); CI uploads it as an artifact.saffron importchecks it against your checkout and files it where a local run would have, sosaffron diff,saffron accept,saffron trace, the report and both IDE panels work on it unchanged. A bundle from another repository (by its path; the same path under another host name, such as an SSH host alias, imports with a warning), one that does not say its repository (unless--allow-unknown-repository), of a newer format, with a damaged file or a report it cannot show, or with a file over 1 GB (or over 100 times its size unzipped) is refused whole, before anything is written, and so are bundles that add up to over 8 GB unzipped together; a proposal made against a different scenario text or committed recording is refused by name, and the rest import. Bundles are read from disk one file at a time, a URL's downloaded to disk first, so every shard of a large run imports at once. Text from a bundle, in errors and reasons too, is printed without control characters. Export keeps to the same limits: it refuses a file over 1 GB and, when a bundle would be over 4 GB, leaves the traces of green scenarios out first and says so. Shard bundles merge into one report. The imported report says where it came from and links to the CI run. Examples for GitHub Actions and Azure Pipelines are in the guide (§19). - Or CI opens a draft pull request with the heals.
saffron export --branch <name>commits the proposal files, not the recordings, onto the commit the run tested (on a pull request build, the head of its branch, where each proposal's binding is checked again and one that does not fit is left out, judged by that commit's own StepSets and data) as a new branch, pushes it and, on GitHub, opens a draft pull request throughghwhose description is each proposal's verification and diff, with the commands to accept, and a staging line that stages exactly what accept changed (git add -A .saffron/cache && git add -u .saffron/proposals). After a tag build, which tested no branch, name the pull request's branch with--base. Only the checkout's HEAD, a commit it merged in or one it descends from is ever built on; apull_request_targetbuild records the base branch and the commit it checked out, never the fork's. The reviewer switches to the branch, runssaffron accept, commits and marks it ready: no agent session on their machine. The CI checkout is left as it was, and an existing branch is never pushed onto. saffron status --jsonsays which CI run a proposal was imported from (importedFrom: repository, branch, commit, run and its link, shard), for as long as it is the proposal that was imported, so the editors can show it past the next local run.saffron runsandsaffron import --run <id>, for GitHub Actions and Azure Pipelines alike.runslists the branch's CI runs (throughgh, orazwith its azure-devops extension), with how many of each run's proposals are pending here;import --rundownloads a run's bundles and imports them. The repository's host picks the service; acikey in the config sets it, and the Azure DevOps organization and project, for a GitHub repository built by Azure Pipelines. An Azure pull request build is listed under the branch it validates, and only the repository's own builds are listed. Bundles of a re-run take each shard from its latest attempt.import --rundownloads into a folder under.saffron/reportsand removes it when done. A download that takes over 10 minutes is stopped and says so. A tag build is listed under its tag on both services. An Azure DevOps SSH remote through a host alias counts as Azure. With--json, everyimportfailure, an invalid config included, is{"error": ...}and exit 2. Aruns --jsonwhoseghorazis missing, signed out or failing answers{provider, branch, runs: [], problem: {kind, message}, error},errorholding the same message, so a client reading either field works; any other failure is{"error": ...}alone; both exit 2. On Azure Pipelines each listing asks for 50 builds first, and for more, up to 500, only while too few are the branch's.export --branchopens the draft pull request on Azure DevOps too, throughaz repos pr create.- Export streams the bundle to disk. Screenshots and traces are copied into the zip in chunks, never all held in memory, and the zip is renamed into place only when complete; a file that changes while it is packed is refused.
- Failures are one line.
exportexits 2 for every failure that pushed nothing (an--outthat is a folder, an invalid config with--branch), keeping exit 1 for a branch pushed without its pull request. Any command's expected failure printssaffron: <message>and exits 2, never a stack;SAFFRON_DEBUG=1shows it. - Safe branch names in the pull request. The description quotes the
branch in its commands, and
--branchor--basewith a space, a quote or any of$ ` ; | & < > ( ) \is refused up front. - A
pull_request_reviewbuild records its pull request's branch, read from the event, not12/merge. import --jsonpasses on only the manifest fields it knows, cleaned of control characters.- The pull request's review steps run saffron-ai by name
(
npx -y -p saffron-ai saffron diff): in a checkout withoutnode_modules, a barenpx saffronwould fetch the unrelatedsaffronpackage. - A bundle names its run as the CI service does (
github:<run id>,azure:<build id>), even withSAFFRON_RUN_IDset, sosaffron runsand both editors count its proposals under the right run. - A green scenario's trace over 1 GB is left out of the bundle like the other green traces, instead of failing the export.
- Import hashes, compares and copies traces in chunks: a 1 GB trace is never held in memory. A bundle that changes on disk while it is imported no longer says "nothing was imported" after some files were written.
saffron runson Azure asks for 50 pull request builds first, and for more only while too few are the branch's, not 500 on every list.
VS Code extension 0.4.0
- CI runs: a new view lists the current branch's CI runs, GitHub
Actions or Azure Pipelines, through
saffron runs. Import runssaffron import --run, then lists under the run what happened to each proposal and any warning (a checkout at another commit, a scenario that changed since). Imported proposals are marked with the run they came from, linked in the tooltip. From a red CI run to an accepted repair without leaving the editor; commit and push with the Source Control view. The CI's own tool keeps the credentials. - Text from CI cannot act in a tooltip. Run titles, branch names and a bundle's notes are escaped so they never form links, headings or other markup; a run links out only to GitHub or Azure DevOps, and never when the address carries a user name. Import results are kept per project. An import can be cancelled, and on Windows cancelling stops the whole process tree. The list follows a branch switch, and makes no CI call while its view is hidden.
- Cancelling an import stops all of it, and says what it left. On Windows the whole process tree is stopped however the CLI was started; on macOS and Linux a process that outlives SIGTERM gets SIGKILL. The message says a cancelled import may have written part of the run.
JetBrains plugin 0.4.0
- CI Runs tab: the same runs and the same Import, through the same two commands, with each proposal's outcome under its run. The Proposals tab names the CI run an imported proposal came from.
- Text from CI is shown as plain text, in CI Runs and in Last Run after an import, never as markup. Import is a background task you can cancel, with no time limit; a run is imported once even if you switch projects meanwhile; Refresh never starts a second list. The tab loads as soon as the project is trusted, and its processes stop when the project closes.
- Choosing another project clears the tabs at once: a run listed for the last project can no longer be imported into the new one, and a status that arrives late for the last project is dropped.
- CI Runs lists again when you check out another branch. A cancelled import says it may have written part of the run.
- A heal made in CI can be reviewed and accepted on your machine.
Runner
Browser tool calls are checked call by call. A security review of the agent's tools found three ways a prompt-injected page could reach the machine:
view-source:file://slipped past Playwright'sfile:check and read any file; thefilenameoption of the snapshot, screenshot, console and network tools wrote page content over any project file (a response body into a bin is code execution on the next run); and a file upload could hand.env, with every secret, to the page. Navigation now opens web pages only, uploads take existing files inside the project by their real path (a link out of the project is out) and never dotfiles,filenameis dropped, and anything the vendor's tools write goes to a private folder of the session, removed when it ends. The checks sit in the one proxy every provider and both paths use, and replay applies the same ones to a cache, which is data anyone can commit. A project reached through a symlinked path (/varand/private/varon macOS) counts as the project. Claude sessions also setstrictMcpConfig, so a.mcp.jsonor user-level MCP server of yours never joins a recording. Codex sessions turn off its remaining execution features (unified_exec,multi_agent,code_mode_host, the in-app and external browsers) besides the shell.Go-to-definition works under a folder with a space,
#or%in its name, and on Windows. The language server sent unencoded file paths as URIs; a strict client rejected them. A nested project without afeatureskey now gets the default folder instead of the outer project's, and on Windows a StepSet in a subfolder is no longer flagged as defined twice.--filter "not @wip @slow"is refused instead of reading asnot @wip or @slow: onceand,notor parentheses appear, every tag needs itsandoror. The plain list forms are unchanged.Outline values no longer replace inside a test id or frame selector by coincidence (
addinadd-to-cart), while a short id with a digit still does (1234inorder-1234), and a<name>that opens a regex named group is left alone at replay.The cost trend leaves a gap for runs whose provider reported no cost instead of plotting them as $0, and every other run keeps its place. Cursor's turn limit counts tool calls, not the pieces their arguments stream in. An Antigravity session that hits the turn limit says so, and how to raise it, even when agy reports the session a success.
No agent can run code on your machine through Playwright. Playwright MCP's
browser_run_code_unsaferuns any JavaScript in the Playwright server process (its own docs call it RCE-equivalent); a Codex test session used it to read a file from the home directory. It is now hidden from every provider's agent, Claude's included, and a call to it by name is refused. Nothing recorded depends on it.browser_evaluate, which runs in the page, stays.Claude recordings no longer carry your claude.ai connectors. On a Claude subscription login, Claude Code loads the account's claude.ai connectors (Gmail, Figma, Canva, ElevenLabs and the like) into every session from its second request on. On one account that added 294 tool definitions, 1.16 MB of schema, to every model call: recording the guestbook example cost $15.74 at API rates and 11.5M cache-read tokens. Saffron's sessions now set
disableClaudeAiConnectors, and the same recording costs $0.37 with 279k cache-read tokens, in 70 s instead of 180 s. The agent never needed those tools; your connectors in Claude itself are untouched.Record and heal on the subscription you already have. Besides Claude (still the default),
"provider"insaffron.config.json(or--provider) now takescodex(a ChatGPT plan, through@openai/codex-sdk),antigravity(Google AI Pro/Ultra, through Google'sagyCLI; experimental) andcursor(through@cursor/sdk). Each runs on the login its official client already has on the machine. These agents reach the browser through a local MCP gateway that records every call with the same hooks as Claude, so their recordings, heals and proof replays work the same.modeltakes the provider's own model names;saffron authortakes--providertoo. The two SDKs are optional peer dependencies: install the one you use. Codex runs without your own~/.codex/config.tomlapart from its login settings (a keyring login works), which needs macOS or Linux for now: on Windows the Codex provider refuses to start rather than run with your own MCP servers.agygets an explicit--print-timeoutsized to the turn budget, and anagythat cannot start fails its scenario instead of the run. Antigravity is verified live with agy 1.2.13 (record, proof replay and heal): its MCP calls go through agy'scall_mcp_tool, allowed by two rules in agy's settings and limited by a hook to Saffron's two servers.saffron login [provider]says what a provider still needs on this machine, and does the one-time setup where one is needed:saffron login cursorruns Cursor's sign-in, andsaffron login antigravitychecks that agy is signed in and allows Saffron's tools in agy's settings (two rules, everything else kept).New docs page: AI providers and sign-in (
docs/PROVIDERS.md), with every provider's sign-in, whatsaffron loginsets up, what Saffron does in its sessions, how to undo it, and troubleshooting.A pass can no longer leave an assertion unchecked.
saffron_donewith success is refused once while an announcedThenstep has recorded no check, and Saffron's tool refusals are now MCP errors, so an agent that batches its calls stops and fixes the step instead of filing an unverified proposal. A second pass stands (the proof replay still marks it unverified), so no session burns its turns on it. A step announced twice keeps one adaptation note, and an agent that fails aftersaffron_donekeeps its verdict.Node.js 20.6 or newer is now required (22.13 for the Cursor provider).
Cost not reported, not $0. Runs whose provider reports no price say so in the summary line and the report instead of showing $0.0000.
The editor finds a project nested inside a bigger folder. When the IDE is opened above the Saffron project (a solution folder with the tests in a subfolder),
saffron lspnow resolves each.saffronfile against the nearestsaffron.config.jsonabove it. StepSets defined in a sibling file and{data:...}references were reported as undefined in that layout; they resolve now, and completion, hover and go-to-definition follow the same root.A browser tool result can no longer flood the agent's context. A recording fetched a 1.5 MB API response body with
browser_network_request(about 400k tokens). Claude Code then compacted the session in a loop and failed the scenario after a few dollars of tokens. Every browser tool result is now cut at 60,000 characters, with a note asking for less (a snapshot of one region, a filtered request list, or asaffron_asserton the field). The Playwright code Saffron records comes first in a result, so a cut result still records its action. The cap applies with and without the trace bridge, and to Codex, Antigravity and Cursor sessions through the tool gateway.Each Outline row acts on its own element through a locator chain. A chain keeps the row's value inside its selector; that value now becomes the row's
<param>(or data-table reference) there too, so another row no longer clicks the recorded row's card. The same goes for every other field replay resolves: an assertion's regex pattern (anhref, a heading's name), a test id, a prompt answer, a frame and a drag's drop target. An Outline checking a link'shreffailed as written on every row but the recorded one.A changed table or data value still written in the recording is found escaped too (in a selector's quoted or regex body, or a pattern), so the recording is made again instead of replaying the old value.
Chains with a comma, an apostrophe or a role option record the element they clicked.
filter({ hasText: 'Paris, France' })andgetByText(/^Don't$/)recorded nothing, a regex with an apostrophe made a selector Playwright read differently, andgetByRole('heading', { level: 2 })recorded any heading. Role options are kept as selector attributes; one with no selector form records nothing.Tag lists read the way 0.9.0 read them. An empty entry (
@a,@b,) is skipped again, and a tag's own parentheses (@issue(42)) stay part of it; a parenthesis after a space or a keyword still groups.Replay fixes. A heal that fixes a network assertion is checked against the response the agent's session produced, instead of failing "as written" while waiting for a new one. A replay no longer inherits an upload armed on its page by an earlier replay (a pending proposal that failed), which failed its first click. A value resolved into a regex pattern (
{env:VAR},<param>,{data:...}) matches as written: the+ina+b@x.comor the$in$9.99no longer change the pattern.A failed tool call records nothing in Codex, Antigravity and Cursor sessions, as in Claude's: an upload refused because no file chooser was open was recorded, and broke replay.
Antigravity gets its prompt on stdin. The prompt carries resolved
{env:VAR}values, andagy -pput it in the command line, readable by every account on the machine.Tools a recording cannot replay are hidden from every agent: going back or forward, reloading, resizing and dropping files.
browser_navigatereaches any page instead, and is recorded.saffron_doneis refused once when aThenstep recorded only setup (a click that opens a panel) and no check, the rule replay applies.Playwright's MCP server gets the whole environment again when the trace bridge is off, so
PLAYWRIGHT_MCP_*settings apply there too.A proposal lock held by a live process on this machine is broken after an hour: a pid reused after its holder died kept it forever.
saffron statussays "cost not reported" for a provider with no price, instead of $0.00.saffron status --jsonleaves such a cost out (the last run's, history's and proposals'), and the editors' last-run lines say "cost not reported" too (VS Code and JetBrains, next editor release).A Codex or Cursor session stopped at the turn limit says so, and to raise
maxTurns, instead of "ended without finalizing".saffron login cursoron Node below 22.13 names the Node version it needs, and a vendor SDK missing a part of its own reports that part instead of asking to install the SDK again.A single locator with options records the element it acted on.
getByText('Save', { exact: true }),getByLabel('Email', { exact: true })andgetByText(/^Save$/)recorded nothing, so the click or fill was missing from the recording, andgetByRole('checkbox', { checked: false })recorded any checkbox. They record as a selector now, as in a chain.An Outline row's value inside a regex matches the row replaying. Playwright writes exact text as a regex (
filter({ hasText: /^Garden Queen$/ })); that value becomes the row's<param>as well, and replay writes the next row's value escaped, quotes, slashes and brackets included.A secret inside a selector or an assertion's pattern is masked where it is written escaped (a quote in a quoted name, a
.in a regex), and replay writes it back the same way. A short one next to an escape (\buat\b) counts as standing alone.The recorder reads only the code Playwright ran. A snapshot or a response body that showed Playwright code (a docs page, a code viewer) was recorded as actions.
Only tools a recording can hold reach the agent, whatever
PLAYWRIGHT_MCP_CAPSturns on: an allowlist of the tools that record or only read replaces the list of hidden ones, so coordinate mouse, storage, cookie and route tools stay out.A heal checks a network assertion against the response that came late. The failed replay's response log keeps listening through the heal, so a response that arrived just after replay gave up counts.
A cut browser result never splits an emoji, which made Codex drop the whole result.
saffron login antigravitywrites through a settings link (a dotfiles repo, even before the file exists there, through chains and linked folders) and keeps the file's permissions; a link into a read-only place (a Nix store) is left as it is, with the two rules to add named.The editor keeps each nested project's vocabulary, built once, in editors that watch its files for it (VS Code, JetBrains), and a changed file refreshes every open project's diagnostics. A project whose build fails keeps its last good one, and coming back to a project checks its documents again. Opening a file it already read (go to definition) rebuilds nothing.
A browser action a Codex, Antigravity or Cursor agent gave up on is still recorded. Playwright finishes a click or a goto whatever the agent does meanwhile, so the recording keeps what the browser did.
Editors (VS Code extension 0.3.3, JetBrains plugin 0.3.3)
- A project nested inside the folder the IDE opened. Both editors
now find the nearest
saffron.config.jsonabove a file and treat that folder as the project: runs start there, status, proposals,.envand the step vocabulary come from there, and the project's ownsaffroninstall is used. JetBrains shows a chooser in the tool window title when the IDE root holds several projects, and adds a Working directory to run configurations. Before, a run from a nested project wrote.saffron/into the parent folder with default settings and recorded everything again. - JetBrains: nothing runs in a project you have not trusted. No
status, runs, diffs or language server until the project is trusted;
the tool window says so and loads as soon as you trust it. The
language server and the Run menu entries appear only inside a Saffron
project (a
saffron.config.json, an installedsaffron-ai, or.saffronfiles), so a Cucumber or Behat folder of.featurefiles no longer launchesnpx -y. A nested project is left out of the outer project's file list and never run from the outer root. - VS Code on Windows: runs no longer pass through a
.cmdshim, which cmd.exe re-parses. The installed CLI runs with node directly; the npx fallback escapes every argument. - JetBrains: reused run configurations no longer keep a Re-record tick
from an earlier use.
.featurefiles appear in the Files tab and run from the context menu. A newlogincommand takes the provider. Links in the Dashboard open in your browser. The tool window refreshes only for Saffron runs and changes inside the project. - Both editors list proposal files the runner cannot read, with the reason, and count them as pending. A run with no reported cost shows as not reported, never as $0.
- VS Code: the panel keeps the project you were on when a webview,
diff or image takes focus, and Select Project lists nested projects;
Run Tagged keeps tags written with parentheses such as
@issue(42); stale status after a run is gone; status errors show the error line, not the last stack frame; lines inside docstrings are not read as steps;Example:is a scenario keyword; Re-record works with Replay only ticked;{data:...}flags an empty list and a header-only CSV; Run Tagged skips tags--filtercannot express and says which; diagnostics clear when a file closes or is deleted.
Runner
- Tag expressions in
--filter.--filter "@e2e and not @wip","(@smoke or @sanity) and not @flaky":and,or,notand parentheses, in the shape Cucumber and pytest use, withandbinding tighter thanor. The list forms keep their meaning:@a,@b,@a @b,@a|@band a repeated flag all meanor, and the@may be left off. A malformed expression stops the run with exit 2 and names the problem. - Replay fixes. A network assertion checked without the run's
response log (the check after a heal) now checks the response body too:
a heal could pass while the API answered
{"ok":false}. A file upload armed for the next file chooser is set by the click that opens it, which waits for a chooser that opens late up to the action timeout. If no chooser opens, or the files cannot be set (a missing file), that click fails, and so does each retry of it, instead of the runner ending or the scenario passing with nothing uploaded.{env:VAR}in the pattern ofexpectMatchesandexpectAttributeis resolved like every other field. - A click through a locator chain records the element it clicked.
When the agent targets an element with a selector string, Playwright
MCP reports a chain such as
locator('article:has(...)') .locator('role=button[name="Select room"]'). The recorder kept only the first link, so replay clicked the whole card instead of its button, and a refine pass then re-clicked the room under the next step. Chains of locators, getBy calls,filter({ hasText }),nth,firstandlastare now kept whole as one selector; a chain part with no selector form is not recorded rather than recorded as a different element. The agent is also told to act through snapshot refs, which record as role and name. - Dates picked through a selector string move with the calendar. A
date in a fallback selector or test id (
role=button[name="2026-10-11"]) is now templated to{date+N}like one in a name or label, and replay resolves it there, so the recording does not expire the next day. - Open Replay opens the run its row lists.
saffron trace --run <startedAt>refuses a later run ("The last run changed since this was listed"), and both editors pass it, so an older row no longer opens a newer run's trace. - A decision under way keeps its lock. A proposal lock held by a live process on this machine is never broken for its age, so a paused decision cannot be overtaken.
Editors (VS Code extension 0.3.2, JetBrains plugin 0.3.2)
- VS Code: a project nested inside a bigger workspace folder finds its
data folder.
{data:...}completion, hover and diagnostics resolve against the nearestsaffron.config.jsonabove the file, not the workspace root. - JetBrains: the Run Tagged tooltip says the CLI also takes tag expressions.
- Open Replay opens the run its row lists, with saffron-ai 0.9.1 or later; with an older runner it opens the scenario as before.
- JetBrains: the tool window opens on IntelliJ 2026.2. The Dashboard tab crashed the whole Saffron tool window there (NoClassDefFoundError: JBCefApp), because 2026.2 moved the embedded browser into its own plugin. The plugin now depends on it where it exists, and without it the Dashboard falls back to Open in Browser.
- VS Code:
.saffronfiles show the Saffron mark in the explorer and tabs, under any file icon theme that has no icon of its own for them (Seti, the default, included).
- Tag expressions in
Runner
- Execution replay:
saffron trace. Not a video. With"trace": "retain-on-failure"(or"on") in the config, or--trace <mode>for one run, a scenario keeps an execution trace, andsaffron traceopens it as a replay in your browser: your Given, When and Then on the left, each step opening onto the actions Saffron replayed and the Playwright calls behind them; the page at the selected action in the middle, rendered as a live DOM you can inspect, before, during or after the action, with a filmstrip of the whole run; on the right the failure, the retries, the call log, the network during the selection, and what the agent changed. The failed step is selected when you arrive. Traces live next to the screenshots under.saffron/artifacts/, are kept for the latest run and pruned with the rest. The report names the command for every traced scenario, asfeature:scenario. The page opens on127.0.0.1orlocalhost, a forwarded port included, and keeps what it loaded when a later run replaces the files. A tab left open from beforesaffron tracewas started again says to reload the page instead of loading the new run's files. The trace is Playwright's own format, rendered by the snapshot engine of the Playwright Saffron runs with; the replay around it is Saffron's. The Saucedemo example config ships withretain-on-failure. - Secrets in a trace.
{env:VAR}values, including one a data file holds, are masked back to their tokens before the trace is kept: in the event log, the DOM snapshots and stylesheets, and every request and response body, URL, header and WebSocket message, in the spellings the value takes there (JSON escapes, percent-encoding as browsers and servers write it, HTML entities, CSS escapes). The unmasked trace never reaches.saffron/artifacts/, even when a run is killed while saving it. The scheme, host and port of a URL are kept, so the replay can still load the page's stylesheets and images; a secret anywhere else in it, credentials included, is written%7Benv:NAME%7D. A value shorter than three characters, one inside base64 or another binary encoding, a value in a URL's host, and the filmstrip's pixels are not masked, so look through a trace before attaching it to a public issue. - Review a heal from the replay. The selected step shows what the
proposal changes for it, the elements Playwright actually matched are
outlined in the page, and the Saffron panel has Accept recording and
Reject proposal. Both act only on the exact proposal associated with
this run: if it was replaced, accepted or rejected since, the replay
refuses and says why. One that went stale against the scenario or the
committed recording, or can no longer be read, cannot be accepted but
can still be rejected. An unverified recording, or one this run showed
failing, needs an explicit acknowledgement first. Suggested feature
rewrites are listed with the command that applies them (
saffron accept --with-feature-edit); they and propagation to other recordings stay withsaffron accept. - Accept or reject the proposal you listed.
saffron status --jsongives each pending proposal arevision, andsaffron accept <file>#<revision>orsaffron reject <file>#<revision>decides that proposal only if it is still the one listed. One a later run replaced is refused ("changed since it was listed. Refresh and review it again."), exit 1, and left in place. A decision and a run filing a proposal never interleave, so what is checked is what is promoted, with or without a revision. Accept and reject act only on files under.saffron/proposals:saffron reject features/login.saffronused to delete the feature file. - Proof replays and the agent in the trace. A recorded or healed
scenario also keeps a trace of each proof replay (
proof-1.zip,proof-2.zip) with its own step outcomes, and Trace at the top of the page switches between them and the run itself. The agent's own browser calls are recorded too, under the step it was working on and in words ("click Login button", "read the page"); what it looked at before naming a step is filed under the step it named next. A healed step is marked ✦, and a healed run opens on it. If that cannot start with the installed Playwright, Saffron says why, once, and the agent works through the standard browser connection without being traced. - Masking fixes outside the trace. A secret whose value appears in
another variable's name or in a
{data:...}key no longer corrupts the recording (ROLE=ADMINbeside{env:ADMIN_PASSWORD}used to be recorded as{env:{env:ROLE}_PASSWORD}, which never replayed). Three-character values (a CVV, a short PIN) are now masked in recordings, reports and history too, as in the trace; in a recording a value shorter than six characters is masked where it stands alone or where the step joins it to other text ("qa{env:ID}"), soENV=devnever turns/devicesinto/{env:ENV}ices. Reports mask a value's URL-encoded and HTML-escaped forms. A recording masks a secret in every field replay reads (labels, placeholders, test ids, fallback selectors, frame names, drag targets, URL patterns, the text for a prompt dialog), not only typed values and URLs, and replay resolves the token in each of them: a selector recorded astd.{env:NAME}still finds its element. A{env:VAR}written as a value in a data file is masked in recordings and reports like one written in the step. - The HTML report embeds only the run's own screenshots. A merged
report (
saffron report --merge) could be made to embed any file the merging machine could read, such as.env; it now embeds pictures from.saffron/artifacts/only, and escapes the setup and teardown lines. - Evidence from an interrupted run is not mixed in. Screenshots and
traces are kept at fixed paths that the next run replaces. A run
stopped before it wrote its report used to leave the previous report
pointing at the new files; the IDE panels now leave out such a
screenshot and
saffron tracerefuses such a trace. - Playwright 1.63 and trace format 9. The runner now depends on Playwright 1.63, and the replay reads both trace formats: 8 (up to 1.62) and 9 (1.63 and later), which files snapshots under the call by phase and screencast frames under their own folder. A project that resolved 1.63 saw broken filmstrip tiles and "No snapshot for this selection".
- The first snapshot shown no longer arrives unstyled. The snapshot engine forgets a trace while its first frame is still loading, so that frame's stylesheets and fonts came back missing and the page looked bare until the next step was selected. The replay now warms the engine with a hidden frame first.
- A step with no snapshot of its own shows the last page there was. A step the agent only inspected (the accessibility tree, the title, the console) leaves no DOM snapshot, since Playwright takes them around interactions. Instead of an empty pane, the replay shows the page after the last interaction before that step and says so above it. A step that never ran, after a failure, shows no page: "This step was never reached." An action that took no snapshot (reading the title, say) shows the page as the last snapshot before it left it, not as its step began. With several tabs open it only borrows from the same tab; Playwright 1.63 does not record which tab such an action ran in, so when that is unknown the replay says so, and the filmstrip marks no frame, rather than show another tab. A tab opened later in the run changes nothing before it, and one closed changes nothing after.
- Every traced scenario in one replay.
saffron tracewith no name (or a name several scenarios share) opens them all, with a Scenario picker in the header, starting on a red one, else a yellow one. A run with"trace": "on"where everything passed opens too.--jsonwith no name prints the scenario the replay opens on. Other scenarios' traces are read when first picked and checked against their digest; one that cannot be read says why. - A step lists what happened in the order it happened. Saffron's page check, the recorded actions, the agent's and the calls outside any step are interleaved by time, and the arrow keys follow that order. A call outside every step is filed under the step running when it started. The keys keep working after a click on a button or tab, leave Cmd, Ctrl and Alt shortcuts to the browser, and the selected row scrolls into view.
- ✦ marks only a step the agent healed. In a first recording, a step the agent worked through reads "recorded by the agent". In a proof replay, or a run that replayed a pending proposal without the agent, a step the proposal changes reads "ran the proposed recording". Neither is called healed, and Details names the kind of run: Heal review, First recording, New recording or Proposed recording.
- A pending proposal that failed is shown apart. When a run replays a pending proposal, it fails, and the run goes on to the committed recording or records again, the proposal's attempt is one section, "Earlier attempt: the pending proposal", instead of both executions being merged into the steps.
- A heal whose assertion does not hold as written says so on that step. The failed step shows the guard's "Assertion failed as written" reason instead of what replay missed first, and the execution trace shows the steps the agent carried after it.
- A failed agent session no longer marks every step failed. When a heal did not succeed, the report and the replay showed all steps as failed, including the ones replay had passed before the agent took over. They now show what happened: passed up to the step replay stopped on, that step failed with its reason, the rest skipped. When the agent carried that step and failed on a later one, the later step is the failed one, with the agent's reason. A failed recording shows the steps the agent got through the same way.
saffron traceandsaffron reportkeep working on a machine with no browser opener (a container, an SSH or WSL remote): they print the address to open instead of crashing.
Editors (VS Code extension 0.3.0, JetBrains plugin 0.3.0)
- Open Replay on a scenario in the Last run panel runs
saffron tracefor it, by feature and scenario name, so two features with a scenario of the same name are told apart. JetBrains also getstraceas a run configuration command: its Scenario field takes a name orfeature:scenarioas it is or wrapped whole in double quotes, and blank opens every traced scenario, starting on a failed one. - A Last run row stays with its run. In VS Code, a row (its
screenshot, Open Replay, Open Scenario) acts on the project it came
from, even after the panel has moved to another one, and says so if
that project is no longer open.
In both editors, a screenshot is checked against the digest its run
recorded when you open it, so a picture a later run put at the same
path is refused instead of shown as this run's, and what opens is a copy
of the picture that was checked, named after the scenario. VS Code keeps
the copies in the extension's own storage, one folder per window,
removed when the window closes; JetBrains in the IDE's system folder,
removed after a day. Neither uses the shared temporary folder. In
JetBrains, double-clicking such a row no longer jumps to the scenario
over the message saying why. In VS Code, Open Scenario on a row whose
feature file was renamed or deleted since the run says so.
saffron status --jsonnow reports each picture'ssha256. - Accept and reject decide the proposal you saw. Both Proposals
panels send the revision they listed, so a proposal a later run
replaced after the panel showed it is refused with a prompt to refresh,
not decided unseen. A replaced proposal comes back unticked in both.
With a runner that lists no revision, they send the file as before.
VS Code's Accept all tooltip now says what
--alldoes: it accepts what is pending when it runs, skipping UNVERIFIED proposals and refusing STALE ones.
- Execution replay:
Runner
- A proposal is bound to what you reviewed. Every proposal now carries
digests of the scenario's steps, of the committed recording it was
diffed against and of its own recording, taken when it was filed.
saffron acceptrecomputes them and refuses a proposal, by reason, when any moved since: a step edited, a colleague's heal accepted, a locator fix propagated, a pull, a hand edit ("step 3 now reads ... and was ..."). There is no override; run the scenario again for a proposal bound to what is there now, or reject it.saffron accept,saffron diff,saffron status --json(stale) and both IDE panels show STALE with the reason first. A run that finds a stale pending proposal records afresh instead of proving it again. Fingerprints are advisory and are not bound; data tables, doc strings and the feature edits that--with-feature-editwould apply are. Proposals filed before this release carry no digests and are accepted with a note. - Recordings and proposals are validated on read. The schema version,
every step, every action type and field are checked, along with the
operands each action needs (a click without a usable target, an
expectUrlwithout a URL, which used to compare "" and pass), and the first problem is named by path. An unreadable recording makes its scenario red with the reason instead of replaying as garbage or being recorded over;saffron statusmarks itinvalidwith the reason (cacheProblem); an unreadable proposal is reported and left out of listings (unreadableProposals) instead of blanking them. A file from a newer runner, a schema version or an action type this one does not know, says to updatesaffron-ai. Fields added within the schema version stay optional, so newer caches read on older runners. Hand-written caches may omitrecordedAtandrecordedBy.
Editors (VS Code extension 0.2.10, JetBrains plugin 0.2.5)
- Both Proposals panels mark a stale proposal STALE, with the reason and what to do, and note a proposal filed by an older runner.
- A proposal is bound to what you reviewed. Every proposal now carries
digests of the scenario's steps, of the committed recording it was
diffed against and of its own recording, taken when it was filed.
Runner
- A check could point at the wrong element. In a live heal the agent
recorded "the Morning Ritual package is applied" against
role=definitionwith no name. The booking summary has one definition per row, so replay read the first one (the room) and the proof replay failed.saffron_assertandsaffron_capturenow refuse a bare role unless it is one that occurs once per page (status,alert,dialog,mainand similar), and ask the agent for a name, text, label or selector instead. - A baseURL's query string and path were lost on replay. With
baseURLset tohttp://localhost:5173/?flow=checkout-guest, a recorded visit to the start page replayed ashttp://localhost:5173/, silently dropping the flag that picks the variant under test. A baseURL with a path fared no better: underhttp://host/app, a cached/roomsreplayed ashttp://host/rooms, and a recordedhttp://host/applewas cached as/le. Underhttp://h/#/home, a recorded visit to/with no hash replayed back to#/home, a different page for a hash router; it is now kept as/#, the way a missing query is kept as?. The baseURL is now a prefix in both directions: cached navigations keep its path and inherit its query, so a cache recorded against?flow=standardreplays against--base-url "...?flow=checkout-guest". A cached URL with a query of its own keeps it, other origins stay untouched, and existing caches replay as before. - Heal summaries could describe a changed flow as a locator tweak.
When an A/B variant folded the guest information page into checkout,
the heal was right (fields re-targeted, the "Continue to payment" click
dropped) but its note only said the selects were now found by label.
The heal prompt now asks the agent to name the kind of change (a locator
or label change, a page removed or merged, a page added or reordered)
and the page it happened on. Saffron also compares the healed recording
with the committed one and adds its own
flow change:notes for any click, key press, goto or tab move that was dropped or added, together with the page replay stopped on. A renamed button still reads as a locator change and adds no note. - Two data references holding the same value could bind a field to the
wrong one. With
users.ownerandusers.recipientboth "Alice", the recorder tokenized every typed "Alice" as the first reference, so a recipient field read the owner. Change the recipient to "Bob" and the scenario correctly went stale, but step reuse seeded the mis-bound actions straight back and the proof replay agreed: verified, zero AI calls, wrong field. The recorder now resolves an equal value by the reference the step itself names, leaves the literal when the step names both or neither (so the stale check catches it), and step reuse refuses a step that names a reference its recording does not read. - The report could list
{unique:...}values that were never typed. A recording seeded from earlier steps, and the proof replay of a pending proposal, each drew their own seed while the report showed the values of a third. One seed per scenario run now reaches every replay, recording and proof, so the report's values are the ones in your application. - A second Ctrl-C during teardown exited before the cleanup finished. The first Ctrl-C started teardown; the next one fell through to Node's default and killed the run on top of it. Saffron now answers it with a note and keeps waiting.
- Re-merging shards after a shard was re-run left the old result in
history. The merge treated the run as already recorded and skipped it.
It now replaces that run's history line, and still never counts a run
twice. In CI the run is identified by the CI run id, the suite and the
base URL rather than by the earliest start time, so this holds whichever
shard was re-run; outside CI,
SAFFRON_RUN_IDgives the same stability. - The proof replay of a recording reused the recording's
{unique:...}values. Fixed alongside the report fix above: the proof is a second execution, so creating the same record again failed, filed the proposal unverified and paid for a refinement pass. Each proof now draws its own values, and the report lists them separately from the recording's. - A heal or a refinement could redo earlier steps on the wrong tab and
file them under step 1. When the agent takes over, it attaches to the
browser over CDP and sees every Saffron page as one tab list, starting
on the first. During a refinement the scenario's own page was still open
beside the proof replay's, so the agent started on the wrong page,
selected a tab, and in one run went back to the start page and redid the
search. Every action performed before its first
saffron_stepwas filed under step 1, and the merge adopted it over the recording replay had already passed: proposals whose step 1 held only "switch to tab 1", or a second copy of the search, and a proof replay that failed on them. Saffron now closes the pages of other browser contexts before an agent session, so the agent sees only the live tabs, and tells it which tab replay left off on: the page replay was really acting on, not the newest tab with a matching URL, which is a different page when two tabs share one. A heal or refinement records nothing until the agent announces the failed step: anything it clicks or navigates before that is diagnosis and is discarded, and the prompt tells it to look with snapshots instead. A seeded recording segment files early actions under the first step it may record, never a step that already ran, and drops an early tab selection. Actions the agent files under an already completed step are discarded, and the merge keeps completed steps' recordings whatever the session holds for them. Each of these leaves arecording:note in the proposal's adaptations. Fresh recordings from step 1 behave as before. The live tab also follows the agent through a seeded recording: a tab it selects, opens or closes in one segment is where the next seeded island replays and the next segment starts, read from the browser tool's own tab list rather than guessed from a URL. - A role with a text no longer matched any element showing that text. Role and text together now mean the element of that role that shows the text, and nothing else.
- Shards from different environments merged silently. A staging shard and a production shard combined into one report labelled with whichever came first. The merge now refuses shards with different base URLs, by name.
Editors (VS Code extension 0.2.9)
{data:...}diagnostics stayed stale after editing a data file. The extension watched feature files and caches but not the data folder orsaffron.config.json, so fixing a missing key inusers.jsonleft the error on the feature until the feature itself was touched. Edits to data files and to the config now refresh every open feature file, including a custom data folder and environment overlays.
- A check could point at the wrong element. In a live heal the agent
recorded "the Morning Ritual package is applied" against
Runner
- A
{unique:...}or{data:...}reference inside a data-table cell is now recorded as the cell reference (<table:1:email>), not as the token itself. The agent types the resolved value, so the recorder matched it against the cell as written and missed. Replay was right either way; the difference is that editing the cell in the feature file now follows the table at zero tokens instead of making the recording stale. - Docs: the
modeloption says what the default really is (whatever Claude Code currently defaults to, which drifts) and which values it takes, so a project can pin its recording model.
- A
Runner
- Split a suite across CI machines:
--shard 2/4andsaffron report --merge.--workersstops at one machine's cores; sharding gives the suite to several. Each shard runs a fixed part, split by scenario rather than by file, so one large feature file spreads over every machine, and every scenario runs exactly once.--shard-by file(or"shardBy": "file") keeps each feature file on one machine instead, for files whose scenarios share expensive or order-dependent state. A shard writes.saffron/reports/shard-2-of-4.jsoninstead oflatest.jsonand leaveshistory.jsonlalone, because part of a run would read as a run in the trends.saffron report --mergefinds the shard reports under.saffron/(or the folders you name) and writeslatest.json,latest.htmland one history line, with the same totals, scenario order and trends as an unsharded run. It refuses a missing shard, the same shard twice, shards that ran a different suite, and shards from different CI runs; merging twice does not count the run twice. Setup and teardown run on each shard and receiveSAFFRON_SHARD. The docs have a GitHub Actions example.
- Split a suite across CI machines:
Runner
- Google Chrome and Microsoft Edge:
--browser chromeand--browser msedge. Run on the branded browser your users actually have, not only the open-source Chromium build. Both speak CDP, so everything works there: replay, recording and healing. Recordings are the same files, so a scenario recorded onchromiumreplays onchromeandmsedgeand the other way round. Saffron uses the copy installed on the machine; when it is missing the run stops before the first scenario, exit code 2, and names the browser (npx playwright install chromeadds it on a CI image). Also accepted asbrowserin the config. - Setup and teardown:
"setup"and"teardown"in the config. Shell commands, one or a list, that Saffron runs once around a run, in the project root: seed the data your scenarios read before any browser opens, and clean up after the last scenario. A failed setup stops the run with exit code 2 and no scenario runs. Teardown always runs: after failures, after a failed setup, after Ctrl-C (which first stops a running setup and waits for it, so nothing writes behind the cleanup). A failed teardown leaves the verdicts alone and turns an otherwise green exit into 2, because what it left behind is the next run's false red. The commands receiveSAFFRON_BASE_URLandSAFFRON_ENV.--no-hooksskips them,hookTimeoutMs(default five minutes) bounds each one, and the report lists every command with how it ended. With{unique:name}from 0.8.0 this makes a suite re-runnable and parallel-safe.
Fixes
- Step reuse could bring back a stale data value. When a
{data:...}value changed, the scenario's recording correctly went stale, but a new recording could be seeded from that same recording's steps, putting the old literal back under a fresh data snapshot, verified, at zero AI calls. A step whose source recording still spells out a value that has since changed is no longer reused. every {data:list}counted hidden content as shown. A value sitting in ahiddenordisplay: noneelement satisfied the check. The target must now be visible, and only text the page actually shows counts. A native<select>keeps its exception, since its options are never visible while it is closed, but only when the select itself is visible. Text is still compared as written, so CSStext-transformdoes not change the result.{unique:name}inside an Examples cell, a table cell or a doc string reached the browser as the literal token. Unique values were resolved before parameters were substituted. They are now resolved in parameter values too, with the same per-run seed, for values and locators alike.saffron prunecould list the screenshots of a live scenario as orphaned when its feature file is outside the configured directory and the scenario has only ever failed, so no cache or proposal named the file. Each screenshot folder now records which feature and scenario it belongs to, prune reads it before judging, and the listing names what the pictures were of.
Editors (VS Code extension 0.2.8)
- VS Code ignored environment-specific CSV files. With environment
stagingandroles.staging.csv,Examples: {data:roles}was reported as not resolving, or described from the baseroles.csv. Diagnostics, hover and go-to-definition now follow the runner: the environment's CSV replaces the base one, and the hover names the file it read.
- Google Chrome and Microsoft Edge:
Test data moves out of the feature file: a
data/folder,{data:...}references, list checks,Examplesfrom a file, values unique per run, and data per environment.Runner
Test data outside the feature file:
{data:file.key}. Put values in JSON files underdata/and reference them from a step, a table cell or a doc string:When I sign in as {data:users.admin.email}. Same family as{env:VAR}: the recording stores the token and replay reads the file, so changing a value changes the next replay at zero tokens. A recording that still spells out the old value goes stale instead of replaying yesterday's data, and a missing file or key stops the run by name before a browser opens. New config keydataDir(defaultdata). Not for secrets: data files are committed.Check a whole list in one step.
Then the status filter should list every {data:enums.OrderStatus}records ONE assertion (expectEach) that names the list and its container. Replay reads the list from the file, so a value added to the data is checked on the next run at zero tokens, and a missing one fails by name with how many were found. Values match as whole text ("Paid" is not "Unpaid"), extra entries such as "All" are allowed, and a closed dropdown works.Examplesfrom a file, in.saffronfiles:Examples: {data:roles}readsdata/roles.csv(header row = placeholders) or a JSON list of records. One recording, one zero-token replay per row; a row added to the file records nothing. A placeholder without a column, a missing file and a.featurefile using it are each parse errors that say what to do.{unique:name}: a value that is new on every run and the same within one, for scenarios that create things. The agent types the generated value while recording and Saffron turns it back into the token; a run that is not green lists what each token stood for.{unique:name:digits}for numeric fields.Data per environment:
--env staging(orSAFFRON_ENV, orenvin the config) laysdata/users.staging.jsonoverdata/users.json, key by key. The reference in the step stays the same on every target.saffron status --jsongainsdata: the folder, the environment, the files, and references that do not resolve.saffron lspcompletes{data:paths, shows values on hover, jumps to the key and flags unresolved references. The bundled grammar highlights{data:...}and{unique:...}.saffron init --examplesinstallsdata/catalog.jsonand a scenario that uses it, and the agent skill, its syntax reference and the AGENTS.md block teach the new syntax.saffron prunecovers failure screenshots. A deleted scenario also leaves its screenshot folder behind under.saffron/artifacts/. Prune lists those folders with the caches and proposals and removes them with--yes; a live scenario's folder is kept, per Examples row.saffron status --jsonreports them as orphans of kindartifact, and both IDE panels (VS Code 0.2.7, JetBrains 0.2.4) show them.
Editors (VS Code extension 0.2.7, JetBrains plugin 0.2.4)
{data:...}in the editor. VS Code completes data files and keys with their values, shows the current value (or a list's values, or an Examples file's columns) on hover, jumps to the key, and underlines a reference that does not resolve, reading the same files and environment overlay the runner does. JetBrains gets the same throughsaffron lsp. Both highlight{data:...}and{unique:...}like{env:VAR}, including on anExamples:line.- Both Orphans lists show the screenshot folders of scenarios that no longer exist, so "Remove All" never deletes something it did not show.
Runner
Failure screenshots. When a step fails, Saffron saves a screenshot of the tab it failed in and shows it in the HTML report, embedded so the report stays one file. A healed scenario keeps the picture of the moment its cached step stopped working, for whoever reviews the proposal; a scenario still red after the agent ran is photographed again at the end.
latest.jsoncarries the paths underevidence. New config keyscreenshot:viewport(default),full-pageoroff. Files live in.saffron/artifacts/, latest run only, andsaffron initgit-ignores it: add.saffron/artifacts/to your.gitignoreif your project was initialised earlier. A screenshot shows whatever your page shows, including a secret it displays;{env:VAR}masking cannot reach pixels.A feature file that fails to parse says which file, and why. Invalid Gherkin reached the terminal as a raw stack trace with positions and no file name. It is now one line per problem,
file:line:column. A standard.featurefile that usesStepSetgets the actual answer instead of a parser dump: rename it to.saffron, nothing else needs to change.saffron status --jsongainslastRun.attention: every scenario of the last run that was not green, red first, with the failed step, the first line of the error and the screenshots that still exist on disk.
Editors (VS Code extension 0.2.6, JetBrains plugin 0.2.3)
- Last run, with the screenshot one click away. Both IDEs list what the last run left to look at: failed scenarios, then healed ones, with the step that failed. VS Code opens the screenshot in its image preview on click; JetBrains opens it in the image viewer on double-click. Each also jumps to the scenario. With a runner that predates failure screenshots, both say so instead of reporting that everything passed.
Docs
- A page for "
.saffronand.featurefiles": what each is, a side-by-side table, which to write, and what renaming a file means for its recordings. The rest of the docs say "feature file" for both, where they used to say.featurealone.
Editors (JetBrains plugin 0.2.2)
- LSP4IJ is an optional dependency. A first upload of 0.2.2 was rejected by the Marketplace verifier on 2026.3 EAP because LSP4IJ had no release for that build yet, and a mandatory dependency that cannot be resolved blocks the whole plugin. Highlighting, run configurations and the tool window now work without it; completion, go-to-definition, hover and diagnostics switch on when LSP4IJ is installed, and a one-time notification says so when it is missing. The presence check uses no plugin-manager API, so it stays clean as that API turns internal.
Runner
expectUrlchecks the active page, not any open tab. A URL sitting in a leftover tab could satisfy the assertion for the wrong reason. A recording made before tab openers were marked, with a slow popup andretries: 0, can now fail where it used to pass: re-record it, or leave the default retry, which re-follows the newest tab.- A changed table value only replays if the recording really uses it. Matching key sets used to be enough, so a recording that still carried the old value as a literal, or never referenced the cell, stayed "valid" and typed stale data. The old value is looked for only where a value can live (typed, matched or named text, with tokens removed), so an empty cell or a short value like "on" does not force a needless re-record.
- Propagation never rewrites an assertion, on either side: assertion
steps and assertion actions are skipped when a fix is extracted and when
it is applied, the step wording must match, and a locator inside a frame
is a different locator. When one heal sends the same locator to more than
one place, nothing propagates and
saffron acceptsays so. saffron pruneandsaffron statusrespect features outside the configured directory. A scenario run assaffron run e2e/login.featurehad its recordings listed as orphans. Feature files named by existing recordings are now inspected too, and one that fails to parse blocks the prune like any other.- Step reuse no longer brings back a stale table value. Changing
alicetobobcorrectly invalidated the scenario's cache, and step reuse then seeded the same actions straight back, matching on table keys alone: a verified proposal at zero AI calls whose table saidbobwhile its actions still typed and assertedalice. Seeds now pass the same rule as cache validation, and a step with no table of its own is refused when it carries another step's old value as a literal. The old value is looked for in locators built from data as well (a test id likeuser-alice, fallback selectors, a frame, a response URL pattern), not only in typed and named text: a fill can be parameterized while the assertion beside it still targets the old user. - A feature edit cannot land on the wrong step. Edits address steps by
index, so a step inserted while a proposal waited shifted every index
after it, and an edit meant for "I click login" rewrote the new step.
saffron accept --with-feature-editnow compares the scenario with the steps the proposal was recorded against and refuses before anything is consumed: the file, the cache and the proposal stay as they were. - A skipped feature edit no longer reaches the cache. When one edit applied, the accepted cache was given the new text of every edit, including a Background edit that was correctly skipped, so the feature and its cache disagreed at once. Only edits that landed are synced. Two edits that reach the same StepSet line through two invocations are applied once when they agree, and both refused when they do not, naming the two wordings: the line can only say one thing.
saffron statusparses each feature file once. The vocabulary and the orphan scan each parsed the whole project again; both now reuse what status already read. IDE panels call status on every refresh.
Editors (VS Code extension 0.2.5)
- Completion badges and duplicate-step diagnostics follow cache creation and deletion. Only changes to existing JSON files were watched, so accepting a first recording or pruning one left the vocabulary stale until something else triggered a rebuild. Recordings are watched for all three events (and only recordings, not every JSON file), and the watcher is disposed with the extension.
- Vocabulary is per project in a multi-root workspace. Step sets,
wording and recorded badges were built once for the whole workspace, so
project B's
Loginstep set could answer go-to-definition in project A, and a step recorded in A showed as recorded in B. - The inline diff button on a proposal row works. An inline action passes the tree item, not the file path the row click passes; both forms are accepted, and the project guard applies to both.
New:
saffron diff,saffron prune, a configurablepollIntervalMs, and a configuration reference in the docs. The rest of this release is correctness work from several rounds of external review.Upgrading. Three fixes can change what an existing suite does, all in the direction of telling the truth:
- A scenario that was green because replay quietly clicked a near-match
("Delete all orders" for a missing "Delete"), or because a
Thenstep never checked anything, now fails and heals or asks to be re-recorded. - Feature files in subdirectories get a new cache directory name (a short
digest of their path is appended), so their scenarios record again once.
Files directly under the features root are unaffected.
saffron prunelists the old directories afterwards. saffron status --jsongainscacheStateandorphans; nothing was removed.
Runner
Correctness fixes from an external review of the runner. Four of these could let a run look green or yellow when the goal was never checked.
Secrets could reach your feature files. Machine-applicable feature edits were the one agent output that skipped
{env:VAR}masking, sosaffron accept --with-feature-editcould write a resolved password into a scenario. Feature edits, the narrative and the suggested edit are now masked everywhere they are stored or reported.An assertion that recorded nothing now fails. A
Thenstep with an empty action list used to replay as a pass, so a scenario could be green without its verdict ever being checked. Replay fails such a step, and the recording is filed unverified. This can turn an existing green scenario red: that assertion was never being checked. Re-record it.Negative assertions consider every recorded locator.
expectNotVisiblereturned as soon as one candidate was hidden, so a real failure could hide behind a missing primary locator while a fallback was visible.Recorded names match exactly first. A step recorded against "Save" could click "Save draft" when it came first in the DOM. Substring matching remains the fallback for reading the page. The agent is no longer told to shorten names.
Cache files are unique per scenario. The cache directory now includes the feature file's directories, so two same-named files in different folders stop overwriting each other's recordings. Flat layouts are unchanged. A run refuses to start if two scenarios would still share a file.
Suspicious recordings are saved unverified. The check ran after the proposal was written, so the file said verified while the run said otherwise, and
accept --alltook it.An unverified proposal shows the proof replay's timeline, not the agent session's, so the report says which step could not replay instead of marking every step passed.
A step that changes between an action and an assertion is stale. Cache matching compared the step's words but not its keyword, so turning
When I see the totalintoThen I see the totalkept the old recording and replayed the action instead of checking anything.Waiting for a new tab happens where a tab opens. Every click paid a 150ms wait, which taxed scenarios that never use tabs and still lost the race with a slower popup. The proof replay now records which action opens a tab, and only that action waits, for as long as any other action may take. A five-step login scenario replays in about a third of the time.
saffron statusdistinguishes a valid cache from a stale one. It reportedcachedby asking whether a file existed, so a scenario whose text had changed still looked recorded. Each scenario now carriescacheState:valid,stale,invalidormissing, and the printed view marks stale ones.saffron lspno longer reports a valid step set as defined twice in projects whose path contains a space: it compared a file system path with a percent-encoded URL path.saffron diffshows what a proposal changes: the committed recording against the proposed one, action by action, with a rewritten action shown as one before-and-after pair and changed assertions counted separately. The narrative is the agent's account of what it did; this is what a reviewer is approving. It also appears in the HTML report and in both IDE panels.An assertion step must contain an assertion. The earlier guard only rejected an empty action list, so a
Thenthat recorded awaitForand nothing else still passed.A control that is found but cannot be used no longer hands the action to a different control. With a disabled "Save" next to an enabled "Save draft", replay clicked the draft button. An interaction now stops at the candidate that matched, so drift goes to the failure and healing path instead of quietly doing something else.
Negative assertions check every match of every locator. Checking only the first match let a hidden first element vouch for a visible second one. A locator that errors no longer counts as proof of absence.
Secrets are discovered through Examples rows. A
{env:VAR}reaching a table cell or doc string through a placeholder was invisible to masking, so it could survive into recordings, reports and feature edits.Cache identity follows the path, not its spelling.
admin/login.saffronandadmin-login.saffronproduced the same cache directory; nested files now carry a digest of their path. The collision check also looks at the whole project, because two features run separately never met before.A recording that says a tab opens must see one. The marker was advisory: when no tab appeared the timeout was swallowed and the next assertion ran against the old page. Tab openers are also captured while recording now, not only during the proof replay, which could miss them.
An interaction never substitutes a different control. A recorded name that no longer matches exactly could still act on any control containing it, so a missing "Delete" could click "Delete all orders" and the run stayed green. Interactions now require the exact name; assertions and captures keep the substring fallback, because reading the page is not acting on it. A scenario whose control was renamed now fails and heals instead of passing.
The proposal diff compares actions, not their descriptions. Two actions that read the same could differ where it counts: dropping a response body check or repointing a selector from
#saveto#deleteboth reported no change. Actions are compared field by field, and a rewritten action now lists exactly which fields moved.Repeated steps stay separate in the diff. Steps were matched by their wording, so a scenario that adds the same item twice showed no change when the proposal dropped one of them.
A popup that arrives late is pinned on the action that opened it. The opener marker waited for the next recorded action, so a slow window was blamed on whatever the agent did afterwards, and replay then waited for a tab that action never opens. Only an interaction can be credited: a wait or an assertion recorded between the click and the popup no longer takes it, and the marker now follows the agent's actual tool calls instead of inferring them. Any action marked as opening a tab waits for one, not only clicks. A popup nobody can be credited with is dropped rather than passed to a later action: one that arrives while the agent is idle and the last interaction is too old, one that lands during a call that failed or could have opened it itself, and one the agent opened with a new tab. Only known observation tools (a snapshot, a wait, listing tabs) may vouch for the click before them, and failed tool calls are now drained so they cannot hold an attribution open.
saffron pruneremoves recordings nothing owns. Deleting a scenario or a feature file left its cache behind, where it stayed in git and no reviewer could tell it from a live one.saffron prunelists every orphaned cache and proposal with the reason, deletes nothing without--yes, and--checkexits 1 for CI. Ownership follows the path, so a stale or unreadable cache that a live scenario claims is kept, and while any feature file fails to parse prune refuses to judge anything.saffron status --jsoncarries the same list for IDE panels.pollIntervalMsis configurable (default 100ms, minimum 10). It sets how often a polled assertion re-checks the page insideactionTimeoutMs: lower reacts sooner, higher costs less on a busy CI box.
Docs
- A configuration reference, grouped by what you are tuning (timing, project, runtime, the agent, gates) with every option's default, replacing the paragraph that listed the keys in prose. It ends with how timing actually works: Playwright's actionability wait, then polled assertions, then action retries, and why there is no whole-scenario retry.
Editors (JetBrains plugin 0.2.2, VS Code extension 0.2.4)
- JetBrains plugin: Accept Selected with nothing ticked accepted every
proposal. Blank paths mean
--allto the runner, so the deliberate review workflow could be bypassed by clicking the wrong button. It now does nothing and says so; bulk acceptance stays its own action. - Both IDEs list recordings no scenario owns any more: a JetBrains
Orphans tab (grouped by cache and proposal, double-click to open one,
Remove All after a confirmation) and a VS Code Orphaned recordings
section with the same action in its title bar. Both name the reason, and
both say plainly when the installed runner is too old to report them
rather than claiming there are none.
pruneis also a JetBrains run configuration command, where listing is the default. - JetBrains plugin: the Proposals tab could show an old diff beside a new narrative. Diffs were cached by file name, and a new run overwrites a proposal under the same name, so Accept could promote something the reviewer never read. A refresh now invalidates every diff, a late result from before the refresh is discarded, and the cache lives on the UI thread.
- JetBrains plugin: a path with a space stays one path.
features/order checkout.saffronbecame two arguments, even for right-clicked and ticked files. Paths are parsed like a command line, and what the plugin writes itself is quoted to match. - VS Code extension: Run selected ran project B's files from project A in a multi-root workspace, with A's config and install. Selections are now grouped by owning folder, one run per project, from the Explorer too. The status-backed sections name the project they show, and Select project switches it, instead of silently using the first folder. Switching clears those sections at once rather than when the new status arrives: rows left on screen stayed clickable, so a proposal ticked in project A could be accepted in project B when both had the same relative file. Ticks are bound to the project that listed them, and accept, reject, run tagged and prune refuse to act while a status is still loading.
- VS Code extension: the proposal diff could not start on Windows, where
Node refuses to launch a
.cmdwithout a shell. Status and diff now share one launcher: the installed CLI runs under node with a plain argument array, and only the npx fallback goes through cmd.exe, fully escaped. - Both IDEs show the proposal diff: JetBrains under the narrative in the Proposals tab, VS Code by opening the diff as a document when a proposal is clicked or from its inline action.
- VS Code extension: runs the CLI as a task with an argument array and the owning folder's working directory, instead of writing a command line into a shared terminal. A file name with a dollar sign or a backtick is no longer shell syntax, and a file from a second workspace folder runs against that folder's config and install.
- A scenario that was green because replay quietly clicked a near-match
("Delete all orders" for a missing "Delete"), or because a
Runner
- Multi-tab scenarios. Plain prose, no keyword: "switch back to the
first tab", "close the preview tab", "open a new tab at the admin page".
The agent's tab operations during recording are cached as
switchTab,closeTabandnewTaband replay in the same order. New tabs opened by clicks are still followed automatically, and an explicit switch back is no longer undone by that follow. Replay gives a click a moment for its popup to appear and waits for a tab before switching to it. - When a pending proposal fails its zero-AI replay and the agent records again, the run now says which step failed and why.
Editors
- VS Code extension 0.2.3: the Marketplace listing is named "Saffron", its short description fits without being cut off, and the overview introduces the tool and how to get started before listing features.
- Multi-tab scenarios. Plain prose, no keyword: "switch back to the
first tab", "close the preview tab", "open a new tab at the admin page".
The agent's tab operations during recording are cached as
Runner
saffron mcpgains aproject_statustool (thesaffron status --jsonoverview) next tosearch_stepsandlist_step_sets, and reports the real package version.
Agents
- Claude Code plugin:
claude-plugin/packages the language server, the MCP server and the skill as one plugin (/plugin marketplace add s-chathuranga-j/saffron-claude-plugin,/plugin install saffron@saffron).saffron lspnow adopts the workspace root the editor sends ininitialize, so plugin-launched servers index the right project. - The bundled skill covers
saffron status, accepting or rejecting chosen proposals, subscription versus API-key cost reporting, Scenario Outline recording, and the IDE integrations; theAGENTS.mdmanaged block points atsaffron status. Runnpx saffron initagain to refresh both in a project.
Runner
saffron statusno longer lists the same step with different quoted arguments ("<password>","{env:SAUCE_PASSWORD}", a literal) as duplicate wordings; steps are compared by their quote-generalised pattern.
Runner
saffron statusreports duplicate wordings by behaviour: different step texts whose recorded actions are identical once the step's own arguments are masked. The previous wording-similarity list flagged "enter X in the username field" against "enter Y in the password field", which are different steps.
Runner
saffron status [--json]: the project overview that IDE panels and dashboards read: feature files and scenarios with cache state, tags, pending proposals with narratives, last run, history, vocabulary health (divergent and near-duplicate steps) and the effective config.saffron acceptandsaffron rejecttake several proposal files at once.
Editors
- JetBrains plugin 0.2.1 tool window gains Proposals, Tags, Health and
Dashboard tabs; VS Code extension 0.2.2 gains Tags, Proposals and
Health sections and an in-editor dashboard. All read
saffron status --json.
Runner
- Cost is reported by how the run was paid for. On an API key the dollar
figure is what was billed. On a Claude subscription (a Claude Code login
with no
ANTHROPIC_API_KEY) the summary line and the report headline the 5-hour window before and after the run, for example12% → 19%, and label the dollars as the API-equivalent. The weekly window is not shown. - Reports and the summary line name the models that answered during the run.
Editors
saffron lsp(JetBrains and other LSP editors) now offers theStepSet:definition keyword as a snippet when you typestep s,Stepsorstepset, and theStepSetinvocation keyword, even in a project with no sets yet. Before, the keyword only appeared once a set existed, and the definition form never did (the VS Code extension already had both).- JetBrains plugin 0.2.0: a Saffron run configuration type (Run/Debug
Configurations → Add New Configuration → Saffron:
runwith files, folders, tags, replay-only, headed, re-record and extra arguments;report;accept), right-click → Run on.saffronfiles and folders, and a Saffron tool window listing feature files with scenario counts, search, Run Selected / Run All, Open Report, Accept All Proposals, last-run totals and pending proposals. Runs use the project's ownsaffron-aiinstall with the login shell's environment. - VS Code extension 0.2.0: right-click → Saffron: Run (also replay
only, headed, re-record) on
.saffronfiles and folders, a play button in the editor title, and Command Palette entries for run all, open the latest report and accept all proposals. Commands run in a "Saffron" terminal from the workspace folder. - VS Code extension 0.2.1: a Saffron side panel: an activity bar view listing feature files with their scenarios (click to jump), inline Run and Replay-only buttons, checkboxes with Run selected and Run all, replay-only and headed toggles that apply to every run from the extension, the last run's totals with the report one click away, and the proposals pending review with Accept all.
- Cost is reported by how the run was paid for. On an API key the dollar
figure is what was billed. On a Claude subscription (a Claude Code login
with no
Runner
- Scenario Outlines record correctly: an assertion recorded from the page's
longer sentence now becomes the
<placeholder>outright when it contains the row's value (substring replacement used to leave a residue that no other row could satisfy), and rows after the first replay the pending proposal at zero AI instead of recording again and overwriting it. When the proposal does not fit a row, the run is red with a clear message rather than silently replacing another row's recording. - The agent is told to record a Then step's quoted text exactly as expected text, never the page's longer sentence.
- A Given step or StepSet guard that describes a state the page is already in is no longer reported as an adaptation. The saucedemo example's checkout set used to show a warning on its first recording for this.
- Yellow scenarios are labelled "pending review" in the run summary and the HTML report instead of "adapted", since a proposal is not necessarily an adaptation.
Editors
{env:VAR}tokens are navigable: Ctrl/Cmd+click (go to definition) opens the variable in.env, or.env.examplewhen no.envexists; hover says where it is defined and never shows the value. In the VS Code extension and insaffron lspfor JetBrains and other editors.
- Scenario Outlines record correctly: an assertion recorded from the page's
longer sentence now becomes the
Runner
- Example suite:
.saffronfiles only, with StepSets invoked both at the start of scenarios and in the middle ("Recover from a locked-out login", and a checkout set that carries a data table).
Editors
- Grammar: quoted arguments,
<placeholders>and{env:…}/{date…}tokens get their own scopes, so editors colour them like Cucumber parameters instead of the whole step line. Ships in the VS Code extension, the JetBrains plugin and the package's TextMate bundle.
- Example suite:
Runner
- Guided first run:
saffron initoffers to install the Saucedemo example suite (--examples/--no-examples): five short scenarios against the public demo shop,{env:VAR}credentials with a.env.example, a Scenario Outline, a data table, and a StepSet in a library file, plus a matchingsaffron.config.jsonwithretries: 1. Only installs into an emptyfeatures/directory. Documented as "Your first run".
- Guided first run:
Runner
saffron run --rerecord: discard the selected scenarios' caches and let the agent record them fresh, seeded from the other recordings. The sanctioned fix for a wrong recording instead of editing cache JSON.- The recorder understands every way the agent types (
fill,pressSequentially,type) and drops the clearingfill('')that precedes typing into the same field. This was the cause of caches with an empty value for a step that passes one. - A fill recorded with an empty value on a step that passes a value marks the proposal UNVERIFIED with the reason, even if the proof replay passed.
saffron accept --allskips UNVERIFIED proposals;--include-unverifiedoverrides, and accepting a single file still works with a warning.- Retries are visible: a
retried ×Nchip per step in the report and a↻line in the terminal. Retries stay per action with backoff; they are not scenario re-runs. - Scenario Outline rows display as
(example N); report, terminal and docs use plain punctuation throughout. - Reports and history carry the real package version.
Docs and site
- Documentation moved to the site's Docs section, generated from the runner's markdown; the previous docs page became Features; the landing page was trimmed.
- Assertion policy documented on the Features page; cache lifecycle gained the staleness and wrong-recording stages.
Runner
- JetBrains highlighting without the plugin: the package ships the TextMate
grammar at
textmate/saffronfor Settings → Editor → TextMate Bundles. saffron lsprun from a terminal prints the editor setup steps instead of waiting silently for a client.
Editors
- JetBrains plugin 0.1.0 (Marketplace listing 34240):
.saffronfile type, bundled grammar,saffron lspwired through LSP4IJ automatically. - JetBrains plugin 0.1.1: the no-package fallback runs
npx -p saffron-ai saffron lsp(the baresaffronname resolves an unrelated npm package); the install hint shows once per project; dark theme file icon; verified on 2024.2 through 2026.2. - VS Code extension 0.1.3: plain punctuation in README, hover and diagnostic messages.
- JetBrains highlighting without the plugin: the package ships the TextMate
grammar at
Runner
- Agent skill shipped in the package (
skills/saffron/SKILL.mdplus reference sheets) following the Agent Skills specification. saffron init: installs the skill into.claude/skillsand.agents/skills(Copilot and Cursor on request), registers thesaffron mcpserver in.mcp.json,.cursor/mcp.jsonand.vscode/mcp.json, appends a managed block toAGENTS.md(andCLAUDE.mdwhen present), and scaffoldssaffron.config.json,features/and the.gitignoreentry.- Multi-tag
--filter(OR semantics, comma-separated or repeated); scenario tags are written to the report JSON andhistory.jsonlso external tools can key results by a test-case tag. saffron --versionreports the real package version.
Editors
- VS Code extension 0.1.1: case- and spacing-tolerant
StepSetkeyword completion plus a definition snippet. - VS Code extension 0.1.2: warning and quick fix when a set name is written
without the
StepSetkeyword.
- Agent skill shipped in the package (
saffron --versionno longer reports a hardcoded value.
Runner
- Network-aware steps in plain prose:
waitForResponse(healable) blocks until a matching backend response,expectResponse(assertion, never healed) audits that one occurred. Matchers: URL pattern, method, status (exact or class), optional body pattern for polling waits. Waits match from the triggering step onward, so "click and wait" never races. - Recording ergonomics: native dialogs, drag-and-drop and iframes record live. The dialog auto-dismiss was the orchestrator's own launching Playwright client, not the MCP; agent sessions now keep dialogs visible.
- Positive assertions recorded without frame scoping fall back to searching child frames; poll-based assertions sweep all candidates under one deadline.
Editors
- VS Code extension 0.1.0 published to the Marketplace.
- Network-aware steps in plain prose:
Runner
- Secrets:
{env:VAR}tokens resolve from the environment (or a git-ignored.env) at replay; caches, proposals, reports and history are masked back to the token; missing variables fail fast by name. - Action vocabulary for real-world pages:
handleDialog,uploadFile,dragTo, and iframe-scoped targets, replaying on Chromium, Firefox and WebKit. - Cross-browser replay (
--browser firefox|webkit), parallel replay workers (--workers N), and a cheaper model for heal sessions (--heal-model).
- Secrets:
First public release, free for any use under the Saffron Free Use License.
- Gherkin-native runner: an AI agent records each scenario once; later runs are replayed by Saffron in a real browser at zero tokens; runtime healing files reviewable proposals; assertions are never healed.
- Verified proposals (zero-AI proof replay), honest cost reporting, run history and trends, chronic-scenario detection.
- Level-1 step reuse: new recordings seed from every step already recorded.
- Data vocabulary:
<param>placeholders, key/value and record tables, doc strings,{date±N}templates, captured values. - Assertion policy:
strict(default) oradaptable-mid; the final assertion block is always strict. - Step sets and the
.saffrondialect (StepSet:), project-wide library files, parse-time expansion. - Step vocabulary tooling:
saffron steps,saffron mcp,saffron lsp,saffron author; behavior-proven duplicate wording detection.