Writing tests

Test data

Values a scenario needs do not have to be typed into the feature file. Put them in JSON files under data/, and point at them from a step:

data/
  users.json

{
  "admin": { "email": "admin@test.com", "role": "Admin" },
  "roles": ["Admin", "Editor", "Viewer"]
}

When I sign in as {data:users.admin.email}
Then the profile should show the role {data:users.admin.role}
And the first role offered should be {data:users.roles.0}

A reference is the file name without .json, then the path to the value, separated by dots. A number picks an entry from a list.

You write	It means
`{data:users.admin.email}`	one value from `data/users.json`
`every {data:users.roles}` in a `Then` step	check each value of a list
`Examples: {data:roles}`	outline rows from `data/roles.csv` or a JSON list (`.saffron` only)
`{unique:id}`	a value that is new on every run
`--env staging`	lay `data/*.staging.json` over the base files

What does not belong in data/. Secrets. Data files are committed like the rest of your tests, so passwords and keys stay in {env:VAR}. A data value can be that token, though: "password": "{env:ADMIN_PASSWORD}" resolves from the environment at replay, and recordings, reports and traces mask it as they would in the step itself.

Where it works. Step text, data table cells and doc strings, in .saffron and .feature files alike, and inside step sets. The folder is data by default; "dataDir" in saffron.config.json moves it.

Single values

What happens to it. {data:...} is in the same family as {env:VAR} and {date+1}. While recording, the agent is given the real value; before the recording is saved, that value is turned back into the token. Every replay then reads the file again. So:

Change a value in the file, and the next replay uses it, at zero tokens. Nothing is recorded again.
If a recording still has the OLD value written out somewhere (the agent matched a sentence that contained it, say), changing the value makes that scenario stale and it records again, rather than quietly typing or checking yesterday's data. The same rule data tables follow.
A file or key that does not exist stops the run before any browser opens, naming the reference, what is missing and the scenario using it.
Where a step uses a value (typing it, matching it), the reference must point at ONE value; pointing at an object is an error that suggests the entry to use. A list is for the step below.

Checking a whole list: "every"

Point a Then step at a list and say that every value should be there:

Then the status filter should list every {data:enums.OrderStatus}
Then the roles table should show every {data:users.roles}

The recording holds ONE assertion that names the list and the place (dropdown, table, list, menu), not one check per value. Replay reads the list from the file each time, so adding "Refunded" to the JSON makes the next run check for it, at zero tokens, and a missing one fails by name: "Refunded" from {data:enums.OrderStatus} is not shown in the target (4 of 5 found). It checks that every listed value is present, matched as whole text ("Paid" is not satisfied by "Unpaid"); extra entries such as "All" or "Select…" are fine. It works on a closed dropdown too.

Counting what is shown

A step that says how many are shown is one check, not one per element:

Then 3 rooms should be listed
Then <rooms> rooms should be listed
Then {data:hotel.rooms} rooms should be listed

It counts the elements its target finds that are shown, in the page or the frame the target names, never another frame's; a hidden one (a template card, a closed menu's copy) never counts. A failure says how many it saw: the target to show 3 element(s); it shows 2. The number is 1 or more: for none, write the step as "no rooms should be shown", which checks that nothing is shown. saffron eject exports it as Playwright's own count of shown elements (expect(locator.filter({ visible: true })).toHaveCount(3)), when the number is written in the recording, its row or its data.

Examples from a file

In a .saffron file, a Scenario Outline can take its rows from the data folder instead of an inline table:

Scenario Outline: Role sees its menu
  Given I sign in as <role>
  Then I should see the <menu> menu

  Examples: {data:roles}

{data:roles} reads data/roles.csv (written exactly so: {data: roles} or {Data:roles} is an error, never an Outline that runs no row), whose header row names the placeholders (a quoted field keeps its own spaces, an unquoted one is trimmed), or a JSON list of records ({data:users.accounts} works too). An outline records once and replays once per row, so adding a row to the file is one more zero-token replay, with nothing recorded. Rows are numbered "(example 1)", "(example 2)" like inline ones. Each row replays with its own values, tokens included: a row whose cell is {data:users.guest.email}, {unique:order}, {date+2} or {date+2:DD.MM.YYYY} enters that row's value, never the first row's. When two columns held the same value in the row that was recorded, Saffron cannot tell which one a step typed, so it keeps the value as typed; a later row where they differ records again instead of replaying the first row's value. A step naming only one of the two columns (I choose "<currency>" beside a display holding the same NOK) wrote that one, so the other rows replay on it. A {data:...} value a step enters while not naming the column (a StepSet's fixed {data:guests.first.surname}) is that step's own, though a row's cell reads the same: the other rows replay on the same recording. A value a step only checks may have reached the page from the row, so the rows that differ are refused there. A placeholder the file has no column for is an error naming both; a .feature file that tries this is told to rename itself to .saffron.

A recording bound to its row. One recording serves every row only where each row's value can be written as <column>. Two shapes cannot be: a field that still holds other rows' values beside the recorded row's (a check that Red, Green, Blue are all offered, Plan changed: Basic to Pro), and the row's value glued inside other text in a step that does not name its column (alice2024@example.com after When I register <user>). A third is a response status, method or attribute that a step not naming the column checks as the row's value (Then the API answers as expected, recorded as 201 on the row whose <code> is 201). Saffron cannot tell which part belongs to the row, so it keeps that field as recorded and marks the recording row-bound for that column. The recorded row, and rows with the same value, replay as usual. Every other row fails red with a message naming the column and the step, at zero tokens: it is not healed or recorded again, and the agent is not called. To fix it, put <user> in the step that uses the value, give the text a check holds an Examples column of its own (| palette | holding Red, Green, Blue, | from | holding Basic), or move a check that does not depend on the row into a Scenario of its own. saffron run --rerecord on a row still records it, on that row.

A check that names a column no step of the recording writes (Then the <who> entry says "Approved") binds the recording too, unless it runs in the browser session that column names (after As "<who>" ...): there the session itself follows the row. Run in another session (As "admin"), it is bound, as the admin page shows every person's entry.

While recording, the agent is told which values the scenario writes as references (each <column> with this row's value, each {data:...} with what it resolves to) and types the reference itself: browser_type text='<plan>' fills in this row's plan, and the recording keeps <plan> as written, as it keeps {env:...} (§8). A request does the same (§23). Nothing then has to find the value in the recording and turn it back into its reference. Dates are the exception: the agent is shown each day, never its token, and checks it as the page shows it, so a day it types or picks is turned back into its {date} token afterwards, as are values the page shows (a calendar button's name). The row proofs and the clock check (§5) then check the result.

A value one step captures and a later step types (a booking number shown once, then entered at sign-in) works the same way: the agent captures it with the text it sees, writes {captured:name} where it types it, and the browser gets the value while the recording keeps the reference. A captured credential (a login's token a request handed out) is never typed through the browser: send it with a request, or sign in through the page with the scenario's {env:...} values. When the element holds more than the value (a booking number beside its Copy button's label), the capture takes a pattern whose one group is the value (([A-Z]{3}\d+), which may hold {captured:...}, {date...} and <column> tokens, and a pattern that backtracks, such as (a+)+, is stopped and named instead of freezing the run); replay keeps that part, a differs check against it compares the same part of the text, and replay fails naming the text when the pattern finds no value or two. A captured value can also name the element a later step looks for (text: "Order {captured:booking}"): replay looks for the value, and a recording holding one is schema version 17.

The recorder also keeps a step to what worked and a check to checking:

An agent that performs a step again after an attempt that did not work (a wrong element, a navigation that went nowhere) drops the earlier attempt, so the step records the attempt that worked, not both.
A click or fill through a locator the agent wrote, not an element of the snapshot, can name a part of the element's name (getByRole('link', { name: 'Sign in' })), which Playwright matches live and replay, matching a name whole, would not find. The recorder takes the full name from the page the agent saw last, where one element of that role holds it ("Sign in to your account"). Where none or several do, the proposal says so: check its proof, or record the step again.
A navigation the agent leaves for the same page with another query or fragment, before doing anything there (trying /search?q=ada, then /search?query=ada), was a trial: it is not recorded, and the proposal says it was discarded. A navigation to another page stays (/logout, then /), and so does one the agent acted on.
A Then never enters, presses keys, picks, ticks, uploads, drags, navigates, opens a tab at an address or sends a request that writes (POST, PUT, PATCH, DELETE). The browser action and the write request are refused before they run: the agent is told the step is a check, nothing reaches the page, and nothing is recorded. If the condition does not hold without the action, the step fails and the agent suggests the action as a When before it. An action that got through some other way is not recorded, and the proposal gets a check changed the page: note, held from accept --all: the proof ran without it, so a check that held only after it passed by chance. Move it to a When and record again. A read the API offers only as a POST may be sent under a Then: a GraphQL query to a /graphql endpoint whose document holds no mutation (comments and strings aside), or a request to an endpoint whose last segment is search or query.
A click on a button that submits or commits (its name starts with a word such as Submit, Save, Log in, Sign in, Send, Confirm, Continue, Pay, Place, Add, Delete or Cancel) is refused under a Then the same way: clicking Log in, or submitting again to bring back a message that vanished, is what a When does.
Any other click, or a dialog answered, under a Then is recorded, as it may only open what is checked (a tab, a section), and held with the same note: one that does what the When should (a click that cancels the order the check reads) would let the check pass on its own action, and nothing the run sees tells the two apart. Accept it by file once you have seen that it only opens.
A step about how an element looks (an invalid field's red border) is checked on its computed style: a CSS property's value as the browser computes it (border-color is rgb(220, 38, 38)), or a pattern, on an element that is shown: a hidden error is not red on screen, and a shown fallback never answers for a recorded element that is on the page but hidden. A check of what can hide an element holds on a hidden one: the styles display, visibility, opacity, content-visibility, a height, width, transform or clip, and the attributes hidden, aria-hidden, inert, class, style, open and data-state (a d-none class, a closed dialog). The agent reads it from the page first, and replay and an exported Playwright test check it alike, in Chromium, Firefox and WebKit. A recording holding one is schema version 16.
A message the page shows for a moment (a toast, an error flash) is watched for from before the action that shows it: the agent arms a watch in the When step, and the Then checks the message was seen since, gone by then or not. Replay watches the same way, reading the element every 50 ms, so a message shown for less is missed. Only a message that appears after the watch is armed counts: one already on screen from an earlier action counts once it has stopped showing that text and shows it again. A check of the message uses its watch up, so a second check needs a watch armed again. The agent never performs the action again inside a Then to bring the message back. A recording holding a watch is schema version 15, and saffron eject writes its scenario as test.fixme for now.
A URL check is recorded the way the step says it: that the URL contains a value (the default), that it does not ("the URL no longer holds the login token"), or that its path is exactly one ("I am on the /account path exactly", its query and fragment aside). A containment check reads its value as a whole part of the URL, and a path (a value with a /) outside the query, unless the value names one (?, =, &): /dashboard does not hold on /login?next=%2Fdashboard, nor /orders/42 on /orders/421 or /orders/42.5, nor name=Jos on ?name=Jos%C3%A9 (a letter or digit written percent-encoded belongs to its word), while shoes holds on /search?q=red+shoes (but not dashboard on /login?next=/dashboard, a query value that is a path, nor /dashboard on /login?next=%2F%23%2Fdashboard, where the encoded # is part of the query). A value that names a query is the page's own: its ? is where the URL's query starts, and a field it names (tab=settings) is one of the URL's own fields, so neither holds on a login page that carries the page in its query (/login?next=%2Fdashboard%3Ftab%3Dsettings). A value with a # names the URL's own fragment, and a field may be one of the fragment's: #tab=settings, /app#tab=settings and tab=settings all hold on /app#tab=settings. A query value that is a path in a fragment's fields (#state=%2Fdashboard) is no part of the path either. A "does not" check reads its value anywhere in the URL: error is in ?errorCode=5. The recorder refuses a "does not" or "exactly" check the step's words do not name, and a containment check where the step says the URL no longer holds the value. Replay and an exported Playwright test check them alike. A recording holding one is schema version 14, which an older runner refuses rather than reading it as containment.
A check the step names as a comparison is recorded as one: "starts with" or "ends with" followed by its value (starts with "Oct") as an anchored pattern (a pattern the agent writes for it without the anchor is refused), a number the step equates ("the same day as") as a number, never as text the page merely contains.
A shortcut (ControlOrMeta+v) or a modified click replays as the browser's own platform has it, Meta on a Mac and Control elsewhere: a remote Linux browser driven from a Mac pastes too.
A suggested rename of a step to an existing wording that recorded the same actions is made only when the two wordings share a word, which the suggestion names: a dialog's "Cancel" and an order page's "Discard changes" click alike and mean different things.

The safe habit: write <column> in every step whose text or typed value depends on the row (INV<date>, orders-<region>, an email built from <user>), and keep a check that lists every row's value out of the Outline. A value recording cannot write as <column> (another case, a slug, URL-encoded, words joined) binds the recording to its row the same way, so the other rows fail with guidance rather than recording each other again. This also covers one- or two-character values in actions and assertions (orders-US, ?page=1), dates without a year or glued to other text (due 13.10, INV20261013), and a recorded blank cell where the step's actions have no placeholder for that column. For a blank cell, record on a filled row so Saffron can identify the value, or give the blank case its own Scenario. When the first row is blank and its step types <column>, the first filled row records once more, and that recording serves the blank rows as well. An element role taken from a row cannot vary in a shared recording; use separate Scenarios for those roles.

A step that names a column but writes nothing of it. When I book for "<mode>" recorded on the "myself" row, where the page already showed "myself", recorded no action, and When I set booking for someone else to "<other>" recorded a tick of a box that holds no row value. Either would do the same on every row, so the recording is bound to its row and the other rows are refused. Only the step's own actions count: another step typing <mode> does not make this one pick it. saffron status marks a refused row with ✗ and gives the reason. Picking an option or radio labelled with the row's value (click "Someone else") writes <mode> and serves every row; a yes/no box cannot, so give each value its own Scenario. A recording saved before 1.0.1 with such a step is recorded again once on a row that differs. A date cell written with its month's name ({date+1:MMM D}) follows the row in every step, as one with its year does. So does a {data:...} cell whose value holds a date: its day is read as replay types it, and a recording of such a row is schema version 18.

A whole {env:...} or {data:...} field that equals a row's value is ambiguous even if the step explicitly names that token. Other rows are refused because the recording cannot tell whether the field meant the token or the column. Record on a row with a distinct value, or separate the independent check. This can also refuse a legitimate login that happens to equal the recorded row's value. A value merely contained inside the login does not cause this refusal.

Step reuse also refuses a changed quoted argument containing several placeholders ("<first> <last>"): it cannot safely split the replacement back into columns. The step records normally instead.

A state check must follow the column it names even when that column starts its sentence (Then <state> remains the save button state). Writing the column in another step does not prove the check follows it. This can also bind a descriptive step such as Then <user> sees the dashboard, which checks only that a dashboard is shown: the refusal says so. Make the checked value follow the column (Then the dashboard greets "<user>", recorded as the greeting's text), use an explicit browser session for the user, or give the cases separate Scenarios. A keyboard press or other browser action in the assertion step does not relax this check.

A blank cell in a check. A blank cell or {data:...} value means "nothing there" in an expected text or text pattern (Then I see no error "<error>"): it passes only on an element that shows no text, and in an attribute's pattern only on an empty or missing attribute. Where blank cannot mean that (an expected URL, a response's URL, status, method or body, the name or text of the element a step finds, a selector's text), the step fails for that row with a message naming the field, instead of matching anything. Expected text that keeps other text around a blank value (Total: <amount>) also fails: it cannot prove the missing amount. So do a download check's text or pattern, an expected URL (/orders/<id> is inside every order's URL) and the text, label or placeholder of the element a step finds (Order <id> finds "Order 42"). A role's name matches exactly, so Save<n> with a blank n is the "Save" button. Text patterns follow this rule too, including blanks inside nested tokens. Patterns that intentionally accept an empty string, such as ^<prefix>\d*$, keep their existing behavior when the prefix is blank. Give that value content or put the blank case in its own Scenario with explicit expected text. A wholly blank expected value still checks for no displayed text. This also catches blanks introduced through an Examples cell or data file, such as a cell containing Saved {env:NAME} with an empty NAME. Tokens inside environment values remain literal text. When recording, saffron_assert requires expected for text and input-value checks; an explicit "" means an empty check.

Exact assertion targets and values. An assertion's recorded role name matches the full accessible name: Save does not match Save draft. Use nameRegex when a partial name is intended. Attribute patterns are case-sensitive: ^/Orders$ does not match href="/orders"; write the intended case or explicit alternatives in the pattern. Input-value checks require the input to be visible. Existing recordings that relied on a partial role name, ignored attribute case or hidden input now fail and must be corrected or re-recorded.

Text conditions in every selector-chain segment, including container selectors and single-quoted text, follow the row-value boundary rules. Optional whitespace before or after a value in a regex does not relax those boundaries. A row value inside :has(...) is checked on the element the inner selector finds, and text under two negations (:not(:not(...))) counts as present. A selector that checks a row value on an element before a sibling combinator (p:has-text("Order <id>") + button), in a selector list's branch too, is refused: the check would land on the sibling. A row value in a nested XPath predicate (//tr[.//a[contains(@href, '/orders/<id>')]]) is checked on an element that meets that whole predicate; one in a predicate that is no plain path to its element is refused. Record a selector that finds the element by its own text or attribute.

An expected text, a row value or one written as it is, matches as a whole: Order <id> for 1 cannot match Order 12, "$10" cannot match $100, and a text target Saved cannot find "You have unsaved changes" (a role's name may still have grown a suffix). A number written in the step may be followed by a zero decimal part ("$10" in $10.00, never $10.50, ₹1,00,000 or 2.0.1), where a row value keeps its decimals whole (<id> 1 is not in 1.00). Neither is found before a digit in an inline tag (Order 12, $10.50). A text ending in a number's point or comma takes no more digits ("Refund of $10." is not in Refund of $10.50), and a number is not found in its negative ("5.00" is not in -5.00, nor "$5.00" in -$5.00; "5" is still in pages 3-5). A value that starts or ends in a script written without spaces (Chinese, Japanese, Thai, Lao, Khmer, Myanmar) is not bounded there: "保存しました" is in "設定を保存しました", though never after a negating prefix ("成功" is not in 不成功 nor 没有成功, "保存" not in 无法保存). Only a negation right before the value is read: write the whole phrase when a sentence negates it further back. Nor does a value run across a block, a table cell or a  : "15.00" is not in the cells 1 and 5.00. Items of a flex row or a grid read as one line, a space apart, so "Total: $10" holds on the items Total: and $10. Row values in selector regex fallbacks and download checks must match the whole value too. Playwright text filters concatenate adjacent nodes, so container text needs a separator between a row value and the next word; otherwise target the value's element. Regex tokens inside character classes remain characters ([<chars>] for ABC matches A, B or C); they do not receive word-boundary assertions or introduce character ranges. Boundaries include Unicode letters, numbers and combining marks: Order 1é cannot satisfy Order <id> for 1. Unicode regex mode is enabled when these boundaries are needed; recorded regexes must be valid in that mode. Tokens inside regex repeat counts (A{<count>}) are refused. Use a separate Scenario with a literal count, such as A{3}. Escaped literal braces and ordinary numeric quantifiers keep their meaning. So is a quantifier right after a row value (<id>+): it would repeat only the value's last character.

The same boundary holds where a row value starts or ends a role's name, a label, a placeholder, an attribute (alt, title, [aria-label*="..."], XPath contains(@href, '...')), :text(...), :text-matches(...), an inner has locator or a frame's selector. Beside a digit, a decimal or thousands part belongs to the number: Total <n> for 1 matches neither Total 1.5 nor Total 1,000, unless the pattern writes the decimals itself (<n>\.\d{2}). Thousands grouped by a space or an apostrophe count too: 250 kr is not in 1 250 kr, nor 250 CHF in 1'250 CHF. A value is read as running on into a following group only when it ends in one to three digits that are not part of a word (Room 1 is not in Room 1 250, while Galaxy S24 is in Galaxy S24 256 GB, and so is 256 GB), and never by a "not shown" check: that one fails while iPhone 15 is on screen in iPhone 15 128 GB. A "not shown" check reads only a decimal point as part of the number, so 2 counts as shown in Tags: 1,2,3, and $5.00 in US$5.00. Where the recording glued a row value into a word (<item>s, recorded on apples), a "not shown" check reads the glued letters as any letters, so row box counts boxes as shown. That holds only where a letter of the value meets a letter, outside scripts written without spaces: INV<n> and Japanese text keep their bounds. Digits of every script count, Thai and Myanmar too, and so do the Arabic decimal point and thousands separator (١٫٥). An amount that starts with its currency symbol is not found after a letter: $5.00 is not in R$5.00 or CA$5.00. The value is checked on the element that holds it, an ancestor too (tr[data-key^="<id>"] > td). Text a selector says must be absent (:not(...), XPath not(...), has-not-text) is left as Playwright reads it.

URL-encoded secrets are masked before recordings are saved, including lowercase and mixed-case percent escapes and escaped regex patterns. The same masking protects capture names, reports and traces, including case-changed encoded forms of long secrets.

Setup steps use the feature's declared language. For example, Latvian Kad is a Given even though other languages use the same word for When. An inherited And after a Given may also describe an already satisfied state.

A missing attribute. An attribute check whose pattern takes only an empty value (disabled checked as ^$) fails when the attribute is gone, and so does one taking any value (.*) or anything but false (^(?!false$) for "is pressed"). A pattern that takes an empty value and false but not true (^(false)?$ for "not invalid") holds on a missing attribute too.

Coincidences that bind a recording. A response status, method or attribute name that a step checks without naming a column binds the recording to its row when it equals that column's value on the recorded row, or what a {data:...} reference keyed by columns reads there: an amount of 200 beside a check of status 200, a post beside method POST, a disabled state beside the disabled attribute. The other rows fail with a message naming the column. If the check follows the row, name the column in that step; if the value only coincides, give the check its own Scenario, or give the expected value its own column.

A pending proposal and the other rows. A proposal filed for one row belongs to the row with those values, wherever the row moves in the table: saffron status, the plugins and the review page name and show that row as the rows stand now. Another row it does not replay for is red, never recorded over it. When that row failed in a step before any step uses its own values (the app ran out of stock in a shared checkout step), the report blames the app's state, not the recording: put the app back in the state the scenario starts from and run the row again.

Recordings from 0.9.x. A recording 0.9.x made for an Outline does not say which row it was recorded on. A row it still holds another row's value for is recorded again once; the new recording names its row. Under --no-agent, as in CI, that row stays red, with a message saying the recording comes from an older saffron-ai, until it is recorded once. An Outline whose rows check different response statuses without naming them in the step is not caught that way: record it again with saffron run --rerecord after upgrading.

Values that must be new on every run: `{unique:name}`

A scenario that creates something (a user, an order, a project) collides with its own leftovers the second time it runs, unless what it types is new each time:

When I register as user-{unique:id}@test.com
Then the welcome banner should greet user-{unique:id}@test.com

{unique:id} is different on every run and the same everywhere inside one run, so a later step finds what an earlier step created. It is 10 lowercase letters and digits, starting with a letter; {unique:id:digits} is 9 digits for numeric fields. While recording, the agent types the generated value and Saffron turns it back into the token. When a run is not green, the report lists what each token stood for, so you can find the record it left behind. A name is letters, digits, _ or -: a {unique:...} that is not a token (unclosed, or an Examples row that leaves {unique:<id>} empty) stops the run before it starts, naming it, with exit 2.

Data per environment

--env staging (or SAFFRON_ENV=staging, or "env": "staging" in the config) lays data/users.staging.json over data/users.json: objects merge key by key, lists and plain values are replaced, and a file that has no staging version is used as it is. The step keeps saying {data:users.admin.email} on every target. For Examples, a roles.staging.csv replaces roles.csv whole. saffron status shows the selected environment and any reference that does not resolve in it.

In the editor

Both IDE integrations complete {data: (files first, then keys, showing each value), show the current value on hover, jump to the key in the data file, and underline a reference that does not resolve. {data:...} and {unique:...} are highlighted like {env:VAR}.