Writing tests

The action vocabulary

Actions (healable): goto click fill press selectOption check uncheck hover waitFor captureText handleDialog uploadFile dragTo waitForResponse waitForDownload. Targets may carry a frame selector, elements inside iframes record and replay through it. Assertions (never healed): expectVisible expectNotVisible expectText expectValue expectUrl expectDiffers expectMatches expectAttribute expectResponse expectEach expectDownload expectAmount.

Tokens a step (or a recording) can carry, all resolved when the action runs rather than when it was recorded:

Token Stands for Example
<name> an Examples column, a table cell (<table:key>), a doc string I enter "<username>"
{env:VAR} a secret from the environment, masked in recordings, reports, history and traces (values of three characters or more) {env:ADMIN_PASSWORD}
{data:file.key} a value, or a list, from data/*.json {data:users.admin.email}
{unique:name} new on every run, the same within one user-{unique:id}@test.com
{date+N} a date relative to today, in a format and locale, or a range {date+1:DD.MM.YYYY}, {date+7 - +9:MMM D, YYYY}

Real-world page furniture is covered: native dialogs (alert/confirm/prompt) replay via an armed handleDialog placed before the triggering action (a blocking dialog needs its handler first); file uploads set files on the input or arm the next file chooser (paths resolve from the project root); drag-and-drop replays dragTo between two semantic targets; iframes scope any locator via target.frame; downloads arm a waitForDownload before the click that starts one, and replay keeps the file in the scenario's artifacts, where expectDownload checks its text (contains, or matches a regex; UTF-8 text up to 5 MB, a calendar file's folded lines unfolded), so "the calendar file should contain DTSTART;VALUE=DATE:{date+7:YYYYMMDD}" checks next week's day on every run. The first four record live (on Chromium, which recording needs) and replay green on Chromium, Firefox and WebKit (validated on the example suite; each recorded for $0.18–$0.66). Assertions recorded without frame scoping still replay honestly: positive lookups (expect*/capture) fall back to searching child frames: main page first, so an agent that misses an iframe boundary produces a cache that verifies anyway. Hand-authoring a cache (recordedBy: "manual") remains a supported, replay-verified path for anything unusual.

Network-aware steps: UI tests often need to wait on the backend, not the DOM: "When I place the order and wait for the order API to return 201", "And I wait until the job status API reports READY". No new keywords: the step is plain prose, and what's new lives in the action vocabulary: waitForResponse (healable) blocks until a matching response is observed, and expectResponse (a Then, sacred, never healed) audits that a matching response occurred during the scenario: "Then the order request should have returned 201" is an API contract check. Matchers cover URL pattern (path-only regex, never a literal volatile ID), method, status (exact or class like 2xx), and a body regex for polling waits ("status"\s*:\s*"READY"). Recording is verify-first: the agent confirms the response actually happened via the browser's network log before recording the matcher. At replay a rolling response log on the browser context does the matching, zero tokens, covers new tabs, and waits are satisfied by responses from the triggering step onward, so "click, then wait" never races.

The dynamic-content checks came out of running Saffron against a live booking site:

  • captureText reads an element's text and stores it under a name (saveAs: "SEK price").
  • expectDiffers asserts a live element or a second captured value differs from a stored one: real functional comparison instead of frozen literals.
  • expectAmount asserts a live element shows the amount an expression over captures states ("before" + "breakfast", "night" * 2), compared to the cent whatever the currency format (NOK 1 234,50, $1,234.56): a total proven right, not merely changed.
  • expectMatches / expectAttribute assert patterns (NOK [\d,]+, href ~ ^https://…) instead of exact volatile strings. Attribute assertions let "X should link to Y" verify the href without navigating.

Locator precision. A recorded accessible name is matched exactly. Without that a step recorded against "Save" can click "Save draft" when it appears first in the DOM, with no AI involved and nothing to notice in the report. Assertions and captures fall back to a substring match, so a label that merely gained a suffix still replays; interactions do not, because acting on a control whose name only contains the recorded one is a silent substitution. A renamed button fails, and a failure is what healing needs to see. A control that matched but cannot be used (disabled, covered) is also a failure rather than a reason to try the next candidate.

Tabs. The replayer follows new tabs: a click that opens target=_blank moves the action stream to the new page. expectUrl checks the active page only: a URL that merely exists in some other tab is not evidence that the step under test got there. Multi-tab scenarios are plain prose, no keyword: "When I switch back to the first tab", "And I close the current tab", "When I open a new tab at the admin page". During recording the agent uses Playwright's tab tool, and each select, close and new is cached as switchTab, closeTab or newTab (0-based, creation order), so replay changes tabs in exactly the recorded order. An explicit switch is not undone by the automatic follow: only tabs the run has not seen before are followed. Closing the active tab falls back to the last tab open; closing the last tab fails the step.

Browser sessions. A step that starts As "buyer" runs in the browser session named "buyer": a browser context of its own, with its own cookies, storage and tabs, so two people (a buyer and a seller, two chat users) act in one scenario. The session carries over to the steps after it until another As "..."; steps before the first one run in the default session. The parser gives each step its session, and nothing new is cached: the session follows from the step texts, and any change to it changes some step's text, which already makes the recording stale. Replay keeps a lane per session (its context, the tab it is on, the tabs it has seen, its own network log, so a session's network checks see its own responses) and opens a session's context the first time a step needs it; the first step's session uses the scenario's own context. Tab indexes count within a session. One agent records, heals and refines across all the sessions: Saffron gives it a Playwright backend per session and sends each browser call to the session of the step it last announced, each backend starting on the tab its session is on. That needs the in-process browser bridge, so a scenario with several sessions uses it whether tracing is on or not. Each session's context gets sessions.<name>.storageState when configured, else storageState. Each session after the first is traced on its own (trace.<name>.zip beside trace.zip, the name lowercased with dashes), and the report tags the step where each session takes over. As "default" returns to the default session; a step that reads as a session step and names none (As "" ...) is a parse error with its file and line. Seeding keeps a step's As "...": another person's recording may type that person's own values, so it never seeds a step for someone else; the editors' vocabulary offers steps without the prefix. A duplicate-wording rename never changes a step's As "...", and a feature edit that would is skipped with the reason. Pages Saffron opens for a session are never mistaken for tabs the agent's actions opened.