Writing tests

The action vocabulary

Actions (healable): goto click fill press selectOption check uncheck hover waitFor captureText handleDialog uploadFile dragTo waitForResponse. Targets may carry a frame selector, elements inside iframes record and replay through it. Assertions (never healed): expectVisible expectNotVisible expectText expectValue expectUrl expectDiffers expectMatches expectAttribute.

Real-world page furniture is covered: native dialogs (alert/confirm/prompt) replay via an armed handleDialog placed before the triggering action (a blocking dialog needs its handler first); file uploads set files on the input or arm the next file chooser (paths resolve from the working directory); drag-and-drop replays dragTo between two semantic targets; iframes scope any locator via target.frame. All four record live and replay green on Chromium, Firefox and WebKit (validated on the example suite; each recorded for $0.18–$0.66). Assertions recorded without frame scoping still replay honestly: positive lookups (expect*/capture) fall back to searching child frames: main page first, so an agent that misses an iframe boundary produces a cache that verifies anyway. Hand-authoring a cache (recordedBy: "manual") remains a supported, replay-verified path for anything unusual.

Network-aware steps: UI tests often need to wait on the backend, not the DOM: "When I place the order and wait for the order API to return 201", "And I wait until the job status API reports READY". No new keywords: the step is plain prose, and what's new lives in the action vocabulary: waitForResponse (healable) blocks until a matching response is observed, and expectResponse (a Then, sacred, never healed) audits that a matching response occurred during the scenario: "Then the order request should have returned 201" is an API contract check. Matchers cover URL pattern (path-only regex, never a literal volatile ID), method, status (exact or class like 2xx), and a body regex for polling waits ("status"\s*:\s*"READY"). Recording is verify-first: the agent confirms the response actually happened via the browser's network log before recording the matcher. At replay a rolling response log on the browser context does the matching, zero tokens, covers new tabs, and waits are satisfied by responses from the triggering step onward, so "click, then wait" never races.

The dynamic-content trio came out of running Saffron against a live booking site:

  • captureText reads an element's text and stores it under a name (saveAs: "SEK price").
  • expectDiffers asserts a live element or a second captured value differs from a stored one: real functional comparison instead of frozen literals.
  • expectMatches / expectAttribute assert patterns (NOK [\d,]+, href ~ ^https://…) instead of exact volatile strings. Attribute assertions let "X should link to Y" verify the href without navigating.

Locator precision. A recorded accessible name is matched exactly. Without that a step recorded against "Save" can click "Save draft" when it appears first in the DOM, with no AI involved and nothing to notice in the report. Assertions and captures fall back to a substring match, so a label that merely gained a suffix still replays; interactions do not, because acting on a control whose name only contains the recorded one is a silent substitution. A renamed button fails, and a failure is what healing needs to see. A control that matched but cannot be used (disabled, covered) is also a failure rather than a reason to try the next candidate.

Tabs. The replayer follows new tabs: a click that opens target=_blank moves the action stream to the new page. expectUrl checks the active page only: a URL that merely exists in some other tab is not evidence that the step under test got there. Multi-tab scenarios are plain prose, no keyword: "When I switch back to the first tab", "And I close the current tab", "When I open a new tab at the admin page". During recording the agent uses Playwright's tab tool, and each select, close and new is cached as switchTab, closeTab or newTab (0-based, creation order), so replay changes tabs in exactly the recorded order. An explicit switch is not undone by the automatic follow: only tabs the run has not seen before are followed. Closing the active tab falls back to the last tab open; closing the last tab fails the step.