CI and review

Export to Playwright Test (beta)

Beta. Export works and keeps its promise of never being looser than replay, and it is still being shaped by use: what it cannot write faithfully yet (a scenario that reads a fixture) stays a test.fixme with the reason, and its output may change between releases.

saffron eject writes the committed recordings as ordinary Playwright Test specs. From then on they are your code: they need no saffron-ai, no agent and no .saffron/cache, and Saffron does not keep them in step with later recordings. Use it to run Saffron scenarios inside an existing Playwright suite, or to leave Saffron.

npx saffron eject                      # every feature file
npx saffron eject features/login.feature --out e2e
npm install -D @playwright/test
npx playwright test

It writes:

  • one spec per feature file (playwright/login.spec.ts), a test per scenario and per Examples row, each Gherkin step a test.step named with its text;
  • playwright/saffron-helpers.ts, which the specs import: bounded text checks, {date...} and {unique...} values, {env:...} values read when the test runs, network checks, captured values, browser sessions, tabs, downloads and file choosers, and API requests with the checks on their responses. It is copied, so it is yours too;
  • playwright.config.ts, only when the project has none, with baseURL (or BASE_URL from the environment), timeouts, trace and screenshots from saffron.config.json.

Never looser than replay. An exported test may fail where replay would pass, never pass where replay would fail. So:

  • An action uses the first locator replay would try. When that one is gone the test fails, where replay would try the next. The summary counts the actions per test that had more.
  • Text checks read the first match's text as replay reads it: text nodes as the page holds them, so CSS text-transform and line breaks between blocks count in neither (replay alone matches a value written on several lines, a two-line doc string, where the page draws those lines: an exported test is stricter there, never looser); expectVisible does not pass over a hidden first match for a displayed one, and expectEnabled and expectDisabled check that first match is shown before they check it is enabled or disabled; expectEach needs each value as a whole line of the container's text, in the case written.
  • What has no faithful form yet is written as test.fixme with the reason, never with the action left out: expectAmount, a schema check with nothing to check against (no contract pinned, no operation for its request, a $ref outside the contract), a scenario of a feature with a Fixture: block, a request group sent once per table row and its checks, a selector or a pattern that holds a row or data value, a JSON field path that holds a value only the running test knows, any field the exporter does not know, a recording that is stale or unreadable, a scenario never recorded or with a pending proposal, a check whose recording checks nothing, an {env:...} name the scenario does not write, a browser step that carries a credential the API handed out ({captured:token}), whose origin replay checks and an exported test cannot yet, and an Outline row replay refuses.
  • A page the spec opens is a web page (http or https, as replay opens), and a file it uploads is one of the project's own: never a dotfile, a folder, or a path outside the project or through a link out of it.
  • An {env:...} value is never written into a file: the spec reads it with s.env("NAME") when it runs. {data:...} values are written as they are when you eject, so a later change to the data file does not reach the spec.

API steps. A request (§23) is written as run.send({...}), sent through the browser session's own request client, so a call after a UI sign-in carries its cookies, and the checks after it as run.expectStatus, run.expectJson, run.expectHeader and run.expectDuration. The helper carries a copy of Saffron's own JSON field checker, so a check reads a response exactly as replay does, in an API step and in the browser's response checks alike. An extracted value is read with run.captured("name") wherever a later step repeats it, a typed body field with s.typed(text, "number"), and a polled request (until) is sent again in the test as it is in replay. Relative paths go to api.baseURL, which API_URL in the environment overrides, and a named API's to api.services.<name>.baseURL, which API_URL_<NAME> overrides (the name upper-cased, other characters as _); without api.baseURL the default API is the test's baseURL. A request that carries an {env:...} value is sent to its API's base URL and nowhere else, and none goes to a link-local address, as in replay. An @api scenario is an ordinary test: Playwright opens its browser for it, and the requests go through that context's client. The request transcript is not written; Playwright's own trace and report stand in. Playwright's trace records typed values and headers unmasked, so a suite that reads secrets ({env:...} values, or credentials the API hands out) is written without trace in its config, with a comment saying why: add it back only where those values may be seen. When the project already has a playwright.config that records a trace, eject leaves it as it is and prints a warning instead.

saffron eject replaces the files it wrote before, unless you edited them since: an edited file, or any other file of the same name, is kept and listed (--force replaces it). The folder and the feature files must be inside the project. Two feature files whose names would give the same spec file (my shop.feature and my-shop.feature) stop the command before anything is written; eject them separately with --out. Browser sessions (As "buyer") start signed in with the project's storageState unless the session has its own, as in replay. An {env:...} variable that is empty counts as unset. --json prints each test's outcome: exported, stricter or fixme, with the reasons. It exits 1 when nothing could be exported.