Writing tests

Setup and teardown

A scenario assumes the application is in a known state: "cancel an order" needs an order to exist. Most scenarios also leave something behind. Without preparing state before a run and cleaning up after it, a suite passes on a fresh database and fails on the second run.

Saffron has no code layer, so it hands this to whatever you already have: an npm script, a SQL file, a curl.

{
  "baseURL": "http://localhost:3000",
  "setup": "npm run db:seed",
  "teardown": "npm run db:clean"
}

Each key takes one command or a list, run in order:

{
  "setup": ["npm run db:reset", "npm run db:seed", "node scripts/create-test-users.mjs"],
  "teardown": ["node scripts/delete-test-users.mjs"]
}

What runs when

setup Once per saffron run, before any browser opens, in the project root.
teardown Once per saffron run, after the last scenario and after the browser has closed.
A setup command fails The run stops with exit code 2 and no scenario runs. Scenarios against a half-prepared application only produce misleading reds. The rest of the setup list is skipped, and teardown still runs, because the failed setup may have created half of its data.
Scenarios fail Teardown still runs.
You press Ctrl-C Saffron stops whatever is running, a setup command included, waits for it to be gone, closes the browser, runs teardown, then exits with code 130. Teardown itself is never interrupted.
A teardown command fails The scenario verdicts stand and the report is written. The remaining teardown commands still run. If every scenario passed, the run exits with code 2 instead of 0, because what was left behind is the next run's false red.
A command hangs It is stopped after hookTimeoutMs (five minutes by default) and counts as failed.

What the commands receive

They run through the shell with your environment, plus these variables, so one script can serve every target:

Variable Value
SAFFRON_BASE_URL The base URL of this run: --base-url, or baseURL from the config.
SAFFRON_ENV The data environment of this run (--env, SAFFRON_ENV or env). Empty when none is selected.
SAFFRON_SHARD The shard of a sharded run, such as 2/4 (§15). Empty otherwise. Every shard runs setup and teardown, so use it to keep the machines' seed data apart.
// scripts/create-test-users.mjs
const api = process.env.SAFFRON_BASE_URL + "/api/test-users";
await fetch(api, { method: "POST", body: JSON.stringify({ email: "admin@test.com" }) });

Their output is shown as they run, on stderr. The report lists each command, how it ended and how long it took.

Skipping them

npx saffron run features/orders.saffron --no-hooks

Use --no-hooks when the state is already prepared, for example when you re-run one scenario against an application you seeded a minute ago.

Setup, teardown and {unique:name} together

They solve different halves of the same problem:

Data Use
Data scenarios only read (a catalogue, an admin account) Seed it in setup.
Data a scenario creates (a new customer, an order) Name it with {unique:name} (§16), so two runs and two parallel scenarios never collide.
Whatever the run left behind Sweep it up in teardown, for example everything matching the pattern your unique values use.

Setup and teardown run once per run, not once per scenario. A scenario that needs its own private state should create it in its own steps, a Background or a StepSet.