CI and review

Review CI heals

A heal in CI files its proposal in a workspace that is thrown away when the job ends. Reproducing it locally would take a second agent session, which costs again and may heal differently. Instead, CI packs what the run left: saffron export writes one zip with the pending proposals, the run's report, the screenshots and traces the report names, and a manifest of where it all came from (repository, branch, commit, CI run and attempt, shard). CI uploads it as an artifact. On your checkout, saffron import checks it and puts everything where a local run would have, so saffron diff, saffron accept, saffron trace, the report and both IDE panels work on it as if the run had been yours.

A heal made in CI, reviewed in the IDE, from the terminal or in a pull request

There are three ways to review it. All three end the same way: saffron accept on a checkout, and the new recordings committed.

Where	How	What it needs
In your IDE	The VS Code CI runs view or the JetBrains CI Runs tab: Import on a run (§20)	The bundle upload below, and `gh` or `az` signed in on your machine
From the terminal	`saffron runs`, then `saffron import --run <id>` (below)	The same
In a pull request	CI pushes a branch with the proposals and opens a draft pull request; you accept on that branch (§21)	`export --branch`, and a CI token that may push and open pull requests

The IDE and the terminal bring the run's bundle into your own checkout: its proposals, report, screenshots and traces. The pull request carries the proposal files only, on a branch of their own. A pipeline can do both, so the same run can be reviewed either way.

In CI

GitHub Actions:

- run: npx saffron run --strict
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # the agent's key
- name: Pack the heals for review
  if: failure()
  run: npx saffron export --out saffron-bundle.zip
- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: saffron-bundle
    path: saffron-bundle.zip
    retention-days: 14
    overwrite: true      # a re-run replaces the bundle instead of failing the upload

Azure Pipelines:

- script: npx saffron run --strict
  displayName: Run the suite
  env:
    ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)   # a secret variable reaches a script only when mapped
- script: npx saffron export --out $(Build.ArtifactStagingDirectory)/saffron-bundle.zip
  displayName: Pack the heals for review
  condition: failed()
- task: PublishPipelineArtifact@1
  condition: failed()
  inputs:
    targetPath: $(Build.ArtifactStagingDirectory)/saffron-bundle.zip
    artifact: saffron-bundle-$(System.JobAttempt)

A CI job heals only with the agent's key or login: Claude, the default, takes ANTHROPIC_API_KEY (a secret in the repository or pipeline), and Cursor a CURSOR_API_KEY; Codex and Antigravity sign in only with a login kept on the machine, so they heal only on a build agent of your own that has one (AI providers and sign-in). Without one, nothing heals and the run says what is missing.

Under --strict a pending proposal fails the job, so exporting on failure catches every heal, and a red run's screenshots and traces travel with it. Export on every run (if: always(), condition: always()) to keep green runs' traces too ("trace": "on"). Give --out a path outside .saffron/: upload-artifact leaves hidden folders out. Without --out the bundle is written to .saffron/reports/saffron-bundle.zip. How long the artifact is kept is the CI's setting (retention-days above).

Re-runs. A re-run of the job exports and uploads again, so the two attempts must not collide. On GitHub, overwrite: true lets the new bundle replace the earlier attempt's; without it the upload fails on the name. Azure Pipelines keeps every attempt's artifacts and refuses a name the run already has, so there the name carries the attempt (System.JobAttempt); saffron import --run finds every saffron-bundle* artifact. Of a sharded run's bundles, import keeps the latest attempt of each shard, per browser and base URL, and says which it left out: another browser's or environment's attempt replaces nothing, and the two are refused together by name. Bundles of two attempts of an unsharded Azure run are refused together: download the latest (az pipelines runs artifact download --run-id <id> --artifact-name saffron-bundle-2 --path bundle) and import that folder.

Azure Pipelines support was tested against a stand-in az, not yet in a real Azure DevOps organization: please report problems.

A bundle holds what import takes: export refuses a file over 1 GB, and when everything together would be over 4 GB (the most one zip holds) it leaves the traces of green scenarios out first and says so. Still too much is refused, naming the size: keep fewer traces ("trace": "retain-on-failure") or shard the run.

On your machine

npx saffron runs                 # this branch's CI runs, with their ids
npx saffron import --run <id>    # that run's bundle, downloaded and imported
npx saffron diff
npx saffron accept --all --include-imported   # or accept each by file
git add .saffron/cache && git commit -m "Accept the CI heal" && git push

These run the project's own saffron-ai, so install first (npm ci). In a checkout without node_modules, a bare npx saffron fetches an unrelated npm package named saffron: run npx -y -p saffron-ai saffron ... there instead.

saffron runs and saffron import --run reach the CI through its own command-line tool, which keeps the credentials: the GitHub CLI (gh, signed in with gh auth login) for GitHub Actions, and the Azure CLI (az) with its azure-devops extension (az extension add --name azure-devops, signed in with az login or AZURE_DEVOPS_EXT_PAT) for Azure Pipelines. The repository's host decides which: an Azure Repos remote means Azure Pipelines, anything else GitHub Actions. A GitHub repository built by Azure Pipelines says so in the config (§14):

{ "ci": { "provider": "azure", "organization": "https://dev.azure.com/acme", "project": "Shop" } }

saffron runs lists the checkout's branch (--branch for another, every branch on a detached HEAD); an Azure pull request build is listed under the branch it validates, and a tag build under its tag (--branch v1.4.0 on either service). Each run says how many of its proposals are pending here. import --run downloads the run's saffron-bundle* artifacts into a folder under .saffron/reports/, imports them (every shard of a sharded run, merged) and removes the folder; one left by an import that was stopped (an editor's Cancel) goes at a later import, once it is an hour old. A run without a bundle says so. Before downloading, import --run asks gh or az who started the run. A run a fork's pull request started (on GitHub, one whose head repository is not this one; on Azure Pipelines, one whose trigger says pr.isFork) is refused unless you pass --allow-fork: the fork's code made that bundle, and could have put anything in it. A downloaded bundle that names another run is refused too. accept --all leaves imported proposals out, so a bulk accept (the IDEs' Accept All included) never takes a CI heal unread: accept each by file after saffron diff, or pass --include-imported. With --json, both print what the editors read. For import every failure (a refusal, an invalid config, a full disk) is {"error": "..."} and exit 2. For runs, a missing, signed-out or failing gh or az is problem (kind and message) beside an empty runs list, and error with the same message, so a client reading either field works; any other failure is {"error": "..."} alone; both exit 2. On Azure Pipelines only this repository's builds of the branch are listed, its pull request builds included: az is asked for the newest 50 builds first, and for more, up to 500, only while too few of them are the branch's. A download (--run, or a URL) that takes over 10 minutes is stopped and says so. Text from a bundle (branch, commit, names) is printed without control characters; a manifest field over 2048 characters (a repository, branch or link) is refused as damaged, never cut short.

You can also download the artifact yourself (gh run download <run-id> -n saffron-bundle, or az pipelines runs artifact download --run-id <id> --artifact-name saffron-bundle-1 --path ., the attempt's number last) and import the file: saffron import takes a bundle file, a folder (searched for saffron-bundle*.zip, the way a CI download lays out several artifacts), or an http(s) URL. After accept, the new recordings are an ordinary change in your working tree: commit and push them with your own git tools. Nothing is pushed for you.

In VS Code and JetBrains IDEs the same takes no terminal: the CI runs view and the CI Runs tab list these runs and import one with a click (§20).

View a run without importing it

Import changes your checkout: it files the run's proposals, copies its screenshots and traces into .saffron/artifacts/, and its report becomes your latest. To see first what a run found, view it:

npx saffron report --run <id>    # the run's report page, in your browser
npx saffron trace --run <id> "Successful login"   # replay one of its traced scenarios
npx saffron report --run <id> --panel > results.html   # the results page the editors show

	View (`report --run`, `trace --run`)	Import (`import --run`)
Downloads the run's bundles	Once, into `.saffron/ci/`	Reuses that download when there is one
Checks the bundles	Every check import makes (the table below)	The same
Proposals	Judged against this checkout: importable, or not and why. Nothing filed	Filed under `.saffron/proposals/`
Report, screenshots, traces	Read from `.saffron/ci/`; your latest report stays yours	Become your latest run
Replay	View only: no Accept or Reject	Accept and Reject in the page

Viewing never writes .saffron/reports/, .saffron/proposals/, .saffron/artifacts/ or the import ledger, and nothing it writes shows in git status. Import stays the only way to file a run's proposals, and viewing a run then importing it downloads the run once.

The .saffron/ci/ cache. Each viewed run is kept in .saffron/ci/<provider>-<id>/attempt-<n>/: the downloaded bundles, the screenshots and traces taken from them, and the report page. A finished attempt's artifacts never change, so the folder is reused, its files checked against their digests each time. A download goes into a .partial folder first and is renamed when it completes, so a cancelled one leaves nothing half written. The cache keeps the newest 10 runs and at most 1 GB, and removes the oldest runs first after each download; these limits are fixed. Creating the folder writes .saffron/ci/.gitignore (*), so git ignores it even in a project set up before it existed. Delete the folder any time.

Large runs. --max-download <MB> refuses a run whose bundles add up to more than that, saying the size; the editors ask before going over their limit (§20). Without it there is no limit.

A run without results says why, and exits 0:

Reason	Means
`running`	The run has not finished. Its bundle is uploaded when it does
`expired`	The bundle was uploaded but the service has deleted it (its retention, `retention-days` above). GitHub says so; Azure Pipelines does not tell an expired artifact apart, so there it reads `none`
`none`	The run uploaded no bundle. The workflow above exports only when a run fails: put `if: always()` (`condition: always()` on Azure) on the export and upload steps to keep every run's results, green ones included

A summary of every run (--report-only). A full bundle carries traces, which is why the workflow above uploads one only when a run fails. A summary is the run's report and screenshots, no proposals and no traces: small enough to upload on every run, so any run, green ones included, can be looked at.

- name: Summarize the run
  if: always()
  run: npx saffron export --report-only --out saffron-summary.zip
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: saffron-summary
    path: saffron-summary.zip
    retention-days: 14
    overwrite: true

Name its artifact saffron-summary* (on Azure Pipelines, add the attempt as for the bundle). In a sharded matrix, give each shard's summary its own name, as its bundle has (name: saffron-summary-${{ matrix.shard }}): with one fixed name and overwrite: true, each shard replaces the others' summary. Viewing a run uses, for each shard and attempt, its full bundle when it uploaded one and its summary otherwise: a shard that passed adds its results beside a failed shard's bundle, and a re-run's summary stands for the attempt it ran. A summary from another browser or base URL is never left out beside a full bundle: importing the two together is refused by name, so one environment's failures cannot hide behind another's pass. A summary has nothing to replay and nothing to import: importing a run that left only summaries says so, and beside full bundles the summaries only add their shards' results. Without --out it is written to .saffron/reports/saffron-summary.zip.

What each run left, in the list. On GitHub, saffron runs (and both editors' lists) mark each run with what it left to view: its results and their size, a summary only, results expired, or none uploaded. One listing of the repository's artifacts answers for the whole list, so it stays quick. Azure Pipelines has no such listing; there a run says when it is opened.

What import checks, before it writes anything:

Check	When it fails
The bundle comes from this repository: one of the checkout's remotes has its path, whether it is spelled over HTTPS or SSH (a fork counts when its upstream remote is)	Refused, exit 2, nothing written
That remote is on the bundle's host too	A warning for the same path under another host name (an SSH host alias such as `github-work`, or a mirror); the bindings decide. An Azure DevOps SSH path (`v3/<org>/<project>/<repo>`) is Azure DevOps through any host alias
The bundle says which repository it comes from	Refused, exit 2, nothing written; with `--allow-unknown-repository` a warning, and the bindings decide
The checkout has a git remote to compare with	A warning; the bindings decide
Its format is one this runner reads	Refused; a newer format says to update `saffron-ai`
Every file matches the digest in its manifest, every path stays inside `.saffron/`, no file appears twice, and the report is complete enough to show: each step's status is one a run writes, and its retries and every count in its trends are numbers	Refused as damaged, nothing written
None of its files is over 1 GB, and none unzips to over 100 times its stored size (judged by the sizes the zip declares, before anything is unzipped)	Refused as damaged, nothing written
The bundles of one import unzip to at most 8 GB together, every shard of a run included (judged by the sizes their zips declare, before any file in them is read; each file is then read from the bundle on disk one at a time, and a URL's bundle is downloaded to disk first)	Refused, nothing written
Each proposal's binding: the scenario as written here and the committed recording here are the ones it was made against (§5)	That proposal is not imported, with the reason; the rest are, and the command exits 1
No different proposal is pending here for the same scenario	Yours is kept; `--overwrite` replaces it
With `--run`: no fork's pull request started the run	Refused before anything is downloaded; `--allow-fork` imports it
With `--run`: every bundle downloaded was made by that run	Refused, nothing written
No two shards of the run filed different proposals for one scenario	Neither is imported, with the reason (a conflict between shards); the rest are, and the command exits 1
The checkout is at the commit the bundle was made at	A warning saying how far apart they are; the bindings decide

On a pull request build the bundle records both the merge commit CI tested and the head of the pull request's branch, the one you check out, when that merge commit really merged that head in. A pull_request_target build tested the base branch: its bundle records that branch and the commit CI checked out, never the fork's branch or commit. Import compares your checkout with the head, and says so when the run tested a merge commit: the proposals' bindings are what decide. Importing the same bundle again changes nothing and says so; after you accept, it says the proposal is already accepted here. saffron accept checks the bindings again, because the branch can move between import and accept.

The imported run becomes your latest run: .saffron/reports/latest.json and latest.html show it, marked "imported from run ..." with a link to the CI run, its screenshots and traces are under .saffron/artifacts/, and saffron trace replays it. Your next local run replaces it, as any run does. Which run each proposal came from is noted beside it (in .saffron/reports/imported.json), so saffron status --json gives each pending proposal its importedFrom (repository, branch, commit, run and link, shard) for as long as it is the proposal that was imported. --json prints the outcome per proposal for tools and editor plugins. Until you accept or reject it, a local run whose committed recording for that scenario is missing or fails reports the scenario yellow, pending review, naming that run: it does not replay a proposal it did not file (§5).

Sharded runs. Each shard exports its own bundle; name the artifacts apart (name: saffron-bundle-${{ matrix.shard }}), and its summary the same way (name: saffron-summary-${{ matrix.shard }}), or the shards overwrite one another's. saffron import --run downloads all of them; by hand, download them into one folder and import the folder:

gh run download <run-id> -p 'saffron-bundle*' -D bundles
npx saffron import bundles

The shards' reports merge into one, the way saffron report --merge merges them (§15). Bundles of different runs are refused together. When a run was re-run, a shard (or a whole unsharded run) exported by more than one attempt is taken from the latest attempt, and the import says which it left out. When CI exports only the shards that failed, the ones there are import, and the report says which shards it holds.

A bundle holds the same files as .saffron/reports/ and .saffron/artifacts/ on the CI machine: {env:VAR} values are masked there already (§18), and screenshots are pixels. Keep the artifact as private as the repository. A remote URL loses any credentials in it before it is written to the manifest.