Recording-quality benchmark
Measured by node scripts/benchmark.mjs on the example suite (examples/)
and its demo app. Rerun it after a change to recording or replay.
saffron-ai 1.3.3, 2026-11-05.
| Measure | Result |
|---|---|
| Stability: replays green on the untouched app | 66 of 66 (100%), 22 scenarios × 3 runs |
| Flaky scenarios (green in some runs only) | none |
| False greens on deliberately broken apps | 0 of 8 (0%) |
| Recording quality (first attempt, held, time, cost) | not measured in this run: --record records the suite with the agent, at AI cost |
Broken apps, each a change that breaks the scenario it targets:
| Mutation | Scenario | Replay |
|---|---|---|
| a renamed heading | login.feature:Successful login | red, as it should |
| another error text | login.feature:Failed login shows an error (example 1) | red, as it should |
| a hidden error text | login.feature:Failed login shows an error (example 1) | red, as it should |
| the order API answers 200 | orders.feature:Order placement waits on the API | red, as it should |
| the job never reports READY | orders.feature:Background job polls until ready | red, as it should |
| a product missing from the list | login-data.saffron:The catalog lists every product in the data folder | red, as it should |
| a product name grown longer | login-data.saffron:The catalog lists every product in the data folder | red, as it should |
| a calendar file a day late | booking.feature | red, as it should |