This is the Health Check report. The Quick Scan report is shorter: an indicative score, the top 5 risks and quick wins, based on your CI configuration and reports, without file-level evidence from source code.
1. Summary
In plain words. The test suite is well organised and covers the main customer flows. The biggest problem is that failures are hidden: failed tests are re-run automatically and the report shows only the final green result. Builds look healthier than they are, and people stop trusting red builds.
What to do first. Make retries visible and save screenshots and traces on failure (about 1.5 hours). Then give each parallel worker its own test user and split the pipeline into parallel jobs. Together this should bring CI from 38 to about 15 minutes and remove most false alarms.
What was reviewed
| Item | Details |
|---|---|
| Repository | northwind-e2e (fictional) |
| Tests | 412 in total: 318 UI and 94 API |
| Stack | Playwright, TypeScript |
| CI | GitHub Actions, one workflow (e2e.yml), 38 minutes per run |
| Materials | Repository archive and reports from the 30 most recent CI runs |
| Not checked | The tests were not executed. Production code, security and performance were out of scope. |
2. Health score: 72 / 100
Band: working with notable risks. The scale is 0–39 fragile foundations, 40–59 needs focused improvement, 60–79 working with notable risks, 80–100 strong foundations. Each category below is scored from 0 to 100 and weighted by importance.
| Category | Weight | Score | In plain words |
|---|---|---|---|
| Architecture | 15% | 78 | Test layers are clear and shared helpers are easy to find. |
| Test reliability | 20% | 62 | Intermittent failures are common and hidden by retries (F1, F2, F3). |
| Maintainability | 15% | 86 | Readable names and small page objects. The strongest area. |
| Coverage strategy | 15% | 76 | Login, checkout and invoices are covered. Refunds are not (F6). |
| CI/CD | 15% | 70 | Works, but slow and running one test at a time (F4). |
| Reporting and observability | 10% | 66 | Failures leave no screenshots or traces (F5). |
| Test data and environments | 10% | 66 | One shared account is used by many tests (F3). |
| Overall | 100% | 72 | Working with notable risks |
How it adds up: score × weight, summed. 78×15% + 62×20% + 86×15% + 76×15% + 70×15% + 66×10% + 66×10% ≈ 72.
3. How the test suite and pipeline are built
Test suite
The structure is sound, which is why architecture and maintainability score well.
tests/ui/holds 318 browser tests andtests/api/holds 94 API tests, so the two layers are clearly separated.pages/contains small page objects, one per screen, with no test logic inside.tests/fixtures/holds shared setup such as login. This is where the shared-account problem (F3) comes from.playwright.config.tsis the single place for settings, but several CI-specific values are missing (F1, F4, F5).
CI pipeline
One GitHub Actions workflow runs all 412 tests in a single job and takes 38 minutes. It uses one worker, re-runs failures twice and does not keep screenshots or traces (F1, F4, F5).
Flaky-test risk
Confirmed: nine tests passed only on retry in the last 30 runs, and four of them collide on the shared account (F3). Not verified: the other five. They need to be reproduced locally with traces turned on.
4. Findings
Six findings, ordered by priority. Each one shows what we saw, why it matters, what to do, how long it takes and what you get.
| ID | Severity | Finding | Effort |
|---|---|---|---|
| F1 | HIGH | Failed tests are re-run automatically, so problems stay hidden | About 1 hour for steps 1–2, then 2–3 days for step 3 |
| F2 | MEDIUM | Fixed waits slow tests down and still fail when the app is slow | 1–2 days |
| F3 | MEDIUM | All tests log in with the same account | About 1 day |
| F4 | MEDIUM | The pipeline runs everything in one 38-minute job | About 1 day |
| F5 | LOW | Failures leave no screenshots or traces | About 30 minutes |
| F6 | MEDIUM | Refunds and plan downgrades have no automated checks | About 2 days |
Failed tests are re-run automatically, so problems stay hidden
- What we found
playwright.config.ts:14setsretries: 2for every test, and.github/workflows/e2e.yml:41publishes only the final result. In 23 of the 30 recent runs, at least one test failed first and passed on retry. Nine tests cause 81% of these retries.- Why it matters
- A real bug that appears only sometimes will turn green and ship. The team also cannot tell which tests are unreliable, so builds get re-run out of habit and failures are no longer trusted.
- What to do
- Show tests that passed on retry as “flaky” in the CI summary.
- Lower
retriesto 1. - Fix the nine repeat offenders, worst first.
- Effort
- About 1 hour for steps 1–2, then 2–3 days for step 3
- Result
- Flaky tests become visible and countable, and the retry rate becomes a number you can bring down.
Fixed waits slow tests down and still fail when the app is slow
- What we found
- 74 hard-coded pauses (
waitForTimeout) in 31 files. For example,tests/checkout/payment.spec.ts:57always waits 3 seconds. Together they add about 3 minutes of pure waiting to every run. - Why it matters
- If the app takes 3.5 seconds, the test fails. If it takes 0.5 seconds, the test still waits 3. It is slow and unreliable at the same time.
- What to do
- Replace each pause with a wait for the real condition, for example
await expect(page.getByText('Payment received')).toBeVisible(). Start with the 10 files that have the most pauses. - Effort
- 1–2 days
- Result
- Faster runs and fewer timing-related failures.
All tests log in with the same account
- What we found
tests/fixtures/auth.ts:12signs every test in as[email protected], and 41 tests create invoices under it. Four of the nine flaky tests fail only when two of them run at the same time.- Why it matters
- Tests interfere with each other when run in parallel. A single test re-run on its own gives a different result than in the full run, which makes debugging slow.
- What to do
- Create a fresh user per parallel worker through the API and delete its data at the end of the run.
- Effort
- About 1 day
- Result
- Removes the collisions behind about 4 of the 9 flaky tests and makes parallel runs safe, which F4 needs.
The pipeline runs everything in one 38-minute job
- What we found
e2e.ymlhas a single job.playwright.config.ts:19setsworkers: 1in CI, and browsers are downloaded again on every run (about 4 minutes).- Why it matters
- Developers wait 38 minutes to learn whether a change is safe, so they batch changes or stop waiting for the result.
- What to do
- Split the tests into 4 parallel jobs with
--shard, cache the Playwright browsers, and raiseworkersto 2 once F3 is fixed. - Effort
- About 1 day
- Result
- CI time drops from about 38 to about 15 minutes.
Failures leave no screenshots or traces
- What we found
playwright.config.ts:22–23setstrace: 'off'andscreenshot: 'off', and the workflow does not upload the report folder.- Why it matters
- To understand a failure, an engineer has to reproduce it locally, which often takes longer than fixing it. Low severity, but a very cheap fix.
- What to do
- Set
trace: 'on-first-retry'andscreenshot: 'only-on-failure', and uploadplaywright-report/as a CI artifact kept for 14 days. - Effort
- About 30 minutes
- Result
- Every failure can be opened and understood in minutes.
Refunds and plan downgrades have no automated checks
- What we found
- The 412 tests cover login, checkout and invoices, but none exercise the refund or plan-change features. Both move money.
- Why it matters
- Mistakes here cost money and customer trust, and manual testing before each release is easy to skip.
- What to do
- Add API tests first, because they are cheaper and more stable than UI tests: full refund, partial refund and downgrade at renewal. Add one UI check per flow later.
- Effort
- About 2 days
- Result
- The highest-risk flows are protected on every release.
5. Quick wins
Low-risk actions your team can finish in one afternoon.
| Action | Time | Benefit |
|---|---|---|
| Save screenshots and traces on failure and upload the report (F5) | 30 min | Every failure can be diagnosed without re-running locally. |
| Show tests that passed on retry in the CI summary (F1) | 1 hour | Hidden flakiness becomes a visible list. |
| Cache Playwright browsers in CI (F4) | 20 min | Saves about 4 minutes per run. |
| Remove the 10 longest fixed waits (F2) | 2 hours | About 1 minute faster and fewer timing failures. |
6. 30-day plan
What to do and when, so the work fits around normal development.
| When | What | Result |
|---|---|---|
| Now (week 1) | The four quick wins, then lower retries to 1 (F1, F5, F4, F2) | Failures are visible and easy to diagnose. |
| Next (weeks 2–3) | Per-worker test users (F3), 4 parallel jobs (F4), start fixing the nine flaky tests (F1) | CI about 15 minutes and far fewer false failures. |
| Later (week 4) | Replace remaining fixed waits (F2), add refund and downgrade API tests (F6) | A faster, steadier suite with money flows protected. |
What to expect after 30 days
| Measure | Before | Target |
|---|---|---|
| CI run time | 38 minutes | about 15 minutes |
| Runs with a hidden retry | 23 of 30 | fewer than 5 of 30 |
| Failures with a screenshot or trace | 0% | 100% |
These are targets for the fictional example. Real reports give estimates based on your materials, not guarantees.
What happens next
Your team decides what to act on. Fixes are not part of the Health Check. If you want help, the Audit + Fix Sprint covers up to 10 hours of fixes delivered as pull requests: request a proposal.
How to read the score
The weighted score is an internal framework, not an industry standard, certification or guarantee of software quality. In a real report, every finding points to evidence in your materials.