← All guides

Flaky tests in GitHub Actions: how to detect them and stop paying for them

A flaky test doesn't announce itself. It shows up as someone clicking "re-run failed jobs" on a PR without a second thought, a scheduled workflow whose failure rate has crept from 2% to 15% over a month with nobody watching the trend, or a retry wrapper someone added a year ago that nobody remembers is there. Each rerun re-bills the same minutes, and the test that caused it stays exactly as broken as it was.

Detect it: rerun counts and failure rate, not vibes

"I think that test is flaky" is a hunch until you have numbers. Pull rerun attempts per workflow:

gh run list --workflow=ci.yml --json conclusion,attempt \
  --jq '.[] | select(.attempt > 1)'

Then look at failure rate over a rolling window (30 days is enough to separate a genuine trend from a bad Tuesday). A workflow trending steadily redder without anyone noticing is the same failure mode as an always-failing workflow — just earlier in its life.

Retries are a diagnostic, not a cure

A retry wrapper — nick-fields/retry around a step, or a test runner's own --retries (Playwright) / --reruns (pytest-rerunfailures) — makes red go green without fixing anything. That's fine as a stopgap while you triage, and it's fine permanently for a genuinely nondeterministic case: vite's CI sets VITEST_SEGFAULT_RETRY: 3 with a one-line comment explaining exactly why (a known flaky segfault class), scoped narrowly instead of wrapping the whole suite. That's the difference between a retry and a cover-up: it's documented, scoped, and someone can point at the reason.

Fix patterns, in the order I'd check them

• Test isolation — tests that pass alone but fail in a full run usually share state: a global, a singleton, an order dependency nobody enforced.
• Missing awaits / races — a test that asserts before an async operation settles passes most of the time, which is worse than always failing.
• Shared fixtures — parallel workers colliding on the same port, temp file, or database row is a classic CI-only flake that never reproduces locally.
• Fixed sleeps instead of real waits — a hardcoded sleep(1000) racing a slower CI runner; wait on the actual condition, and set a job timeout as the backstop for the case that still hangs.

Honest nuance: retries aren't the sin, silence is

next.js runs a dedicated workflow that reruns its canary build-and-test up to three attempts through the GitHub API, then posts to Slack once retries are exhausted. That's a defensible retry: bounded, automated, and someone finds out when it stops working. The failure mode isn't retrying at all — it's a retry nobody bounded, nobody tracks, and nobody reviews, quietly doubling your minutes while the underlying test never gets fixed.

Common questions

Are retries always a bad sign? No. A narrow, documented retry around a known nondeterministic case — a timing-sensitive segfault class, a flaky third-party network call — is a reasonable permanent fixture. The problem is a blanket retry around an entire suite with no comment and no ceiling, which hides which tests are actually flaky.

How do I find which tests are flaky in my repo? Check rerun/attempt counts per workflow via gh run list or the Actions API, and track failure rate over a 30-day window. A workflow that needs a rerun every few runs, or one whose failure rate is climbing without anyone noticing, is the tell.

What's the difference between nick-fields/retry and a test runner's own retry flag? nick-fields/retry re-runs an entire step or job's command from the outside with no idea which test inside it failed. A test runner's native --retries / --reruns retries at the individual-test level and reports which tests needed it — prefer the runner-native option when it's available; it gives you the list you'll need to fix them.