Codex and Flaky Tests: What Actually Helps

A flaky test is a problem for a human contributor too, but it's a genuinely different, sharper problem for an agentic session iterating against test results automatically — a human eventually recognizes "oh, that one's just flaky" and moves on; a session treating every failure as a real signal to fix can burn real turns chasing a ghost.

Why flakiness is worse for agentic iteration than for human review

A human running a test suite locally has accumulated context about which tests are historically unreliable, often without even consciously tracking it — a kind of institutional memory built from repeated exposure. A fresh Codex session has none of that memory by default; every failure looks equally real unless it's told otherwise, which means a genuinely flaky test can send a session down a path of "fixing" code that was never actually broken, potentially introducing a real regression while chasing a phantom one.

Telling Codex explicitly which tests are known-flaky

The single most effective, lowest-effort fix is simply stating which specific tests are known to be unreliable before a session starts iterating against the suite — a short list, named explicitly, treated with appropriate skepticism if they fail. This doesn't require fixing the flakiness itself first; it requires being honest about its existence rather than letting a session discover it the hard way, turn by turn, by chasing a failure that isn't real.

Distinguishing a flaky failure from a new regression

The genuinely hard case is a known-flaky test failing in a way that's plausibly connected to the actual change being made — is this the usual flakiness, or did this change genuinely make it worse or trigger a real, related issue? There's no shortcut that fully replaces judgment here, but running the specific flaky test several times in isolation, independent of the rest of the suite, is a reasonable way to get more signal before deciding whether to treat a given failure as noise or as real.

Why fixing the flakiness is usually worth prioritizing over working around it

Every workaround — telling Codex to ignore specific tests, retrying failed runs automatically, treating a subset of the suite as advisory rather than blocking — is a patch around a problem that's cheaper to actually fix once and stop patching around repeatedly. A test suite with persistent flakiness is a liability for every contributor, human or agentic, and an agentic session hitting it repeatedly is often the forcing function that finally makes the cost of not fixing it visible enough to prioritize.

Configuring retries thoughtfully, not reflexively

Automatic retry-on-failure for known-flaky tests is a reasonable mitigation while a genuine fix is pending, but it's worth configuring deliberately rather than applying broadly — a retry policy that silently re-runs and passes a test that's actually failing for a real, new reason (not just historical flakiness) can hide a genuine regression behind what looks like normal instability. Scoping retries specifically to tests with a documented history of flakiness, not applied suite-wide, keeps this mitigation from becoming its own source of false confidence.

What a session should do when it can't tell the difference

It's worth explicitly permitting a session to say "I can't confidently tell whether this failure is the known flakiness or a real regression from my change" rather than forcing it to guess one way or the other and move on silently. That uncertainty, surfaced honestly, is more useful to you than a confident wrong guess in either direction — and it's a natural extension of the broader principle that a session should ask rather than guess on things a wrong guess would be expensive to get wrong about.

Tracking flakiness over time rather than rediscovering it repeatedly

A list of known-flaky tests that only exists in one person's memory, restated fresh to each new Codex session, is fragile and easy to let go stale. Recording it somewhere durable — a note in AGENTS.md, a tag in the test suite itself — means every future session, not just the one where you happened to remember to mention it, benefits from the same institutional knowledge a human contributor would eventually accumulate on their own.

The broader shape of the lesson here

Flaky tests are a specific instance of a more general pattern: an agentic session is only as well-calibrated about a codebase's quirks as what it's actually been told, and quirks that live purely in human memory don't transfer automatically. Writing down what a new contributor — human or agentic — would otherwise have to discover the hard way is, across every page this site covers on agentic workflows, close to the single highest-leverage habit available.

The cost angle worth remembering here too

A session chasing a flaky failure isn't just wasting your time waiting on it — every iteration re-reads the accumulated context, the code, and the growing history of failed attempts, which adds up in tokens the same way any other unproductive loop does. A flaky-test list maintained explicitly isn't just a correctness fix; on a codebase with genuinely persistent flakiness, it's a direct, ongoing cost saving every time it prevents a session from chasing a failure that was never real in the first place.

A final note on trust

Once a session has burned real turns chasing a known-flaky test because nobody told it, it's worth treating that as a signal about your own process, not the model's competence — the gap was in what context got handed over, not in the model's ability to reason about test failures generally. Fixing the gap once, by documenting the known flakiness somewhere durable, benefits every future session working in that codebase, agentic or human.