When to Let Codex Run Tests Itself

Letting Codex run your test suite itself, inside a session, is a genuine productivity gain when it works — and a genuine trust question the first few times you try it, because it means an agent is now the one telling you whether its own change succeeded.

The case for letting it run tests directly

A session that can run tests itself closes a loop that would otherwise require you: make a change, run the suite by hand, report the failure back, wait for the next attempt. Letting Codex run the suite itself and iterate against real failures — not its own guess about whether something works — turns several rounds of manual back-and-forth into one supervised pass, and it tends to produce a more genuinely working result than a change that was never actually executed before you reviewed it.

What sandbox mode this actually requires

Running tests means running shell commands, which needs workspace-write at minimum — read-only won't execute anything. If the suite needs network access — hitting a real database, calling an external service, downloading a dependency — remember that workspace-write turns network off by default; a test run that mysteriously fails to reach a resource it needs is often this default, not a bug in the test or the code. If the suite is well-contained and doesn't need approval prompts interrupting a long test run, the "edit freely, gate untrusted commands" approval preset is usually the right middle ground — known-safe commands like your test runner execute without a stop, genuinely unfamiliar ones still get a prompt.

Trusting the result versus trusting the summary

A model reporting "tests pass" after running them is a genuinely stronger signal than a model asserting a change should work without running anything — but it's still worth spot-checking the actual test output occasionally, not just the summary, especially early in working with a given codebase's test suite. A flaky test, a suite that silently skips something under certain conditions, or a test that passes for the wrong reason are all things a summary can paper over that the raw output usually won't.

When self-run tests are the wrong call

Destructive test suites — ones that modify real external state, hit production-adjacent services, or aren't safely idempotent to rerun — are a bad fit for an agent iterating against failures automatically, because "iterate and retry" assumes each run is a clean, side-effect-free attempt. If your suite isn't safely rerunnable, that's worth fixing before handing test execution to an agentic loop, not a reason to avoid agentic coding generally — it's a reason to fix a test suite that was already a liability for human contributors too.

Flaky tests change the calculus

A suite with known-flaky tests is a genuinely harder case: a model iterating against failures can chase a flaky failure as if it were a real regression, burning turns (and tokens) on a problem that doesn't exist in the code it's actually changing. If you know specific tests are unreliable, it's worth telling Codex explicitly which ones to treat with skepticism, or fixing the flakiness itself before leaning on self-run tests heavily — a flaky suite is a bad foundation for an agent's own judgment about whether a change worked, same as it's a bad foundation for a human's.

Long test suites and the context they consume

A large suite's full output, read back into context turn after turn as Codex iterates, adds up — especially a verbose suite that isn't summarizing its own failures cleanly. For anything approaching a real context budget concern, a test runner configured to report failures concisely rather than dumping full verbose output on every run is worth the setup time, both for the model's context budget and for how much of a long session's accumulated input a slow, chatty test runner ends up contributing.

A reasonable default

Start by letting Codex run a scoped subset of the suite relevant to what it's actually changing, rather than the full suite on every iteration — faster feedback per turn, and a full run at the end once the scoped tests are passing. That's a smaller ask of both your test infrastructure and your context budget than a full suite on every single turn, and it still gets you the core benefit: a change that's actually been executed, not just written.

A test that fails because the sandbox blocked something it needed — a network call under workspace-write with networking off, a write outside the workspace boundary — looks, in raw output, a lot like a test that failed because the code is actually wrong. Learning to recognize the sandbox-restriction shape of a failure (a permission or connection error rather than an assertion failure) saves real time that would otherwise go into debugging code that was never actually broken, and it's worth explicitly ruling out before assuming a failing test points at a real bug.

Letting Codex fix its own test failures, within limits

Once a session is running tests and seeing real failures, letting it iterate — read the failure, adjust the code, rerun — is usually the point of the whole setup. Where it's worth stepping in rather than letting the loop continue unattended: if the same test keeps failing after several genuinely different attempted fixes, that's often a sign the model has misunderstood what the test is actually checking, and another attempt from the same starting understanding is unlikely to succeed where the last few didn't. That's the moment to look at the failure yourself and either correct the model's understanding directly or decide the test itself needs revisiting.

What "letting it run tests" doesn't mean

None of this is an argument for skipping your own review of the final diff because "the tests passed." A passing test suite tells you the change didn't break what the suite covers — it says nothing about what the suite doesn't cover, and it says nothing about whether the change is the right change, only that it's an internally consistent one. Self-run tests raise your confidence in a change; they don't replace the judgment call about whether it's the right change to have made.