AI-Generated Code Quality: A 12-Point Review Checklist Before You Merge

AI-generated code has a specific failure mode: it looks finished. Names are sensible, formatting is clean, there's a docstring on every function. The problems are underneath, in a dependency that doesn't exist, an error that gets swallowed, a helper that already lived two folders over. Human-written bad code usually looks bad. This kind doesn't, and that is why reviewing it by feel doesn't work.

There's evidence that feel is a poor guide here. In METR's early-2025 study, 16 experienced open-source developers took 19% longer on real tasks when they were allowed AI tools, yet afterwards they believed the tools had made them about 20% faster. If people misjudge their own speed that much, they can misjudge a diff too. A written checklist is the boring fix.

Below is the one we use on pull requests where an agent wrote most of the code. It assumes you already work in a research, plan, implement loop; if not, start with our piece on context engineering and the RPI framework, because most bad AI code is a context problem before it is a review problem.

Before you read a single line

1. Is the diff reviewable? If it touches dozens of files for one feature, send it back to be split. Nobody reviews 1,500 lines carefully, and agents are happy to produce them.
2. Does it match the plan? Put the plan or ticket next to the diff. Anything in the code that isn't in the plan is either scope creep or a decision nobody made on purpose.
3. Did the tests actually run? In CI, on this commit. "All tests pass" in the agent's summary is a claim, not a result. Our RPI with Claude Code subagents write-up calls this checking the world, not the transcript.

Correctness: the plausible-but-wrong layer

4. Every new dependency is real and is the one you meant. In a USENIX Security 2025 study of 576,000 generated code samples, at least 5.2% of packages suggested by commercial models and 21.7% by open-source models did not exist, with 205,474 unique made-up names. An attacker who registers one of those names gets installed by whoever trusts the suggestion. Check the registry page, the maintainer and the download history before the lockfile changes.
5. Every API call exists in the version you pin. Models mix up major versions. A method that was renamed two releases ago will still look right in review and fail at runtime.
6. Edge cases are handled, not just the happy path. Empty input, null, a very large file, a timezone other than UTC, non-Latin text, a second request arriving while the first is still running.
7. Errors are not swallowed. Look for broad catch blocks that log and return a default, retries without a limit, and fallbacks that hide a failure the caller needed to know about.
8. Tests check behaviour. A test that only asserts a mock was called proves very little. Also diff the test files: if an existing assertion was loosened in the same PR, ask why.

Fit: does it belong in this codebase?

9. It doesn't duplicate something you already have. Search the repo for the new helper's job before accepting it. Agents rewrite date formatters and HTTP wrappers constantly, because they didn't see yours.
10. It follows your conventions. Layering, naming, logging, config handling. Code that works but ignores the house rules costs more every time someone touches it later.
11. No dead weight. Unused parameters, commented-out alternatives, comments that narrate the obvious, and defensive checks for things that cannot happen.

Security: the part you can't skip

12. New inputs are validated and new endpoints check permissions. Look for string-built SQL or HTML, secrets in code or logs, personal data written to logs, and any route that checks authentication but not authorisation. Generated code often gets the login check right and forgets to ask whether this user may see this record.

Making the checklist cheap enough to use

A checklist that lives in a wiki gets ignored by the third PR. Put it where the work happens:

• Add the twelve items to your pull request template as checkboxes.
• Write the conventions from items 9 to 11 into the instruction file your agent reads (CLAUDE.md, AGENTS.md or similar), so fewer violations reach review in the first place.
• Let tools do items 4, 5 and 11: a dependency allow-list, a type checker against pinned versions, and a linter for unused code.
• A second agent can pre-review against the list and flag items. A person still signs off, because the point of review is that someone accountable read it.

Context is the upstream fix

Items 9 and 10 fail mostly because the agent never saw your existing helpers or your house rules. The fix is to give it the relevant internal documentation, and to ask it to say which document a decision came from. That citation habit is the same one we built FanMind around for document questions: an answer that names its source is one you can check in seconds. For code, it turns "why did you do it this way?" into a link instead of a guess.

Frequently asked questions