How we test an AI code reviewer: prompts, reasoning effort, codemode and a security pass

In short:

• We keep 29 frozen cases for reviewer regression testing, with eight reserved for held-out checks. For the experiments below, we used the relevant cases and scored findings blind to the configuration.
• The clearest gains were in finishing reviews: stating each output field’s type and returning clearer validation errors took finished reviews from 13 to 16 of 16. Reading the READMEs and decision records behind a change rated more defects at the right severity, at 3.1 times the cost.
• A focused security pass caught trust-boundary defects the general review missed. It also added false findings and sometimes made the final verdict worse.
• On the runs we measured, xhigh reasoning effort caught nothing high missed, at two to three times the cost. Pi Durable’s codemode lowered the cost of a review by 22%.
• Small test cases can’t tell LLMs apart. Comparing models took real pull requests.

One good review proves nothing

Lightfactory reviews pull requests with an LLM agent that reads the diff and the repository around it. Its quality depends on many parts at once: the prompt, the model, the reasoning effort, and the agent runtime that lets the model read code. A change to any of them can make reviews better, worse, slower or more expensive, and a single convincing review shows none of that.

What we needed from a test

We wanted to know, before shipping a change, whether the reviewer would find more real defects, raise fewer false ones, finish its reviews and cost less. The answer had to be repeatable, so that two versions could be compared on the same inputs, and blind, so that whoever scored a review couldn’t favor the version they expected to win.

A frozen corpus, scored blind

We keep 29 cases. Each is a frozen before-and-after snapshot of a code change, with a label written before any run.

• 24 cases are synthetic, most with one planted defect, such as an empty input that now panics, a retry loop that runs once too often, or a required CI check that a cancelled job satisfies.
• 5 cases are snapshots from our own pull requests where a live review got something wrong: a defect rated too low, a false finding a passing test refuted, a defect fixed only later.
• 19 cases are labelled with defects, some with more than one, 8 are clean and 2 are expected to end incomplete. The labels are provisional until a person adjudicates them.
• 8 cases are held out. We don’t tune against them.

The reviewer never sees the labels. Each experiment ran the cases relevant to its question, and two or three LLM agents matched each review’s findings to the labels without knowing which configuration wrote them. From their verdicts we score defects found, false findings, whether the verdict was right, whether the review finished, and cost and latency.

What improved review quality

Prompt contracts beat clever prompts. Some reviews found the defect, then failed because a field came back as a list instead of a string. On a two-case probe of exactly those failures, one sentence naming each field’s type, together with a correction that named the field at fault, turned both into finished reviews. The full comparison also included a fix to the runtime’s event limit, so the 13 → 16 isn’t the prompt alone.

Documentation finds what the diff hides. A rule a change breaks often lives in a README the change never touches. Reading those documents rated more defects at the right severity. The same release also changed the reviewer’s system prompt and completion rules, and the cost figure leaves out reviews the provider failed. Without a cap, one review kept reading until its requests grew from 49 KiB to 594 KiB and it ran out of time.

A focused security pass catches more, and costs precision. We ran it on 11 real changes from our own repository, 6 with a labelled trust-boundary defect and 5 without, 20 paired runs in all. A second pass that looks only at credentials, isolation, data leaving the host and checks that can be skipped found defects the general review missed, among them:

• a setup script running as root that wrote into a service user’s home directory, so a planted symlink could redirect the write anywhere;
• an installer that skipped its checksum whenever the binary printed the expected version;
• an address check meant for a private range that accepted public addresses too.

At xhigh, the pass took detection from 2 to 6 of 16 labelled defect opportunities. It also raised false findings from 8 to 11, and right change-request verdicts fell from 19 to 17 of 20. At high, the gain was 2 to 3 of 16. On gpt-6.1-terra, the model we review with today, we reran seven selected cases: the pass took detection from 3 to 5 of 8, at an estimated USD 1.26 a round at list price, but right verdicts fell from 7 to 4 of 7, partly on severity labels and partly on one false High.

What only added cost

Reasoning effort. We compared xhigh and high on gpt-6-luna, on four cases from live reviews and one replay of a real review that had timed out. Both caught the same one labelled defect. high cost a half to a third as much, in a third to a half of the time, but raised a false High on a clean case. In a separate development run at high, the reviewer found 10 of 11 defects, at an estimated USD 0.0045 a review. We switched to high.

Many small tool calls. In 63 live reviews, 201 of 1,411 tool calls failed: 144 read past the end of a file and 42 asked for more than the 200 lines a read allows. Each failure cost a model turn.

Most of the saving came from the combined prompt and read changes, including a list of the repository’s documents, so the model stopped probing for files that don’t exist. The corpus has no oversized diffs, so it can’t show what sending whole diffs saves on its own.

Pi Durable’s codemode lowered the estimated charge by another 22%. The model writes one short JavaScript program, run in a sandbox with no file system or network, that calls the read and search tools many times and returns only what it needs. The first codemode run matched all 12 detections from the run without it. A repeat found 11, and both codemode runs left one review incomplete: a script asked for a file the case lacks, failed as a whole, and the review never finished. Catching that regression is what the corpus is for.

Changing the agent runtime. We moved from Pi’s RPC mode, which starts every review from nothing, to Pi Durable, which keeps one conversation per pull request across review rounds. In fresh conversations, Durable found 12 of 15 labels versus 11 for RPC, at roughly the same general-review charge. One uncertain call accounts for the difference. Because each case started a fresh conversation, the test says nothing about the benefit of carrying context across rounds.

Where a small corpus stops helping

We also asked whether stronger models pay off. Routing each case to a cheap, medium or strong model by its estimated risk found the same 12 of 15 defects as the cheap model alone, at about 3.5 and 6.1 times the estimated round charge. That baseline ran on a different build and host. The one extra finding from Claude Opus 5.5 was false.

That says more about the corpus than about the models. Its cases change a line or two, and on small changes most models find the same thing. On 22 real pull requests from ReviewBench, the model we were using found about half as many defects as the others, and we switched.

What an AI code review costs

On our small development cases, one general review costs about half a cent at list price. On real pull requests, the median general review cost:

The Claude figures are usage-based charges. The OpenAI figures are list-price estimates, so they are an upper bound. A Lightfactory round on a change to code runs the general review and the security pass. From terra’s median, we budget about USD 0.40 to 0.50 for such a round.

Limits

Most comparisons are one run per configuration over about 20 cases, and a difference of one defect is within run-to-run noise. Most cases are planted defects, smaller and cleaner than real ones. The labels are provisional, and LLM agents, not people, matched findings to them.