Six LLMs on 22 ReviewBench pull requests: why we chose gpt-6.1-terra

In short:

• On 22 pull requests from the ReviewBench benchmark, with 113 known defects, gpt-6-luna, the model we were reviewing with, found 22. The other five models found 35 to 43.
• gpt-6.1-terra had the highest observed count, 43, and the fastest median review. The test did not clearly separate the five models ahead of luna.
• The gap was smallest below 300 changed lines and larger on bigger pull requests.
• Claude Haiku 5.5 added more labelled defects next to terra than a second terra run did. We repeated terra once and ran Haiku once, so we’d want more pull requests before adding a second reviewer.
• We made terra the default for our production reviewer, including its security pass.

Every model looked the same on our own tests

We test every change to Lightfactory’s AI code reviewer on a frozen corpus of 29 cases. When we ran stronger models on it, they found the same defects as the cheap one. The corpus’s cases change a line or two, though, and real pull requests don’t.

The question: which model finds the most real defects for its cost

We wanted to know whether a stronger LLM finds more defects in real pull requests than gpt-6-luna, our model at the time, whether the gap grows with the size of the change, and what each model costs per review. That needed real pull requests of real size, with an answer key we hadn’t written ourselves.

Six models on 22 ReviewBench pull requests

ReviewBench collects public pull requests with golden findings: the defects each one introduced, found partly by the projects’ reviewers and partly by frontier models, and audited by people. Our reviewer couldn’t prepare 3 of its 25 test pull requests: two large changes, and a 76-line change in a 613 MB repository. The remaining 22 hold 113 distinct defects of medium severity or worse.

Our harness attempted one general review per model and pull request, with the same prompt, at high reasoning effort, on the same build: OpenAI’s gpt-6-luna, gpt-6.1-terra and gpt-6.1-sol, and Anthropic’s Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5.

This is our own comparison on a subset, not a ReviewBench score. ReviewBench reserves official leaderboard scores for its full corpus, and we grouped duplicate findings and scored with our own rules.

How we scored the reviews

Blind matching. A finding rarely uses the answer key’s words. Two LLM agents matched every finding to a golden finding, or to none, from packets that named no model. They disagreed on 5 of 394 calls.

Count defects, not findings. In a three-PR pilot, one defect was repeated in six workflow files, so the answer key listed it six times. luna filed one finding per file and got the best raw score, then the worst once each defect counted once. For the full run, an agent grouped such repeats from the answer key alone, before any review ran.

Paired sign tests. Some pull requests are hard for every model, so comparing totals mixes up a weaker model with harder pull requests. Every model reviewed the same pull requests, so for each pair we looked only at the defects exactly one of them found, and ran an exact sign test on that split. Defects within one pull request can be related, so the tests are a guide, not proof.

Fixing the scorer. Code review of our scoring script found bugs our data hadn’t triggered yet; Lightfactory’s own review of the script caught one of them. A skipped verdict counted as “no match”, models were scored over pull requests they never reviewed, an unknown cost was added as zero, and a mistyped defect id could count as a match. We fixed each test-first, and rescoring both runs reproduced the published numbers exactly.

Results: luna found about half as many defects

The Claude figures are usage-based charges. The OpenAI figures are list-price estimates and an upper bound.

• Every other model caught more labelled defects than luna. terra found 23 defects luna missed, and luna 2 that terra missed (p < 0.0001). Against the other four, p ranged from 0.0003 to 0.007. Four of the five comparisons hold after correcting for all 15 model pairs; Haiku’s p = 0.007 does not.
• The test did not clearly separate the five models ahead of luna. terra had the highest observed count. sol and terra split 6 to 8 on the defects only one of them found; Opus and Haiku 14 to 9. Every pair among the five had p of 0.17 or more.
• The gap grew with pull request size. Under 300 changed lines, every model found 6 to 8 of 23 defects. From 300 to 1,000 lines, luna found 8 of 43 and the others 12 to 15. Over 1,000 lines, luna found 8 of 47 and the others 14 to 21. This breakdown describes the sample; it is consistent with, but can’t prove, why our small corpus showed every model level.
• Opus against Haiku. Opus found 40 to Haiku’s 35, and the evidence is inconclusive on whether that would repeat. Its total charge across the 22 attempts was about 9 times Haiku’s, and its median charge about 16 times.

Is a second reviewer worth it?

terra and Haiku together found 52 defects in the original scoring, against terra’s 43 alone. That could be the effect of any second review, so we ran terra again on the same 22 pull requests and rematched both runs blind.

The rematch moved terra plus Haiku from 52 to 53 on one borderline Haiku match. terra’s two runs shared 40 of their 43 defects: its misses repeat. Haiku found 8 that neither terra run found, two of them high severity, among them a path-traversal check missing a trailing separator. Paired against either terra run, Haiku’s addition gave p = 0.039 or 0.109. Without the borderline match, about 0.07 or 0.18.

Haiku added more labelled defects than a second terra run on these pull requests. We repeated terra once and ran Haiku once, so we’d want more pull requests before adding a second reviewer.

We also had Claude agents read the code for each of the 84 findings from those three runs that the answer key lacked. They judged 78 real, without confirmation from the projects’ maintainers. Each run had 2 findings judged false, so 97% to 98% of its findings were judged correct. That shows gaps in the answer key; it can’t establish the true recall or precision of all six models.

Why we chose gpt-6.1-terra

terra had the highest observed defect count, about twice luna’s, and the fastest median review at 34 seconds. On defects per dollar, Haiku led on the recorded charges. A replay over the same benchmark outputs, routing each pull request to a cheaper or stronger model by risk, found 41 defects for about four times terra’s charge.

So on 2 November we made gpt-6.1-terra the default for our production reviewer, including its security pass. We haven’t added a second reviewer per round yet; Haiku’s gain rests on a handful of defects, and we want more pull requests first.

Limits

Each model reviewed each pull request once, except terra. The judges were Claude agents, blind to the model. The answer key came partly from frontier models, which may favor stronger ones, though that doesn’t explain luna’s gap, since the small Haiku did better. The pull requests are public and may be in the models’ training data. Every review ran at one reasoning effort. The OpenAI and Anthropic charges are measured differently, so cost ratios across providers are approximate.