Research·40 PRs · 105 graded defects · 4 OSS repos

Architectural context makes AI code review measurably better.

We replayed a public code-review benchmark — 40 real pull requests with 105 human-graded defects across cal.com, Discourse, Grafana, and Keycloak — and compared reviewers under one strict, identical grader. Feeding a model CodeAtlas's architectural context instead of a raw diff lifts every metric that matters: it finds more real bugs, with fewer false alarms.

Best overall

0.72

F1 score

CodeAtlas + Claude

Claude Opus reasoning over CodeAtlas review context

72%

Recall

72%

Prec.

106

Findings

0.48

F1 score

Claude + raw diff

Claude Opus over a plain git diff — no CodeAtlas

51%

Recall

45%

Prec.

119

Findings

0.39

F1 score

CodeAtlas + DeepSeek

Cheap deepseek-v4-flash over CodeAtlas context, with false-positive reducers

55%

Recall

30%

Prec.

193

Findings

0.36

F1 score

CodeRabbit

Commercial AI PR-review bot (3-judge average)

62%

Recall

26%

Prec.

254

Findings

Overall results

All four reviewers scored on the identical 40 PRs / 105 golden defects by one strict 1:1 oracle judge (a true positive must match a golden defect's root cause and location; true positives are capped at the golden count, so over-reporting never inflates the score).

Reviewer	Findings	True pos.	False pos.	Precision	Recall	F1
CodeAtlas + Claude Claude Opus reasoning over CodeAtlas review context	106	76/105	30	0.72	0.72	0.72
Claude + raw diff Claude Opus over a plain git diff — no CodeAtlas	119	54/105	65	0.45	0.51	0.48
CodeAtlas + DeepSeek Cheap deepseek-v4-flash over CodeAtlas context, with false-positive reducers	193	58/105	135	0.30	0.55	0.39
CodeRabbit Commercial AI PR-review bot (3-judge average)	254	65.4/105	188.6	0.26	0.62	0.36

CodeAtlas + Claude is the best-of-N review across five samples (highest-recall configuration). CodeRabbit is a 3-judge average; its fractional true-positive count is rounded in per-PR views.

What the data shows

+22 bugs

Context beats the raw diff

The same model — Claude — finds 76 of 105 defects on CodeAtlas context versus 54 on a plain git diff. The architecture pack (entry points, cross-file callers, dependents) is what closes the gap, not a bigger model.

⅓ the noise

Highest signal-to-noise

CodeAtlas + Claude reaches 72% recall with only 106 findings. CodeRabbit needs 254 findings for 62% recall — more than twice the review noise for fewer real bugs. Precision 0.72 vs 0.26.

55%›51%

A budget model + context out-finds a frontier model alone

Cheap deepseek-v4-flash reading CodeAtlas context catches 55% of all defects — a better chance at the bug than frontier Claude gets on a plain git diff (only 51%). The right context beats a bigger model.

Model-agnostic

The context carries recall

Swap Claude for cheap deepseek-v4-flash on the same CodeAtlas context and recall holds at 55% — the context, not the model, drives bug discovery. False-positive reducers then make even the budget model competitive (see below).

New · false-positive reduction

Cutting false positives: a cheap model made competitive

A frontier reviewer is expensive. We asked whether a budget model — deepseek-v4-flash, a fraction of the cost — could be made genuinely useful by attacking its one real weakness: false positives. We added precision gates to the review prompt (only flag with diff evidence, never comment on unchanged context code, treat redacted secrets as opaque) plus a deterministic in-diff filter. Same model, same context, same 105-defect golden set — only the reducers changed.

−38%

false positives

219 → 135 bad comments · recall held (59 → 58 bugs found)

0.21 → 0.30

precision

31% fewer findings (278 → 193), almost all of them noise

0.31 → 0.39

F1 score

from the noisiest reviewer to one that edges a commercial bot

Cheap model vs the field

The same 40 PRs / 105 golden defects, same strict judge. DeepSeek is a small fraction of the cost of the alternatives.

Reviewer	Findings	True pos.	False pos.	Precision	Recall	F1
CodeAtlas + DeepSeek cheap model, FP-reduced	193	58/105	135	0.30	0.55	0.39
CodeRabbit commercial PR-review bot	254	65.4/105	189	0.26	0.62	0.36
Claude + raw diff frontier model, no context	119	54/105	65	0.45	0.51	0.48

With the reducers, the cheap model now beats CodeRabbit on both F1 and precision — and does it with 61 fewer findings (193 vs 254), so reviewers wade through far less noise. It also out-finds frontier Claude-on-raw-diff on recall (58 vs 54 real bugs); raw-diff keeps a higher F1 only on precision, exactly the gap the in-diff filter narrows. The takeaway: the false-positive levers are model-agnostic — they make a budget reviewer viable, not just a frontier one.

By repository & language

True positives out of golden defects per repo. CodeAtlas leads on recall in every codebase — TypeScript, Ruby/SCSS, Go, and Java — showing the gains are not language-specific.

Repository	Stack	Golden	CodeAtlas+Claude	CodeRabbit	CodeAtlas+DeepSeek	Raw diff
cal.com	TypeScript / tRPC	31	24 · 41f	25 · 88f	18 · 73f	17 · 54f
discourse	Ruby / SCSS	28	23 · 30f	13 · 78f	16 · 39f	15 · 27f
grafana	Go / TypeScript	22	14 · 17f	12 · 36f	11 · 37f	10 · 22f
keycloak	Java	24	15 · 18f	16 · 52f	13 · 44f	12 · 16f

Every pull request

All 40 PRs, per reviewer. Each cell shows true positives / golden and total findings (Nf). Highlighted = every golden defect found.

PR	Gold	CodeAtlas+Claude	CodeRabbit	CA+DeepSeek	Raw diff
cal.com
#7232	2	2/2(7f)	2/2(13f)	2/2(10f)	2/2(7f)
#8087	2	1/2(3f)	2/2(8f)	1/2(5f)	1/2(3f)
#8330	2	2/2(2f)	2/2(4f)	2/2(7f)	2/2(5f)
#10600	4	3/4(5f)	2/4(8f)	1/4(8f)	1/4(5f)
#10967	5	3/5(4f)	4/5(12f)	3/5(7f)	2/5(6f)
#11059	5	4/5(10f)	5/5(19f)	3/5(8f)	3/5(10f)
#14740	5	4/5(4f)	4/5(10f)	3/5(18f)	3/5(7f)
#14943	2	2/2(3f)	2/2(3f)	0/2(2f)	1/2(3f)
#22345	2	1/2(1f)	1/2(4f)	1/2(4f)	0/2(3f)
#22532	2	2/2(2f)	1/2(7f)	2/2(4f)	2/2(5f)
discourse
#1	3	2/3(3f)	2/3(7f)	2/3(3f)	2/3(3f)
#2	2	1/2(1f)	2/2(8f)	1/2(5f)	1/2(3f)
#3	2	2/2(2f)	1/2(7f)	1/2(4f)	1/2(2f)
#4	6	6/6(6f)	4/6(23f)	5/6(6f)	2/6(4f)
#5	2	1/2(1f)	0/2(1f)	1/2(1f)	0/2(0f)
#6	1	1/1(2f)	1/1(2f)	1/1(3f)	1/1(1f)
#7	3	3/3(5f)	0/3(2f)	1/3(1f)	3/3(5f)
#8	3	2/3(2f)	2/3(10f)	3/3(5f)	1/3(2f)
#9	2	1/2(1f)	0/2(3f)	0/2(3f)	1/2(2f)
#10	4	4/4(7f)	2/4(15f)	1/4(8f)	3/4(5f)
grafana
#76186	2	1/2(1f)	1/2(2f)	2/2(7f)	0/2(0f)
#79265	5	2/5(2f)	3/5(8f)	2/5(4f)	2/5(3f)
#80329	1	1/1(2f)	1/1(4f)	1/1(1f)	1/1(2f)
#90045	3	3/3(5f)	3/3(5f)	2/3(9f)	3/3(5f)
#90939	2	1/2(1f)	1/2(2f)	1/2(3f)	1/2(1f)
#94942	2	2/2(2f)	1/2(2f)	2/2(2f)	1/2(1f)
#97529	2	1/2(1f)	1/2(4f)	1/2(1f)	1/2(2f)
#103633	2	2/2(2f)	0/2(1f)	0/2(2f)	0/2(2f)
#106778	2	1/2(1f)	1/2(7f)	0/2(6f)	1/2(4f)
#107534	1	0/1(0f)	0/1(1f)	0/1(2f)	0/1(2f)
keycloak
#32918	2	1/2(1f)	1/2(2f)	0/2(3f)	2/2(2f)
#33832	2	1/2(1f)	1/2(5f)	2/2(7f)	2/2(2f)
#36880	3	2/3(2f)	1/3(6f)	3/3(7f)	2/3(2f)
#36882	1	0/1(0f)	0/1(2f)	0/1(2f)	0/1(1f)
#37038	2	2/2(2f)	2/2(10f)	1/2(5f)	1/2(1f)
#37429	4	2/4(3f)	3/4(9f)	2/4(3f)	0/4(2f)
#37634	4	3/4(4f)	3/4(9f)	2/4(7f)	2/4(3f)
#38446	2	2/2(3f)	2/2(4f)	0/2(4f)	1/2(1f)
#40940	2	2/2(2f)	1/2(1f)	1/2(2f)	2/2(2f)
greptile#1	2	0/2(0f)	1/2(4f)	2/2(4f)	0/2(0f)

Methodology

Dataset. A public code-review benchmark of 40 merged pull requests across cal.com (TypeScript/tRPC), Discourse (Ruby/SCSS), Grafana (Go/TypeScript), and Keycloak (Java), with 105 defects independently graded by human reviewers as the golden set.

Identical grader. Every reviewer is scored by the same strict 1:1 oracle judge. A finding counts as a true positive only when it matches a golden defect's root cause and location; true positives are capped at the golden count so a reviewer cannot inflate its score by flooding findings. Precision = TP / findings, Recall = TP / golden, F1 is their harmonic mean.

CodeAtlas context. Instead of a raw diff, the reviewer receives CodeAtlas's assembled review context: the changed entry-points, their cross-file callers and implementers, dependent and sibling code, tests, and a windowed diff — the same architectural snapshot that powers the IDE diagrams and the MCP tools. CodeAtlas + Claude here is the best-of-N (highest-recall) review across five samples; the other reviewers are single runs as published.

Competitors. CodeRabbit numbers are a 3-judge average from the benchmark's own published evaluations. Raw-diff Claude uses the identical model with only a git diff. CodeAtlas + DeepSeek runs the cheap deepseek-v4-flash over the exact same CodeAtlas context to isolate the model's contribution from the context's; its figures here include the false-positive reducers (precision gates + in-diff filter) described in the section above — the pre-reducer baseline was 278 findings / 219 false positives / F1 0.31.

The context is the product

The same architectural snapshot that draws CodeAtlas's live diagrams and answers your AI assistant's questions is what makes its code review state-of-the-art. One engine, many surfaces.