Architectural context makes AI code review measurably better.
We replayed a public code-review benchmark — 40 real pull requests with 105 human-graded defects across cal.com, Discourse, Grafana, and Keycloak — and compared reviewers under one strict, identical grader. Feeding a model CodeAtlas's architectural context instead of a raw diff lifts every metric that matters: it finds more real bugs, with fewer false alarms.
CodeAtlas + Claude
Claude Opus reasoning over CodeAtlas review context
Claude + raw diff
Claude Opus over a plain git diff — no CodeAtlas
CodeAtlas + DeepSeek
Cheap deepseek-v4-flash over CodeAtlas context, with false-positive reducers
CodeRabbit
Commercial AI PR-review bot (3-judge average)
Overall results
All four reviewers scored on the identical 40 PRs / 105 golden defects by one strict 1:1 oracle judge (a true positive must match a golden defect's root cause and location; true positives are capped at the golden count, so over-reporting never inflates the score).
| Reviewer | Findings | True pos. | False pos. | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
CodeAtlas + Claude Claude Opus reasoning over CodeAtlas review context | 106 | 76/105 | 30 | 0.72 | 0.72 | 0.72 |
Claude + raw diff Claude Opus over a plain git diff — no CodeAtlas | 119 | 54/105 | 65 | 0.45 | 0.51 | 0.48 |
CodeAtlas + DeepSeek Cheap deepseek-v4-flash over CodeAtlas context, with false-positive reducers | 193 | 58/105 | 135 | 0.30 | 0.55 | 0.39 |
CodeRabbit Commercial AI PR-review bot (3-judge average) | 254 | 65.4/105 | 188.6 | 0.26 | 0.62 | 0.36 |
CodeAtlas + Claude is the best-of-N review across five samples (highest-recall configuration). CodeRabbit is a 3-judge average; its fractional true-positive count is rounded in per-PR views.
What the data shows
Context beats the raw diff
The same model — Claude — finds 76 of 105 defects on CodeAtlas context versus 54 on a plain git diff. The architecture pack (entry points, cross-file callers, dependents) is what closes the gap, not a bigger model.
Highest signal-to-noise
CodeAtlas + Claude reaches 72% recall with only 106 findings. CodeRabbit needs 254 findings for 62% recall — more than twice the review noise for fewer real bugs. Precision 0.72 vs 0.26.
A budget model + context out-finds a frontier model alone
Cheap deepseek-v4-flash reading CodeAtlas context catches 55% of all defects — a better chance at the bug than frontier Claude gets on a plain git diff (only 51%). The right context beats a bigger model.
The context carries recall
Swap Claude for cheap deepseek-v4-flash on the same CodeAtlas context and recall holds at 55% — the context, not the model, drives bug discovery. False-positive reducers then make even the budget model competitive (see below).
Cutting false positives: a cheap model made competitive
A frontier reviewer is expensive. We asked whether a budget model — deepseek-v4-flash, a fraction of the cost — could be made genuinely useful by attacking its one real weakness: false positives. We added precision gates to the review prompt (only flag with diff evidence, never comment on unchanged context code, treat redacted secrets as opaque) plus a deterministic in-diff filter. Same model, same context, same 105-defect golden set — only the reducers changed.
Cheap model vs the field
The same 40 PRs / 105 golden defects, same strict judge. DeepSeek is a small fraction of the cost of the alternatives.
| Reviewer | Findings | True pos. | False pos. | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
CodeAtlas + DeepSeek cheap model, FP-reduced | 193 | 58/105 | 135 | 0.30 | 0.55 | 0.39 |
CodeRabbit commercial PR-review bot | 254 | 65.4/105 | 189 | 0.26 | 0.62 | 0.36 |
Claude + raw diff frontier model, no context | 119 | 54/105 | 65 | 0.45 | 0.51 | 0.48 |
With the reducers, the cheap model now beats CodeRabbit on both F1 and precision — and does it with 61 fewer findings (193 vs 254), so reviewers wade through far less noise. It also out-finds frontier Claude-on-raw-diff on recall (58 vs 54 real bugs); raw-diff keeps a higher F1 only on precision, exactly the gap the in-diff filter narrows. The takeaway: the false-positive levers are model-agnostic — they make a budget reviewer viable, not just a frontier one.
By repository & language
True positives out of golden defects per repo. CodeAtlas leads on recall in every codebase — TypeScript, Ruby/SCSS, Go, and Java — showing the gains are not language-specific.
| Repository | Stack | Golden | CodeAtlas+Claude | CodeRabbit | CodeAtlas+DeepSeek | Raw diff |
|---|---|---|---|---|---|---|
| cal.com | TypeScript / tRPC | 31 | 24 · 41f | 25 · 88f | 18 · 73f | 17 · 54f |
| discourse | Ruby / SCSS | 28 | 23 · 30f | 13 · 78f | 16 · 39f | 15 · 27f |
| grafana | Go / TypeScript | 22 | 14 · 17f | 12 · 36f | 11 · 37f | 10 · 22f |
| keycloak | Java | 24 | 15 · 18f | 16 · 52f | 13 · 44f | 12 · 16f |
Every pull request
All 40 PRs, per reviewer. Each cell shows true positives / golden and total findings (Nf). Highlighted = every golden defect found.
| PR | Gold | CodeAtlas+Claude | CodeRabbit | CA+DeepSeek | Raw diff |
|---|---|---|---|---|---|
| cal.com | |||||
| #7232 | 2 | 2/2(7f) | 2/2(13f) | 2/2(10f) | 2/2(7f) |
| #8087 | 2 | 1/2(3f) | 2/2(8f) | 1/2(5f) | 1/2(3f) |
| #8330 | 2 | 2/2(2f) | 2/2(4f) | 2/2(7f) | 2/2(5f) |
| #10600 | 4 | 3/4(5f) | 2/4(8f) | 1/4(8f) | 1/4(5f) |
| #10967 | 5 | 3/5(4f) | 4/5(12f) | 3/5(7f) | 2/5(6f) |
| #11059 | 5 | 4/5(10f) | 5/5(19f) | 3/5(8f) | 3/5(10f) |
| #14740 | 5 | 4/5(4f) | 4/5(10f) | 3/5(18f) | 3/5(7f) |
| #14943 | 2 | 2/2(3f) | 2/2(3f) | 0/2(2f) | 1/2(3f) |
| #22345 | 2 | 1/2(1f) | 1/2(4f) | 1/2(4f) | 0/2(3f) |
| #22532 | 2 | 2/2(2f) | 1/2(7f) | 2/2(4f) | 2/2(5f) |
| discourse | |||||
| #1 | 3 | 2/3(3f) | 2/3(7f) | 2/3(3f) | 2/3(3f) |
| #2 | 2 | 1/2(1f) | 2/2(8f) | 1/2(5f) | 1/2(3f) |
| #3 | 2 | 2/2(2f) | 1/2(7f) | 1/2(4f) | 1/2(2f) |
| #4 | 6 | 6/6(6f) | 4/6(23f) | 5/6(6f) | 2/6(4f) |
| #5 | 2 | 1/2(1f) | 0/2(1f) | 1/2(1f) | 0/2(0f) |
| #6 | 1 | 1/1(2f) | 1/1(2f) | 1/1(3f) | 1/1(1f) |
| #7 | 3 | 3/3(5f) | 0/3(2f) | 1/3(1f) | 3/3(5f) |
| #8 | 3 | 2/3(2f) | 2/3(10f) | 3/3(5f) | 1/3(2f) |
| #9 | 2 | 1/2(1f) | 0/2(3f) | 0/2(3f) | 1/2(2f) |
| #10 | 4 | 4/4(7f) | 2/4(15f) | 1/4(8f) | 3/4(5f) |
| grafana | |||||
| #76186 | 2 | 1/2(1f) | 1/2(2f) | 2/2(7f) | 0/2(0f) |
| #79265 | 5 | 2/5(2f) | 3/5(8f) | 2/5(4f) | 2/5(3f) |
| #80329 | 1 | 1/1(2f) | 1/1(4f) | 1/1(1f) | 1/1(2f) |
| #90045 | 3 | 3/3(5f) | 3/3(5f) | 2/3(9f) | 3/3(5f) |
| #90939 | 2 | 1/2(1f) | 1/2(2f) | 1/2(3f) | 1/2(1f) |
| #94942 | 2 | 2/2(2f) | 1/2(2f) | 2/2(2f) | 1/2(1f) |
| #97529 | 2 | 1/2(1f) | 1/2(4f) | 1/2(1f) | 1/2(2f) |
| #103633 | 2 | 2/2(2f) | 0/2(1f) | 0/2(2f) | 0/2(2f) |
| #106778 | 2 | 1/2(1f) | 1/2(7f) | 0/2(6f) | 1/2(4f) |
| #107534 | 1 | 0/1(0f) | 0/1(1f) | 0/1(2f) | 0/1(2f) |
| keycloak | |||||
| #32918 | 2 | 1/2(1f) | 1/2(2f) | 0/2(3f) | 2/2(2f) |
| #33832 | 2 | 1/2(1f) | 1/2(5f) | 2/2(7f) | 2/2(2f) |
| #36880 | 3 | 2/3(2f) | 1/3(6f) | 3/3(7f) | 2/3(2f) |
| #36882 | 1 | 0/1(0f) | 0/1(2f) | 0/1(2f) | 0/1(1f) |
| #37038 | 2 | 2/2(2f) | 2/2(10f) | 1/2(5f) | 1/2(1f) |
| #37429 | 4 | 2/4(3f) | 3/4(9f) | 2/4(3f) | 0/4(2f) |
| #37634 | 4 | 3/4(4f) | 3/4(9f) | 2/4(7f) | 2/4(3f) |
| #38446 | 2 | 2/2(3f) | 2/2(4f) | 0/2(4f) | 1/2(1f) |
| #40940 | 2 | 2/2(2f) | 1/2(1f) | 1/2(2f) | 2/2(2f) |
| greptile#1 | 2 | 0/2(0f) | 1/2(4f) | 2/2(4f) | 0/2(0f) |
The actual review comments
Numbers hide what a review actually reads like. Pick any repository and pull request below to read the real, verbatim comments each reviewer wrote — Claude on a raw diff, Claude on CodeAtlas context, and the cheap DeepSeek model on CodeAtlas context — side by side, next to the human-graded defects (⭐). All 40 PRs, same strict judge as the tables above.
- G1MediumAsynchronous functions deleteScheduledEmailReminder and deleteScheduledSMSReminder are called without await inside forEach loops. This occurs during booking rescheduling/cancellation, and…
- G2HighWhen immediateDelete is true, the deleteScheduledEmailReminder function cancels the SendGrid email but fails to delete the corresponding WorkflowReminder record from the database. This creates…
deleteScheduledEmailReminder/deleteScheduledSMSReminder are now called inside an un-awaited nested forEach and were removed from the awaited prismaPromises set (now `[attendeeDeletes, bookingReferenceDeletes]`), so handler returns 'Booking successfully cancelled' before the reminders are actually cancelled. On a…
The forEach calls deleteScheduledEmailReminder/SMS without await, so the wrapping try/catch can only catch synchronous throws (there are none) and cannot catch the un-awaited promises' rejections; the cancellations are fire-and-forget, so a rescheduled booking's previous reminders may still fire and the catch's…
The PR removed `await Promise.all(remindersToDelete)` and now invokes deleteScheduledEmailReminder/SMS inside an un-awaited forEach. Cancellation is a DB update setting cancelled (email) or a delete (SMS); fire-and-forget means it may not commit before the mutation returns, so the previous booking's reminders are not…
After removing the per-reminder prisma.workflowReminder.deleteMany and `await Promise.all(deleteReminderPromise)`, reminder cancellation now runs via un-awaited deleteScheduledEmailReminder/SMS calls (including a `remindersToUpdate.forEach(async (reminder) => {...})` whose callbacks are never awaited). The…
The newly-added loop awaits a SendGrid client.request per reminder inside a single try/catch; one request failure throws out of the for-loop, skipping cancellation and deletion of all remaining reminders, and is only console.log'd. Other cancelled bookings then keep sending reminder emails until a later run. Handle…
The handler sets `req.userId` to -1 when no session is present, allowing unauthenticated users to attempt booking cancellation. The route is state-mutating (DELETE) but lacks authentication middleware or explicit rejection of unauthenticated requests. The absent middleware list confirms no auth guard.
If a SendGrid cancel request fails mid-loop, the for block breaks and the previously collected delete promises are never awaited, leaving cancelled reminder records in the database. This causes orphaned state and potential duplicate cancel attempts on subsequent runs.
The comment says the code should cancel reminders for both cancelled and rescheduled bookings, but the Prisma query filters only on `cancelled: true`. If rescheduled reminders are not marked with `cancelled: true`, they will be missed and the reminder will be sent anyway.
The added `client.request` call to cancel scheduled sends has no explicit timeout. If the SendGrid API is slow, the handler may hang indefinitely, potentially exceeding the serverless function timeout and causing the entire cron run to fail.
The array `workflowRemindersToDelete` is typed as `Prisma.Prisma__WorkflowReminderClient<WorkflowReminder, never>[]`. The second type parameter `never` does not match the actual return type of `prisma.workflowReminder.delete()`, which may cause a TypeScript compilation error or type incompatibility.
The PR removes the `prisma.workflowReminder.deleteMany` call that previously deleted workflow reminder records after rescheduling a booking. The new code only calls `deleteScheduledEmailReminder`/`deleteScheduledSMSReminder` but does not delete the database records. This leaves orphaned reminder records that can cause…
The PR rewrites the reminder-cancellation loops leaving every call to `deleteScheduledEmailReminder` and `deleteScheduledSMSReminder` unawaited inside `forEach` without a `.catch` handler. Because these are async functions performing outbound network requests, any rejection becomes an unhandled promise rejection. The…
The old code guarded every cancellation call with checks on `reminder.scheduled` and `reminder.referenceId`. The PR removes these guards and passes `reminder.referenceId` directly. Since the DB schema allows `referenceId` to be null, this passes `null` or `undefined` to downstream functions that expect a string…
The PR removes calls to `ctx.prisma.workflowReminder.deleteMany` that used to delete reminder records from the database after canceling external reminders in multiple code paths (event type disable, step edit, deactivation, etc.). This leaves orphan records that can cause data inconsistency and affect subsequent…
In some code paths, `deleteScheduledEmailReminder` is called with three arguments (reminder.id, reminder.referenceId, true), while in others it is called with two arguments (reminder.id, reminder.referenceId). This inconsistency suggests a copy-paste error and may lead to unexpected behavior depending on the code…
The import is `import logger from "@calcom/lib/logger"` but the error handler uses `log.error` which is not defined. This will cause a ReferenceError at runtime.
The function signature was changed from (referenceId: string) to (reminderId: number, referenceId: string | null, immediateDelete?: boolean). The dependent file reminderScheduler.ts (symbols scheduleWorkflowReminders, sendCancelledReminders) still calls the old signature and is not updated in this PR.
The old code collected the reminder DB-delete promises into `remindersToDelete` and awaited them through the final `Promise.all`. The new code calls `deleteScheduledEmailReminder(reminder.id, reminder.referenceId)` / `deleteScheduledSMSReminder(...)` inside `updatedBookings.forEach(...)` without `await`, and the…
The PR removed `await Promise.all(remindersToDelete);` and now just fires `deleteScheduledEmailReminder(reminder.id, reminder.referenceId)` / `deleteScheduledSMSReminder(...)` inside `bookingToReschedule.workflowReminders.forEach(...)` with no await and no collection of the returned promises. The DB state mutation…
Several loops were converted from awaited DB deletes to fire-and-forget calls. The PR removed `await Promise.all(deleteReminderPromise);` and the per-reminder `ctx.prisma.workflowReminder.deleteMany(...)`, replacing them with unawaited `deleteScheduledEmailReminder(reminder.id, reminder.referenceId)` calls. Worse, one…
For the default (non-immediate) path `deleteScheduledEmailReminder` no longer cancels the SendGrid scheduled send; it only marks the row: `await prisma.workflowReminder.update({ where: { id: reminderId }, data: { cancelled: true } });`. The actual SendGrid cancellation is deferred to the hourly cron in…
In the new loop `await client.request({ url: "/v3/user/scheduled_sends", ... })` runs per reminder inside the `try`, and the collected `prisma.workflowReminder.delete(...)` promises are only awaited after the loop via `await Promise.all(workflowRemindersToDelete);`. If any single `client.request` throws, control jumps…
The `if (immediateDelete) { await client.request({...status: "cancel"}); return; }` branch cancels the SendGrid send but does not delete the `workflowReminder` row nor set `cancelled: true`. The old call sites used to follow the cancel with an explicit `deleteMany`, which this PR removed. For callers where the row is…
The new block wraps `originalRescheduledBooking.workflowReminders.forEach((reminder) => { ... deleteScheduledEmailReminder(reminder.id, reminder.referenceId, true); ... })` in a `try { ... } catch (error) { log.error(...) }`. Because the `deleteScheduled*` functions are async and are not awaited inside the synchronous…
Comments are verbatim from the benchmark runs (bodies trimmed to the core claim; code tokens kept as-is; long panels folded as “+N more”). Graded-recall figures are the canonical judged numbers from the leaderboard. Public golden defect text is available for the cal.com PRs; for the other repos the graded defect count and recall come from the same strict judge.
Methodology
Dataset. A public code-review benchmark of 40 merged pull requests across cal.com (TypeScript/tRPC), Discourse (Ruby/SCSS), Grafana (Go/TypeScript), and Keycloak (Java), with 105 defects independently graded by human reviewers as the golden set.
Identical grader. Every reviewer is scored by the same strict 1:1 oracle judge. A finding counts as a true positive only when it matches a golden defect's root cause and location; true positives are capped at the golden count so a reviewer cannot inflate its score by flooding findings. Precision = TP / findings, Recall = TP / golden, F1 is their harmonic mean.
CodeAtlas context. Instead of a raw diff, the reviewer receives CodeAtlas's assembled review context: the changed entry-points, their cross-file callers and implementers, dependent and sibling code, tests, and a windowed diff — the same architectural snapshot that powers the IDE diagrams and the MCP tools. CodeAtlas + Claude here is the best-of-N (highest-recall) review across five samples; the other reviewers are single runs as published.
Competitors. CodeRabbit numbers are a 3-judge average from the benchmark's own published evaluations. Raw-diff Claude uses the identical model with only a git diff. CodeAtlas + DeepSeek runs the cheap deepseek-v4-flash over the exact same CodeAtlas context to isolate the model's contribution from the context's; its figures here include the false-positive reducers (precision gates + in-diff filter) described in the section above — the pre-reducer baseline was 278 findings / 219 false positives / F1 0.31.
The context is the product
The same architectural snapshot that draws CodeAtlas's live diagrams and answers your AI assistant's questions is what makes its code review state-of-the-art. One engine, many surfaces.