A diff is the wrong unit of context
Change a function's signature and the diff shows you the function. It doesn't show the caller two files away that still passes two arguments. A reviewer working from the diff alone can't see that break. One fed entire files pays for code that doesn't matter and still has to guess which parts do.
Cheap, and blind to anything outside the hunk.
Sees more, pays for all of it, and the signal gets diluted.
Knows what can break up front, then pulls only what it needs.
GraphReview takes the third path, and the eval harness runs all three as an ablation so the difference is measured, not asserted.
A review path and a retrieval path
- Left column, the review path. Gateway → four agents → aggregator → critic → publisher. Every hop between services is a Kafka topic.
- Right column, retrieval. The indexer builds a code graph (Redis) and a vector index (Qdrant) ahead of time. The context service answers the agents' tool calls from them.
| Kafka topic | Key | Producer → consumer group(s) |
|---|---|---|
pr.review.requested | job_id | gateway → agent-static, agent-reviewer, agent-security, agent-tests (fan-out) |
review.agent.results | job_id | agents → aggregator (all results for a job land on one partition) |
review.aggregated | job_id | aggregator → critic |
review.completed | job_id | critic → publisher |
repo.index.requested | repo | gateway → indexer |
review.dlq | original | any worker, after 3 failed attempts |
Services talk only through Kafka
- Gateway verifies the
X-Hub-Signature-256HMAC, fetches PR files and patches, dedupes redeliveries per head SHA withSETNX, stores the job in Redis and publishes it. - Agents each own a consumer group, so every role sees every job while replicas within a role split partitions. All four run their tool loops at the same time.
- Aggregator joins results per job in Redis, clusters near-duplicates and raises confidence when independent agents agree. A deadline sweeper flushes partial results if an agent never reports.
- Critic judges each LLM finding against the diff and packed graph context, drops false positives, recalibrates severity and writes the verdict. Deterministic static findings skip the judge.
- Publisher posts one GitHub review. Findings on diff lines become inline comments; findings outside the diff, like a broken caller in an untouched file, go in the review body.
Fan-out without a router
That's how one PR reaches all four agents without the gateway knowing they exist, and how one role scales out on its own. A seventh reviewer replica would sit idle with only six partitions to own, so autoscaling caps at six.
A bounded tool loop, not a giant prompt
The reviewer, security and tests agents are one engine with different system prompts, tool lists and effort levels. Claude gets the diff with new-side line numbers in the margin, so it can cite exact lines, plus a short graph impact report. Then it decides what else to look at.
tool_use loops, end_turn carries schema-valid findings, anything else becomes an error result so the pipeline keeps moving. At most 8 turns; the last one disables tools.| Tool | Question it answers | Backed by |
|---|---|---|
get_callers | Who calls this function, 1–3 hops out, and what does it call? | code graph |
find_tests | Which tests exercise this symbol? | code graph |
search_code | Where else does the codebase do something similar? | vectors (code) |
search_docs | What do this repo's own docs say the rule is? | vectors (docs) |
read_snippet | Show me lines 40–80 of this file. | PR head / index |
- Parallel tool calls run concurrently and all results go back in one message. Splitting them would teach the model to stop batching.
- Assistant turns are replayed verbatim, thinking blocks included. History is append-only, so the prompt cache hits on every turn.
- PR text is data, never instructions. Every system prompt frames diff, code and docs as untrusted, and
read_snippetblocks path traversal.
A graph for structure, vectors for meaning
Vector search can't reliably tell you who calls a function, and a graph can't find code that is merely similar. So each question goes to the structure that can answer it.
| Question a reviewer asks | Answered by |
|---|---|
| Which functions did this hunk touch? | graph line → innermost enclosing symbol |
| Who breaks if this signature changes? | graph reverse BFS over calls / imports / inherits |
| Is this behaviour tested? | graph tests reachable from the changed symbols |
| How does the rest of the codebase handle X? | vectors dense BGE-small + sparse BM25, RRF fusion |
| Does this violate a team convention? | vectors over repo docs (CONTRIBUTING, ADRs) |
read_snippet can rebuild any line range without cloning at review time.From an edited line to what it can break
Seeded case 05 gives compute_tax a new required parameter. The graph finds the innermost function around the edited line, then walks edges backwards, callee to caller, up to two hops. Callees are skipped: changing compute_tax can't break what it calls.
order_total still calls it with two arguments and isn't in the diff. A diff-only reviewer never sees it.This is the text every agent receives at the top of its prompt:
Hybrid search: meaning and exact words
compute_tax. Reciprocal rank fusion puts results that rank well in both first.Graph builder. The indexer runs graphify extract --code-only (tree-sitter, 25+ languages, zero LLM cost) and falls back to a built-in, import-aware Python AST builder that emits the same schema. Graph blobs live in Redis, gzip-compressed and versioned, so context-service replicas stay stateless.
The same retrieval tools are exposed over MCP, so the index can be queried from Claude Code or Cursor:
Agreement is evidence; a skeptic gets the last word
Four agents on the same diff often flag the same problem. The aggregator clusters findings in the same file within ±3 lines that share a category or similar wording, keeps the most severe and confident one, and records who agreed.
Two agents at 0.90 → 0.94. Independent agreement is treated as evidence, not noise.
The critic is a second Claude pass with a different job: be skeptical. It sees the diff, the packed graph context and the proposed findings, returns keep/drop, confidence and severity per finding, drops anything under 0.6, and writes the summary and verdict. Rejected findings aren't deleted; the dashboard lists them under "Rejected by critic".
At-least-once delivery, idempotent everywhere
review.dlq. One bad message can't block a partition forever.| What goes wrong | What happens |
|---|---|
| GitHub redelivers a webhook | Same repo + PR + head SHA returns the existing job id (SETNX dedupe:…) |
| Claude refuses or the API errors | SDK retries 429/5xx; server-side fallback re-runs refusals; otherwise the agent publishes a result carrying the error |
| An agent never reports | Deadline sweeper flushes partial results after 300 s; agent shown as timeout |
| The same result arrives twice | Stored per (job_id, agent), so the second write is a no-op; flush guarded by SETNX |
| Posting to GitHub fails | Guard key removed so the retry can post; after 3 failures → review.dlq |
| A pod is killed mid-review | Offset not committed, another replica re-reads it; a 300 s grace period lets most reviews finish |
| PR says "ignore previous instructions…" | All PR content framed as untrusted data; the output schema limits what the model can return |
Long LLM handlers get max.poll.interval.ms=600000 and max_poll_records=1. The producer is idempotent with acks=all.
Scale agents on lag, not CPU
Agents spend their time waiting on Claude, so CPU says nothing about how far behind they are. The right signal is consumer lag: messages waiting for the group. KEDA turns lag into replicas, with max replicas equal to the partition count.
ScaledObject resources.Measured: 40 PRs at once on a laptop cluster
- Strimzi runs Kafka (KRaft) from
Kafka,KafkaNodePoolandKafkaTopicresources. Prod overlay: 3 brokers, RF=3, min ISR 2, TLS Ingress. - Hardened pods: non-root, read-only root FS, all capabilities dropped, seccomp, default-deny NetworkPolicies, PodDisruptionBudgets.
- Observability: Prometheus metrics for bus outcomes (ok / retry / dlq), LLM tokens by kind including cache reads, tool calls, retrieval latency, critic rejections and webhook-to-review latency, with a provisioned Grafana dashboard. JSON logs carry
job_idacross services.
Seeded bugs, neutral PR titles, three context modes
A small storefront app with team conventions in CONTRIBUTING.md, plus 11 cases. Each is a find/replace edit under a neutral PR title, so the agent can't read the answer off the description.
| # | Seeded bug | Needs | Offline engine |
|---|---|---|---|
| 01 | Email lookup switched to an f-string SQL query | security | caught |
| 02 | Discount computed with floats | convention | missed |
| 03 | Authorization check removed from a handler | securityconvention | missed |
| 04 | Payment call loses its timeout | convention | caught |
| 05 | compute_tax gains a parameter; callers not updated | graph | missed |
| 06 | Refund failures swallowed | error handling | caught |
| 07 | PBKDF2 replaced with MD5 | security | caught |
| 08 | Pagination offset off by one page | logic | missed |
| 09 | average() divides by a possibly empty list | edge case | caught |
| 10 | Stock check inverted | logic | missed |
| 11 | Docstring-only change | negative control | clean |
The deterministic offline engine scores precision 1.0, recall 0.5: it finds the five pattern bugs and nothing else. The other five need reasoning, team conventions or the graph, which is exactly what the Claude ablation measures. make eval runs diff_only vs full_files vs graph_rag and reports recall, precision, false positives on the clean PR, tokens, estimated $ per review and latency.
Lean on the API's guarantees, not regex
Final answers are schema-constrained JSON and tool schemas are strict. Parsing never fails.
Each loop turn re-reads system prompt, tools and earlier turns from cache. Grafana tracks cache-hit share per agent.
claude-opus-5-5 by default; high effort for reviewer and security, medium for tests and critic.
Server-side fallback re-runs a declined request on a fallback model instead of failing the review.
Diff, code and docs are framed as untrusted data; repo names are validated before cloning.
A second, skeptical pass decides what ships, so four agents don't bury a PR in plausible noise.
Decisions I'd defend, and what they cost
- One consumer group per agent role gives fan-out without a router and lets each role scale on its own lag. A slow security agent doesn't hold back static analysis.
- The graph is built from the default branch. Symbols a PR adds aren't in it yet, so the impact report falls back to file-level seeds and agents read the PR head directly.
- Single-message processing (
max_poll_records=1) keeps commit semantics simple. Throughput comes from partitions × replicas, not in-process concurrency. - Graph blobs in Redis keep context replicas stateless with one cache-invalidation signal. For very large monorepos they'd move to object storage.
Next
- A "review memory": index resolved and dismissed comments so the critic learns from reviewer feedback.
- GitHub App auth and Check Runs instead of a personal token.
- OpenTelemetry tracing across Kafka hops.
- Per-repo agent configuration via
.graphreview.yml.