Our pull request reviewer started life as one HTTP call. Take the diff, send it to a model, ask what is wrong, post the answer as a review. Cheap, fast, and good enough to be useful.
Then a human reviewer found duplication the agent had missed. Someone had written a helper that already existed elsewhere in the repository, and the agent had nothing to say about it.
It could not have. The new helper and the old one share no tokens. A diff contains the changed lines and a little context around them, and the existing implementation is not in there. Asking a model to check for duplication while showing it only the changed lines is asking it to remember something it was never told.
That is a whole class of finding a diff only reviewer cannot produce. Duplicated helpers, a utility reinvented under a different name, a component that already exists three directories away. It is also one of the most common things a human reviewer catches.
So we rebuilt the reviewer on omp, our command line coding agent, and gave it the ability to read the repository before it judges.
What the reviewer is now
Four steps, run as a workflow on Argo.
- Post a pending commit status, so the pull request shows a review is running.
- Clone the repository at the pull request head.
- Run the agent against the diff.
- Post the findings as an inline review.
The agent step is where the work is. It runs omp headless with read, grep and glob available, hands it a prompt file, and collects the event stream it writes to stdout. The prompt tells it explicitly: for every non trivial function, helper, formatter, component or constant the diff adds, search this repository for an existing implementation of the same behaviour before judging it. When it finds one, report it as a reuse finding and name the file and line to call instead.
Two triggers feed it. One fires on pull request opened, synchronize and reopened, but skips drafts and skips the automated branch sync pull requests, because those only carry commits that were already reviewed. The other fires when someone comments /review, which is the manual path when you have addressed the previous round and want a fresh opinion.
Reasoning effort will eat the whole budget
The most expensive bug was not a crash. It was a review that found nothing and reported success.
The model defaults its reasoning effort to maximum. At that setting it spends the entire output budget thinking, and the answer comes back truncated. Measured on a real diff: the call hit its length limit at 16,000 tokens out of 16,000 with no findings in it at all.
The workflow then posted “no issues found”, which is the worst outcome available, because a clean report and a failed run look identical to the person reading it.
Two changes fixed it. Reasoning effort is now set explicitly rather than left to the model default, with the parameter carrying a comment saying it must never be left empty. And a review that does not complete is reported as a review that did not complete, with the reason attached, instead of being reported as clean.
That second change is the one I would keep in any reviewer. A tool that can mistake its own failure for a pass is worse than no tool, because it spends the reviewer’s attention building false confidence.
Knowing what it already said
A reviewer with no memory is worse than no reviewer.
The automatic trigger runs on every push, so a pull request with five pushes gets five reviews. With no state, four of them repeat findings that are still unfixed, and none of them notice the ones that were fixed. The repetition is not identical text either, so reviewdog’s exact match deduplication misses it and the author reads the same complaint reworded.
So the clone step gathers everything this agent has already said on this pull request before the review starts. It reads both the inline comments and the plain ones, because findings that cannot be attached to a line are posted as a normal comment instead. It tags a comment as outdated when the code beneath it has since changed, using the API giving it no current position as the signal.
The prompt then carries those earlier findings with rules: do not report a problem that is listed and still present, do not report a listed problem that the current code has fixed, and treat a reworded or moved version of a listed problem as not new. A comment marked outdated gets checked rather than assumed gone.
The whole thing degrades safely. If fetching the earlier comments fails, the review runs without them, because reviewing without memory is worse than reviewing with it and better than not reviewing.
There is one more detail worth copying. When a review finds nothing new, the posted comment says exactly that and names how many earlier findings are still open, rather than announcing that the pull request is clean.
Reading the agent’s output is its own small project
An agentic run emits a stream of events rather than one response. Turning that stream into findings means picking the right event out of a captured run of 5,768 lines. Two things make that harder than it sounds.
With tools enabled, the run emits many assistant messages, one per turn. The captured run had eight. The first seven carried no text at all, because they were the agent deciding which tool to call next. Only the last one, the one that ends by completing rather than by calling a tool, carries the answer. A parser that takes the first assistant message gets nothing.
And the answer is not always clean JSON, even when the prompt asks for JSON. The model sometimes writes a sentence first or wraps the object in a code fence. We hit both: one run’s answer did not begin with an opening brace, the JSON parse failed, and a finished 21 tool call review was thrown away as if the model had said nothing. The parser now scans for the first complete JSON object and ignores whatever surrounds it.
Both failures present identically. The agent produced nothing usable, the step reports zero findings, and the pull request looks clean.
So this piece lives in the image as its own file with its own tests against captured event streams, instead of being a few lines inlined into the workflow. Anything sitting between the model saying something useful and the reviewer seeing it deserves a test file, because a bug there is indistinguishable from the model having nothing to say.
Posting findings with reviewdog
Finding a problem is half the job. The other half is putting the comment where it belongs, and the API rejects an inline comment anchored to a line that is not part of the diff.
Rather than write that logic ourselves, the agent emits its findings as rdjson, reviewdog’s diagnostic format, and hands them to reviewdog with the pull request reporter. Reviewdog fetches the diff itself and decides where each finding can legitimately attach. That replaced a step that used to parse diff hunks and cross check line numbers by hand.
Reviewdog’s filter mode matters, and its default was wrong for us. The default only posts findings on added lines, but a real finding often sits on an unchanged line beside the change. The clearest case is a test that no longer covers a new shape: the test file was edited, the test line did not move, and the finding lands on a context line. Under the default that finding was dropped in silence, which again looks exactly like the review not running. Switching to the mode that accepts any line inside a diff hunk fixed it, and that is also everything the API can anchor a comment to.
One more failure mode is worth knowing. Even with the right filter, a finding can point outside every hunk, either because the model invented a line or because the API sent no patch for that file. Reviewdog drops those without a word. So after posting, the workflow works out which findings had no anchor and posts them together in a single plain comment. A finding should never disappear just because it could not be attached.
Two smaller things that cost time
The prompt goes to the agent as a file attachment, not as an argument. Linux caps a single argument at roughly 128 KB, so any pull request whose diff pushed the prompt past that failed instantly with an argument list too long error, long before the model was involved. The symptom was the same “no answer” message as every other failure, which sent us looking in the wrong place first.
And there is a live progress tailer. The agent writes its event stream to a file that the next step parses, so the workflow log showed nothing for the several minutes a review takes, and a running review was indistinguishable from a hung one. A background process follows that file and prints one short line per tool call and per turn, plus a heartbeat. It only reads the file, so a crash in the tailer cannot affect the review.
Concurrency, and why reviews queue per pull request
The automatic trigger fires on every push with no deduplication, and the manual comment trigger can land mid review. Without coordination, a push during a running review starts a second complete workflow: its own clone, its own diff, its own model call, its own posted review. The two race, you pay for both, and you can end up reviewing a commit that has already been superseded.
The fix is a lock keyed by pull request number, so a later run waits for the earlier one instead of running beside it. Different pull requests still review in parallel, because the key is per pull request rather than one global lock.
The detail worth knowing if you use the same tooling: the lock is declared as a list with one entry, not as a single object. The singular form broke the diff engine the deployment tool uses to compare desired and live state, which is a confusing failure to debug unless you already know it.
What it bought
The reviewer now finds things the single call version could not reach, which was the entire point. On the diff that started this, it found the duplication a human had found by hand, plus other findings anchored to specific lines. The run that proved it took 13 tool calls across 11 turns, in 4 to 5 minutes.
One practical note before you copy this. Installing the agent takes 13 to 16 minutes cold, against a review that runs in 4 to 5, so it has to be baked into an image rather than installed per run. That is a packaging problem, not a review problem, and it is the first thing you will hit.
The honest summary is that the model call is the easy part. Output budget, output parsing, state between runs and concurrent execution are the work, and none of them are about prompting.
And keep the diff as a hard filter. The agent can read the repository to check whether what you wrote already exists. It should not be writing comments about code the author did not change in this pull request. Tool access is there to ground a judgement, not to widen the scope of the review.