How often does an agent's fix survive review? What 4,000 real incidents taught us about patches that merge and patches that don't.

1Introduction

Benchmarks tell you whether an agent can produce a patch that passes a test. They do not tell you whether an engineer, reading that patch on a busy afternoon, will merge it. The second question decides whether an agent saves time or creates review work, and it can only be answered in production.

This note reports what happened to every pull request Orrin opened over five months, for teams that agreed to share outcomes: how we counted, what we found, and what we changed as a result.

2Method

The sample is 4,212 pull requests from 61 teams, opened between March 3 and July 31, 2026, each followed for 14 days. A pull request merged unchanged when the merged diff matched Orrin's exactly, ignoring formatting; merged with edits when a person pushed to the branch first; and closed when it was closed without merging or still open after 14 days.

We left out runs on repositories with fewer than 50 tests, where reviewers told us they treat every pull request as a draft, and runs a person cancelled before approval. Both choices raise the numbers, so the excluded runs are in the notes.

3Results

Size mattered less than we expected. Fixes of one to five lines merged unchanged 67% of the time; fixes over 30 lines, 52%. The stronger signal was the reproduction step. When Orrin wrote a failing test that reproduced the bug before changing any code, 74% of its pull requests merged unchanged. When it could not, because the bug depended on data or timing it could not recreate, the share fell to 38%.

The kind of bug mattered too. Type and null errors, the most common, were fixed well. Concurrency bugs were not, and reviewers were right to be wary of them.

4Why fixes were edited

We read all 800 pull requests that merged after edits. Most edits were small, and most were about fit rather than correctness.

• Naming and style, 41%: the fix was right but did not read like the code around it.
• A missing case, 27%: the fix handled the reported input but not a sibling one.
• Scope, 19%: the reviewer wanted a smaller change, or a larger one that removed the cause.
• Other, 13%.

5What we changed

Two changes followed. Orrin now tries harder to reproduce: it may spend up to a third of a run on the failing test before it touches the code, and it says plainly when it cannot. And when it cannot reproduce a concurrency bug, it no longer proposes a fix; it writes up what it found and hands it to a person. Since both shipped, the share merged unchanged over the last 90 days has risen to 71%.

6Limits

The teams in this sample chose to use Orrin and to share their outcomes, and are likely more willing than most to merge an agent's work. A merged fix is not always a correct one: 9 merged pull requests were later reverted, all in the logic error group. We will report reverts on their own in the next note.