Why Orrin never merges on its own The case for a person's hold on every change an agent makes.
By Ada Lindqvist
Abstract
Orrin can open a pull request but cannot merge one, on any plan, with any setting. Customers ask us to change that about once a week. This essay sets out why we have not, and what would have to be true for us to reconsider.
1The request
It usually comes from a team that has used Orrin for a few months and trusts it. The fixes are good, the tests pass, and approving feels like a formality. Why not let it merge the ones it is sure about? It is a fair question, and this is our answer.
2Approval is not a formality
Since March, people have changed or closed 37% of Orrin's pull requests. The edits were mostly small, but some caught real problems: a fix that handled the reported input and broke a sibling, a change that was correct and in the wrong place. A step that catches one change in three is not ceremony. It is the part of the system doing the most work.
- Approved as written63%
- Edited before merging20%
- Closed17%
3Confidence is not correctness
The obvious design is a threshold: merge when the agent is confident enough. We tried it offline on the 3,640 pull requests from the same months. At the threshold that would have merged half of them on its own, 1 in 14 of those was a change a person had in fact edited or closed. That is 130 changes nobody would have read.
| Threshold | Merged on its own | Of those, edited or closed by a person |
|---|---|---|
| 0.99 | 356 | 8 (2.2%) |
| 0.95 | 1,198 | 53 (4.4%) |
| 0.90 | 1,820 | 130 (7.1%) |
| 0.80 | 2,567 | 347 (13.5%) |
4A hold costs little
The case for merging alone is speed. But the median time from Orrin asking to a person deciding is 14 minutes, and most of that is the person finishing what they were doing. We are making it shorter still by bringing the approval to the thread where the error was reported. A hold costs seconds; an unread change in production can cost a night.
5What would change our mind
We would reconsider for a narrow class of change, such as a security patch whose diff touches only a lockfile, if three things were true: the class could be defined tightly enough to audit in a sentence, a year of production data showed people never changed those pull requests, and each customer opted in per repository. We are not there, and we will publish the data either way.
6Until then
Orrin opens the pull request, explains it and waits. Merging is a person's decision. It says so in our terms, and in the code: Orrin never asks GitHub or GitLab for permission to merge, so there is nothing to switch on.