Orrin 3: runs that explain themselves Every run now ends with a plain account of what changed, why, and what it tested.
By Sam Hartley and Wen Zhao
Abstract
Orrin 3 ends every run with a written account: what it changed, why it believes the change is right, what it tested, and what it did not check. Across 2,112 beta runs on 38 teams, reviewers reached a decision in a median of 9 minutes instead of 14, and changed fewer pull requests.
1What a reviewer needs
When a pull request comes from a colleague, the reviewer can ask. When it comes from an agent, the diff has to answer the questions on its own: why this line, why not the obvious alternative, and how sure are you. In Orrin 2 the answers were in the run's history, spread across forty tool calls. Reviewers told us they skipped it and read the diff cold.
Orrin 3 writes the answers down. At the end of every run, before it asks for approval, it produces an account in four parts.
- What changed: the files and the behaviour, a sentence each.
- Why: the evidence that led to the change, with links to the trace, the commit and the failing test.
- What it tested: the commands it ran and their results, including the test it wrote to reproduce the bug.
- What it did not check: the paths it could not exercise, such as an API it had no sandbox credentials for.
2The last part matters most
The fourth part was the hardest to get right and has had the largest effect. An agent that reports only what went well invites a reviewer to trust it more than it deserves. An agent that names its blind spots tells the reviewer where to look. In the beta, 41% of accounts listed at least one thing Orrin did not check, and when a reviewer changed a pull request, they changed one of those things 3 times in 5.
| Orrin 2 | Orrin 3 | |
|---|---|---|
| Median minutes to a decision | 14 | 9 |
| Approved without a change | 66% | 73% |
| Changed in review | 24% | 17% |
| Closed | 10% | 10% |
3Written for the reviewer
Accounts are short by design: the median is 112 words. We tuned them against reviewers' edits, not against length, and every account is checked against the run's own record before it is shown. A claim about a test has to match a command Orrin ran, or the line is dropped.
account.md
What changed
taxFields() reads country and VAT ID from billingProfile.
Why
a41c9e2 moved tax details from customer to billingProfile at 13:58.
Since then customer.tax is undefined for every EU card.
What I tested
checkout/tax.test.ts: a new test, failed before the change, passes after.
The full suite: 214 passed.
What I did not check
Orders created before a41c9e2 still carry customer.tax.
No test covers them. Worth a look.4Available now
Orrin 3 is live on every plan. Accounts appear on the pull request, in Slack and in the run's history, and a project setting turns them off. The changelog lists everything else in the release.