Research engineer, evaluation. Build the measurements that tell us whether Orrin's fixes are right, from offline replays to production outcomes.

The role

Our evaluation team decides what better means. It ran the study of 4,212 pull requests that changed how Orrin reproduces bugs, and it owns the replays every model change has to pass before it ships.

You will design those measurements with Leah Okafor and the research team, and make them fast enough to run on every change.

What you'll do

• Build offline replays from real runs, with customers' permission and their code kept where it lives.
• Design metrics that predict what reviewers will do, and check them against what they did.
• Run studies like our fix survival note, and write them up for the research log.
• Tell product engineers when a change is worse than it looks, and show them why.

What we look for

• Three or more years writing production software in Python or TypeScript.
• Experience designing experiments or evaluations, in research or in industry.
• Care for the gap between a benchmark and the real thing.
• Clear writing: most of what this team makes is read by people outside it.

Good to have

• Published work on code models or program repair.
• Experience with large data pipelines.

How we hire

1. 01A call30 minutes with the hiring manager, about the role and what you want next.
2. 02The workA paid take-home or a pairing session, your choice. We pay £400 or $500 for your time.
3. 03The teamHalf a day with the people you would work with, in person or remote.
4. 04An offerReferences, then an offer, usually within a week of the half day.