Review on an AI-Native Team: A Loop, Not a Gate

Last post ended on a fair worry: if we let owners ship, including people who can't fully read their own diffs, how does the team not drown in pull requests? This is the answer, and it's the part of our setup I'm most sure generalizes.

Start with the old reality, because it's everyone's. Review was a scarce human resource. Our p75 time-to-first-review was around 21 hours. A quarter of PRs sat most of a day before anyone even looked. And the feedback, when it came, landed after the code was written, the worst possible time: the author has moved on, the context is cold, and half the comments are things a machine could have caught. Reviewer availability gated the whole pipeline. More authors would have snapped it in half.

So we stopped treating review as one human event at the end and spread it across the life of a change.

Most of it happens before a human looks. By the time one of us opens a PR, a stack of checks has already run: the mechanical stuff (does it build, do the tests pass, does it follow our conventions) plus a set of agent reviewers that read the diff for the problems we used to catch by eye. The human reviewer isn't the first line of defense anymore. They're the last, and they get a much cleaner diff. This isn't just an internal habit. The same pre-PR process is built into our product through dedicated agents and validation, so a change arrives already checked instead of waiting to be.

The PR shows up pre-labeled with what it is. Those size, importance, and conflict-risk labels from the alignment post get attached at creation, computed from the actual diff, not the author's optimistic description. That's the quiet backbone of the whole thing. The same /whats-next queue that tells a person what to build also tells reviewers what to review. It reads the labels and routes the riskiest, most important, most conflict-prone PRs to the front, so work gets routed without anyone routing it. The triage meeting we used to hold to establish those facts is gone. The facts are on the PR.

Some of us get reviewed by an agent in our own voice. This is the weird one. We built reviewer personas from each senior engineer's own past review comments: the things they always flag, in the way they'd flag them. Before a PR reaches a human, the relevant personas pre-flight it, catching the predictable comments early so the human review is about judgment, not nits.

We automated the per-PR checks, then stopped treating PR-level QA as the final safety net. Too many features moving to gate each one with manual QA at the PR. The automated checks catch what they can on the way in. The real safety net is downstream: everything merges to a pre-dev branch, and we run group checks across the whole batch, agents fanning out over the set of changes to find risk and interactions no single-PR review would catch. Only then does it promote: pre-dev to dev to main. Review became something that happens to the integration, not just the individual change.

And it doesn't stop at merge. The piece I'd least want to give up. After a change ships, agents go back and check whether it actually does in production what it was supposed to, comparing intended behavior against real traces over the next few days. When intended and actual diverge, that becomes a tracked follow-up that comes back into the work queue. Review stopped being a gate you pass once. It became a loop that closes in production, where the truth actually lives.

None of that works unless your telemetry is reachable by the agents, and this is the part I'd push hardest on if you're setting it up: make your logs agent-queryable. Ours go into Logfire in a form an agent can query directly, so it can ask "is this new code path firing, and is it doing what we said?" and answer from real traffic, no human exporting a dashboard. The moment your observability is something an agent can interrogate instead of something a person has to go read, the feedback loop starts closing itself. The system notices its own regressions and files the follow-up. If your logs are only human-readable, you don't have this loop. You have a dashboard nobody checks.

The principle: humans set the policy, the system enforces it continuously. We decide what good looks like, what's risky, what each persona cares about. Then that judgment runs on every PR, all the time, no waiting for a free afternoon.

Now the honest part.

The personas can manufacture false confidence. Early on, a clean persona pass made people feel a PR was reviewed when it had only been pre-flighted. We had to be explicit: the agent catches the predictable problems, it does not replace a human deciding whether the change is a good idea. The moment a team treats the automated pass as the review, quality quietly drops and everyone feels faster.

Being reviewed by a bot in your own voice is strange, and not everyone loved it. A couple of engineers found it uncanny, or felt it flattened their judgment into a checklist. The framing that helped: it's a draft of your review that you can override, not a verdict issued in your name.

The post-merge checks are only as good as our definition of "intended." When we're vague about what a change was supposed to do, the agent can't tell drift from design. The discipline that makes the loop work is upstream, writing down what we expect before we ship, which loops right back to the alignment post.

That last point is the thread through everything so far. The labels, the personas, the standards, the definition of "good", none of it works if it lives in people's heads, because heads forget and people leave. It works because it lives somewhere durable and shared. That's the next post: where our institutional knowledge actually lives now.