Ten AI Agents, One Reviewer: We Automated the Wrong Bottleneck

I don't want ten agents waiting for me

Ten agents writing code at once sounds productive. Ten pull requests waiting for the same person sounds like Monday.

I care about the second number.

If I have to reconstruct the intention behind every diff, check whether the agent solved the right problem, and explain the same conventions again, I haven't removed the work. I've moved it into review.

That is the AI code review bottleneck: we make producing a change cheap, then discover that understanding and accepting it still takes human attention.

My answer isn't to ban parallel agents. It's to design the work around the person who has to say, "Yes, this belongs in our product."

Smaller tasks. Explicit acceptance criteria. Diffs that tell one coherent story. And a measure of productivity that ends after review, not when the terminal stops scrolling.

What Superset's builders actually said

In Superset's Hacker News launch post, its builders describe running coding agents in parallel and the machinery that accumulates around them: worktrees, ports, terminal sessions, environment setup, diffs, tasks, and PRs.

Their point is more interesting than "we can launch lots of agents." They report that, with five or ten running, remembering what each agent is doing and performing human review often become the bottleneck.

Superset's response includes task and issue tracking, keeping the path from issue to agent to diff to PR to review connected.

I think they're identifying the right problem. Isolation helps an agent work without trampling another checkout. It doesn't help me remember which assumption changed three conversations ago.

This is a builder's observation, not a controlled productivity study. It doesn't establish that five agents are too many, or that ten are worse than two. Nor am I treating the title of this post as a report of my own ten-agent experiment.

What it gives us is a useful question: when we add another agent, which part of the delivery process actually gets faster?

A faster writer doesn't automatically mean a faster team

Imagine agents producing review-ready changes faster than a reviewer can accept them.

The queue grows. While those changes wait, the target branch moves. Requirements get clarified. A shared contract changes. Now some of the queued work needs another pass before the reviewer even opens it.

That's a simple capacity problem, not a model benchmark.

There's a second problem hidden inside it: an agent can be finished without the work being ready for review. "Implemented and tests pass" tells me very little if the tests encode the wrong requirement.

This is why I wouldn't use tokens generated, files changed, or PRs opened as the headline productivity metric. They measure activity upstream of the decision we actually need.

The research also deserves more care than a convenient headline.

In its July 2025 study of experienced open-source developers, METR randomly assigned 246 issues across 16 developers to allow or disallow AI. With the early-2025 tools in that setting, AI-allowed tasks took 19% longer, even though participants estimated that AI had made them faster.

But that page now explicitly marks those results as outdated. METR's February 2026 update says developers are likely getting more benefit, while warning that selection effects make its newer estimates unreliable. It also calls out difficulty measuring time when people use concurrent agents.

Neither study establishes an optimal agent count. The older experiment didn't measure this proposed workflow or prove that human review caused its slowdown.

My takeaway is narrower: feeling busy, feeling faster, and delivering faster are different things. We need to measure the last one.

Start with one behaviour I can accept

Here's an illustrative task for an ASP.NET Core API with an Angular frontend:

Improve invoice validation.

I wouldn't delegate that yet. It leaves the agent to decide what "improve" means, which rules matter, and how much of the application to touch.

I'd turn it into this:

Outcome
When creating an invoice, a due date earlier than its issue date
is rejected, and the user can correct it without losing their input.

Existing contract
Both fields are required date-only values.
Use the application's existing validation response and form patterns.

Acceptance criteria
- An earlier due date returns HTTP 400 using the existing
  field-validation error shape, with the error assigned to dueDate.
- The rejected request creates no invoice.
- A due date equal to the issue date remains valid.
- A later due date remains valid.
- The form displays the server error next to the due-date input,
  associates it accessibly with that input, and preserves all values.
- Correcting the date allows submission to succeed.

Scope
Create-invoice flow and its tests only.
No database migration, new package, shared validation framework,
or changes to invoice editing.

Evidence
API tests for earlier, equal, and later dates, including persistence.
A browser test for rejection, preserved values, correction, and success.
Actual check commands, results, and any checks not run.

Stop and report
If the current error contract cannot represent a field error,
or this requires changing shared infrastructure, report that conflict
before broadening the task.

Those are example product decisions, not universal invoice rules. In a real project, I'd confirm them with whoever owns the behaviour.

Notice the frontend requirement. A correct API that makes somebody re-enter an entire invoice is still a poor result. I want the acceptance criteria to describe the user's experience, not just the HTTP status.

I also want the agent to know when to stop. Otherwise a missing convention can turn a small validation fix into a new framework I never asked for.

The time spent writing this isn't free. It belongs in the cost of the change. The bet is that a few clear decisions up front remove repeated clarification and rework later.

Small means coherent, not an arbitrary line count

I wouldn't split that invoice task into "one agent writes the validator, another writes the response, another writes the UI."

Now three agents have to agree on a contract that one small change could have kept together.

For this example, I'd keep the API behaviour, its tests, and the form response in one reviewable slice, assuming the existing patterns make that practical. If discovery shows the slice is larger than expected, I'd redesign the split before producing a huge diff.

Google's engineering guidance on small changes makes a useful distinction: a small change should be self-contained, include its related tests, and leave the system working. It isn't just a low line count.

That's established review guidance, not evidence that this particular agent workflow produces a specific percentage improvement.

My practical test is whether I can describe the behaviour being changed in one sentence and inspect the evidence without following three unfinished PRs.

A mechanical rename across many files might be easy to review. A ten-line authorization change might need careful scrutiny. I'd separate unrelated formatting and cleanup, but never remove useful tests just to make the diff look smaller.

If there is a genuine dependency between changes, I make it explicit. A worktree isolates files; it doesn't make two conflicting API designs compatible.

The issue-to-PR workflow I'd start with

I don't need another orchestration platform to try this. An issue tracker, isolated workspaces, CI, and a PR template are enough.

I'd use this path:

Ready -> Implementing -> Checking -> Review-ready -> Accepted
                |           |             |
                +-----------+-------------+--> Blocked / Needs decision

Review feedback returns the change to Implementing.
Revised changes pass through Checking again before Review-ready.

Ready means the outcome, boundaries, acceptance criteria, and reviewer are known. Research tasks can have their own output, such as a recommendation, but I wouldn't pretend that recommendation is an implemented change.

Implementing means one agent owns the slice in an isolated workspace. The issue links to that workspace, the starting commit, and any dependencies. Local services need distinct ports and, where applicable, isolated test data. Separate directories aren't enough if every agent mutates the same database.

I give the agent the applicable repository instructions and a nearby example of the intended pattern. My guide to context files for AI coding tools covers the reusable conventions. The issue should carry the decisions specific to this task.

I also avoid attaching every document in the repository. The distinction between persistent rules and task-specific material matters; I've covered that in how AI coding context gets loaded. Here, I want the relevant contract and example to be easy to find.

Checking means the agent inspects its own diff, removes unrelated edits, and runs the project's required checks. For a bug, I want a regression test that demonstrates the original failure where practical, not only a new test that passes on the new implementation.

For visible behaviour, I want evidence through the browser. The invoice example needs to demonstrate that the user's input survives a rejected submission. My Playwright and .NET browser-testing guide covers tooling for that part.

If a required service is unavailable or a check fails, the agent reports it. "Couldn't run" is not a green result.

Review-ready means the PR includes a short handoff:

Outcome:
What user-visible behaviour changed?

Acceptance evidence:
For each criterion, identify its test or other verification.
Include the commands actually run and their results.

Review map:
Identify the main implementation, tests, and UI changes.

Boundaries:
What deliberately stayed unchanged?

Risks and gaps:
What remains uncertain or unverified?
What is the rollback approach?

Traceability:
Link the issue and identify the commit these results apply to.

That's an evidence index, not a substitute for reading the code. An agent's summary can be wrong too.

Accepted, for this workflow, means the responsible human approved the behaviour, required checks passed on the version being integrated, and the change merged. Another update to the diff invalidates stale evidence where relevant.

Deployment and user impact still matter. I'm defining a clear endpoint for this development measure, not claiming that merging a PR delivers value by itself.

Put a limit on unfinished work

For a pilot with one reviewer, I'd start with at most two changes in implementation or checking, and at most two waiting for review. Those are starting settings, not research-backed optimal numbers.

When the review queue fills, I stop starting new implementation work. Existing work can reach a safe stopping point; then attention goes to feedback, missing evidence, and acceptance.

Otherwise the "limit" just pushes the pile into another column.

I want the task record to show its current state, blocker, next action, last update, and the person expected to act. "Running" isn't useful if the process has been waiting for a decision all afternoon.

Notifications should mean something: ready for review, needs a decision, or failed a required check. I don't need an interruption every time an agent edits a file.

This is developer UX. A dashboard with ten animated spinners can look impressive while making the reviewer do all the mental bookkeeping.

More parallelism can still be useful. Independent, well-understood changes with available reviewers are good candidates. An agent researching an unfamiliar API while I inspect another diff can help. Several agents rewriting the same shared contract are a different proposition.

I'd also use an AI reviewer to identify suspicious code, missing edge cases, and mismatches against the acceptance criteria. But another agent agreeing with the first is not independent proof. The human still owns the acceptance decision.

Measure the time to an accepted change

I'd track two things separately: elapsed time and human effort.

The elapsed clock starts when we commit to doing the issue, before writing its acceptance criteria, and ends at acceptance. I also keep a timestamp for when the issue became Ready, so I can see specification time rather than hide it.

Human effort includes specifying, steering, reviewing, repairing, and integrating. Agent runtime and model cost get their own fields.

If three agents run during the same hour, that isn't three hours of human work. Nor should an hour spent on an unrelated task be charged as supervision just because an agent was running in the background.

My minimum scorecard would be:

For a first experiment, I'd record a baseline week, then try this workflow on comparable work the next week. I'd keep the reviewer arrangement and tools stable where practical and record differences in task complexity, risk, and scope.

I'd report counts and unfinished or abandoned tasks alongside the completed ones. Quietly dropping failures makes almost any workflow look good.

With a small sample, I'd inspect individual changes as well as the median. A two-week pilot is a local diagnostic, not a controlled study or proof that the workflow caused every difference.

And I wouldn't game throughput by splitting one useful outcome into ten trivial PRs. The unit is a comparable accepted behaviour change, with parent outcomes visible when work is split.

Where the 75% target belongs

My ambition is to make the human side of this work 75% easier. That is a design target, not a result I've measured for this workflow.

For an illustrative baseline of 80 human minutes per comparable accepted change, a 75% reduction would mean reaching 20 minutes. We'd have to count the upfront specification, steering, review, and repair in both numbers.

Then I'd check the guardrails: elapsed time must not get worse, and quality must not deteriorate. Saving reviewer time by accepting unexamined code doesn't count.

I would not promise that reduction. I would use the target to ask where the avoidable work is: repeated explanations, ambiguous scope, missing evidence, and reconstructing agent state.

Those are the things I want to remove. Not the judgement needed to accept a change.

Before I launch the next agent

I'd take one issue we actually need to ship and write down its acceptance criteria. I'd give one agent a coherent slice, require evidence, and record how much human work it takes to get that change accepted.

Then I'd increase concurrency only if the completed work justifies it.

The next bottleneck may genuinely be implementation. If so, more agents could help. But if review is already backed up, another agent is more likely to give me another thing to remember.

I want fewer changes waiting for me, not more agents waiting for instructions.