Do Agent Teams Just Try Harder?

Prometheus benchmark pilot · research notes · 21–24 October 2026

I gave one LLM agent, or a team of three or ten, a fixed time to make code faster: first a C++ minimum-cost flow solver, then twelve PolyBench kernels and an AtCoder heuristic problem. Every result was graded on inputs the agents never saw.

TL;DR

Short answer: on flow, mostly yes, and a single agent can try just as hard. On two other tasks three agents did beat one, but only the task with many independent things to improve kept gaining up to ten agents.

Hidden-set scores, higher is better (PolyBench and flow: speedup over the starting code; AtCoder: score relative to a simple starting solution). Means of two or three 30-minute runs, with teams in SharedNet goal mode. Single-agent flow runs ranged from 1.06 to 1.47.

• Keeping agents working mattered more than adding agents. On flow, stopping at the first passing build gave 1.03–1.09× in every setup. Working until a fixed deadline raised the best runs to 1.35–1.41×, a single agent's included.
• Three agents beat one where there was enough to find. PolyBench rose by 0.63 and 0.76. On the AtCoder problem three agents added 0.019, about 190th to 60th place of 494 in contest terms. On flow, single runs varied so much that teams were no better on average, and the best flow run of all was a single agent (1.470).
• Ten agents beat three only on PolyBench (+0.42 and +0.24, neither significant on its own; one to ten agents gained 1.04, t = 5.3). On flow, ten agents averaged 0.05 more than three, well inside run-to-run noise, and no ten-agent run beat the best three-agent run (1.387 against 1.408); four different ten-agent designs all scored about 1.33 while spending 0.3M to 6.5M tokens. On the AtCoder problem ten scored slightly below three.
• Why: teammates search alike. Flow scores track how many of five code changes ("levers") a run kept (Spearman 0.91). Shared findings spread fast, so teams converge on the same changes: ten agents coding in private copies still kept the same levers (49 and 50 of 50 copy-by-lever indicators agreed), and behaved like one or two agents. Dedicated timing cores changed nothing, and a protocol built against herding (explore alone, split the work, merge) cut the chatter but scored only 1.16–1.24.
• Cost. Ten agents used 18–24 times the tokens of one, for gains of 1–27%.
• Rule of thumb. Before adding agents, make one agent keep working until the deadline. Go beyond three agents only for tasks with many separable improvements.

1. The task: a faster minimum-cost flow solver

Minimum-cost flow is one of the workhorse problems of operations research. A network has nodes that supply or demand a commodity and arcs that each have a capacity and a cost per unit. The job is to route all supply to all demand at the lowest total cost. Transportation and distribution planning, production scheduling, assignment and matching problems either are min-cost flow problems or call a min-cost flow solver inside a larger model. The problem has integral optimal solutions and a clean duality theory, so an answer is exact and checkable: a flow either respects every capacity and node balance and costs exactly the optimum, or it is wrong.

The practical method of choice is the network simplex algorithm. My editable baseline is PNS, a compact open-source sequential primal network simplex in about 1,000 lines of C++. It reads a DIMACS file, keeps a spanning-tree basis, and repeatedly brings an arc with negative reduced cost into the tree and drives another out, until no improving arc is left. Its hot loop has two parts: scanning blocks of about √m arcs for an entering arc, and updating node potentials along the tree after each pivot. The code already contains AVX2 and AVX-512 versions of the scan, switched off in the default build.

c  NETGEN flow network, seed 2107 (a visible instance)
p min 32768 262144          32,768 nodes, 262,144 arcs
n 1 1664                    node 1 supplies 1,664 units
n 32768 -682                node 32,768 demands 682 units
a 1 21353 0 1664 99         arc 1→21353: lower 0, capacity 1,664, cost 99
a 1 2272 0 803 19
…

PNS output: timing comments, one flow line per arc, then the optimal cost
c Init Time           : 20 ms
c # iterations        : 42762
c Time                : 56 ms
f 1 11238 0
…
s 16834392

This makes flow a good test of whether agents can do real engineering rather than pass a test suite:

• Correctness is exact. Every answer is checked as a feasible flow whose recomputed cost equals the optimum certified by LEMON, with its network simplex and cost-scaling solvers agreeing, plus an independent primal–dual certificate check.
• Speed is a continuous score. Wall time against the unmodified PNS binary on the same machine.
• Generalisation is testable. Agents tune on six visible instances; I grade on 24 hidden instances from the same generator (PyNETGEN) with different seeds.
• The speed levers are varied and independent. Input parsing, the pivot search, the potential update, instrumentation, output and build flags can each be improved separately.

The four instance sizes. Each hidden family has 6 instances; the visible set has 2 each of bands 2–4 and none of band 1.

2. Experimental setup

Agents and the four arms

Every agent is gpt-6-luna at medium reasoning effort, driven by the Codex CLI inside its own container with 1 CPU and 2 GiB of memory and web search off. The SharedNet CLI's goal runner creates a Room, starts the agents, and enforces the end rule. All arms start from the same seed workspace, image and task text. They differ only in how many agents there are and how messages wake them.

Protocol

During a run the agents can call a visible guard inside their container. It runs baseline and candidate alternately three times on the six visible instances, verifies every answer, and reports a speedup. Nothing about the hidden set is visible to them. At the end of the run I freeze the source, rebuild it offline with a locked GCC 14 builder, and grade it on the hidden set in one interleaved bundle with the baseline, while no agent is running. Every graded record is then replayed and re-verified.

Metric. For each instance I take the median of three attempts and compute baseline time ÷ candidate time. Within each band I take the geometric mean of those ratios, then the arithmetic mean over the four bands. I call this the hidden macro speedup; 1.00 means no change.

What I varied, and what I expected

1. H1, end rule. Under stop on pass, agents stop after the first change that clears the guard, so keeping going raises hidden speedup at the same time budget.
2. H2, only teams benefit. An early result suggested keeping going helps three-agent teams and not a single agent.
3. H3, teams beat single. At the same 30 minutes, three agents reach a higher hidden speedup than one.
4. H4, equal nominal compute. A single agent given three times the time (90 minutes, the agent-minutes of a team) matches the teams.

Eighteen episodes ran on 21 October 2026. Round 1 ran 08:16–08:50 UTC, the 30-minute run a 13:15–15:17, and the 15-minute block 15:53–17:27, all one episode at a time. To finish the same day, run b, the 90-minute single and an extra 15-minute single ran in three parallel lanes from 17:27 to 19:00. That put up to five agent containers on a 4-CPU host, so their in-container timing was noisier. All grading ran with no agent active, and all 1,512 hidden attempts verified.

3. Results

Figure 1. Hidden macro speedup by end rule, time budget and arm. Bars start at the baseline (1.00×). Every keep-going episode that used its full time beat all four stop-on-pass episodes, except single's run a.

Single's 15-minute bar is the extra full-length run. The first 15-minute single stopped working after two minutes and scored 1.054; it is kept in the data table and explained in section 5.5.

Two smaller patterns are worth stating, with little data behind them. Policy was highest in both 30-minute runs (1.396 and 1.408) and near the top at 15 minutes. More time helped less than not stopping: the team mean went from 1.25 at 15 minutes to 1.32 at 30. The rest of this post explains where these numbers come from.

4. Where the measured time goes

PNS prints its own parse time, solve time and pivot count, and the grader keeps every output. So each of the 1,512 hidden attempts can be split into parse, solve, and other (everything else in the measured wall time). For the baseline, an average instance takes about 662 ms: 53 ms parsing, 275 ms solving, and 334 ms of other time.

Figure 2. Mean time per hidden instance, split by phase. Agents cut parse and solve time a lot. The grey "other" block barely moves, and it is half of the baseline's time.

Means over 24 hidden instances of each instance's median attempt. Each episode is compared with the baseline graded in its own bundle (661–668 ms).

Most of the "other" time is not the solver. I timed the baseline on a band-4 visible instance inside the pilot image. Writing its 8.2 MB of output to a file took 764 ms in total. Sending the same output through Docker's attached stdout, which is how the grader captures it, took 1,350–1,564 ms, with parse (127 ms) and solve (549 ms) unchanged. The extra time is Docker streaming the output back to the host while the solver waits to write. It grows quickly with output size: about 29 ms per instance on band 1 and 680 ms on band 4.

This has three consequences:

• All hidden speedups are compressed. If parse and solve went to zero, band 4 could only reach about 1.9×. Only band 1, where "other" is about 29 ms, leaves real room.
• The comparison stays fair. Every arm pays the same overhead, and it was measured the same way for all of them.
• Visible and hidden scores measure different things. The visible guard writes output to a file without Docker, so output-formatting changes look large there and mostly disappear in the hidden grade.

Removing the overhead changes the ranking. Measured on parse plus solve only, with the same macro formula, the best builds came from three different arms:

5. Error analysis

5.1 Five levers explain the scores

Reading all 18 frozen diffs against the per-phase savings, almost every hidden gain traces to five changes. Their sizes below come from comparing episodes, not from ablations, and add roughly.

Figure 3. Which levers each frozen build kept, ranked by hidden speedup. The number of levers kept tracks the score closely (Spearman ρ = 0.92 over 18 episodes). Arm identity does not.

Kept levers are read from each frozen diff and confirmed in the hidden data: parse time for the parser, the pivot-count ratio for block factor 2, the patch that removes or samples the per-pivot clock reads for timers, and at least 15 ms less "other" time for output. "Partial output" means unsynchronised stdio without a buffered writer. "Tried and dropped" marks levers the logs show an agent tested and reverted. I identified the levers after seeing the results, so the correlation is descriptive.

Grouped by count, one lever gave 1.03–1.09, two gave 1.09–1.14, three gave 1.20–1.34, and four gave 1.29–1.41. Every arm produced at least one four-lever build. The arms differ in how reliably they put the full set together, and the reasons are specific.

5.2 Where policy won

In all four policy episodes the starter, codex-3, opened within about 20 seconds by asking codex-1 to profile and codex-2 to inspect the code, both with "no edits". That profile-first opening surfaced the parse cost early.

• Round 1 (1.033). The only change was the -DAVX2 -mavx2 flag, made at 1.9 minutes. At 4.9 minutes codex-3 asked everyone to post DONE, and the run ended at 7.7 of 15 minutes. The stop rule ended policy's run after one lever.
• 30 minutes, run a (1.396). A real three-way split. codex-2 implemented most of it: the SIMD reduction, timers sampled every 16 pivots, a one-hop prefetch in the potential update, unsynchronised stdio and a 64 KiB output buffer. codex-3 did the build flag and the parser; codex-1 profiled, tested and reviewed without editing. Concurrent rebuilds killed four guard runs between minutes 2 and 7. After that the team adopted a one-at-a-time rule for timing. About ten ideas were tried and rejected, including -march=native and block factor 2.
• 30 minutes, run b (1.408). A different process with the same result. codex-2 changed the Makefile twice without being asked, which killed a teammate's guard and briefly produced a segfaulting build. From minute 8, codex-3 did 17 of the 19 guard runs and all 20 patches. codex-1 and codex-2 sat idle for about 25 and 19 minutes; their messages were addressed to codex-3, who was in one long turn, so they woke no one.
• 15 minutes (1.302). AVX-512, a whole-file parser, timers removed and block factor 2 gave the best pure computation of any episode (1.69 on parse plus solve). It never reached the output path, so band 1 held it back.

Why policy was consistent. Both 30-minute runs ended on the same recipe: parser, cheap timers, SIMD scan, prefetch and buffered output. Their band-1 speedups are 1.896 and 1.904. On parse plus solve alone they are mid-table (1.54 and 1.51). Policy's lead on the official metric comes from being the only arm that reached the output lever in both 30-minute runs, and that lever pays almost only on band 1. I read the mention-only routing and single starter as helpful mainly because they kept agents from timing at the same time. That is an inference; I did not test it.

5.3 Where goal lost: three agents, one CPU budget, noisy decisions

Goal starts all three agents at once and broadcasts every message, and all three edited, built and timed against the same shared binary. Most of their timing runs overlapped a teammate's run: 15 of 26 in run a, 21 of 30 in run b, and 16 of 17 in the 15-minute run. An identical binary varied 2.5–5% when timed alone and up to 22% under contention; the unmodified seed scored 1.09 and 1.12 against itself. Rebuilding the shared binary also invalidated running guards in every episode.

• Run a (1.350) kept parser, compiled-out timers, AVX-512 and block factor 2, the best solve saving in the arm (87 ms) and 1.68 on parse plus solve. codex-3 worked off the shared binary, found the timer overhead from the solver's own phase counters, and re-tested block factor 2 after the team stopped timing in parallel. It lacked only a buffered writer.
• Run b (1.220) rejected block factor 2 on one contended single-repetition run, and codex-2 vetoed timer removal as "fake timing". It saved only 17 ms of solve time. It did have the best output change in the arm (30 ms).
• 15 minutes (1.141) reverted block factor 2 and AVX-512 plus factor 2 on single contended guards (1.304 and 1.329 against 1.38–1.41), although a private run had shown the factor-2 build cutting visible solve time by about 19%. It shipped exactly the two weakest levers.

Goal's agents found every lever that mattered. In two of three runs they then discarded the most valuable ones on measurements that the contention had made unreliable.

5.4 Where wake lost: one coordinator did all the work

In every wake episode the coordinator gave each helper one read-only profiling or review task in its first 30 seconds. The helpers answered within 1–2 minutes, made no source edits, and were never addressed again, so each sat idle for about 26–28 of 30 minutes. The coordinator made every edit and ran every guard inside one long turn: 234 tool calls in run a and 368 in run b, about 270 of them polling a running guard.

• Run a (1.367) kept parser, removed timers, AVX2 with -march=x86-64-v3 and loop unrolling, and the output buffer: four levers, close to policy.
• Run b (1.197) reached the parser only at minute 22, kept the per-pivot timers, built AVX2 into one function with runtime dispatch, and never touched output. About 9 minutes went to work it did not keep: a portability rewrite, an AVX-512 build that tied, a scalar-load experiment that failed correctness, and loop alignment. Of its 0.170 gap to run a, 0.136 is band 1 alone.
• 15 minutes (1.292) had the best band 2–4 result of any wake run and lost only on band 1, because it had no output change.

In practice wake was a single agent with two short reviews, and it varied like a single agent.

5.5 Where single's gains came from, and why its two 30-minute runs split

The two 30-minute singles had the same prompt, seed and model and scored 1.059 and 1.342. The difference traces to the first profiling command.

• Run a grepped the solver output for Time followed by sixteen spaces. That matches only the solve-time line, so c Init Time was filtered out from the start. Every later probe kept that solve-only view. It spent 15 of its 19 guard runs on block factors and compiler flags, choosing among variants whose differences (±0.03) sat inside its own same-binary spread (0.065). It saw a parse time once, at 7.4 minutes, while debugging an invalid unroll, and did not act on it. It kept only the AVX2 flag and stdio changes.
• Run b ran the guard on the untouched binary first and read the full output at 0.9 minutes: 173 ms parsing against 291 ms solving. It rewrote the parser at 3.1 minutes, added AVX-512 at 5.7 and removed the per-pivot timers at 7.8. Its last kept change was at 22.1 minutes. On parse plus solve it scored 1.65, third best of all 18 episodes.
• 90 minutes (1.349) had every change that mattered by minute 23: AVX-512 at 2.2, parser and a bulk output writer at 6.3, timer removal at 23.1. After about minute 25 its visible timing stayed flat. It gained 20 ms on output over run b and lost a little on parse and solve, so the two are effectively tied.
• The 2-minute 15-minute run (1.054) changed one Makefile flag, then ran pkill -f with a pattern that also matches its own command line, because the runner passes the whole prompt as an argument. The turn ended with exit code 0 and no completion event, which fits a self-kill, though I did not verify the signal. With no teammate to post a message, nothing woke it again. The extra run, single 15b, kept parser, timers and AVX2 and scored 1.237.

In every full-length single run, a guard was running for 61–82% of the wall time; editing took under 5%. Single did not lack effort. When it lost, it was anchored on the wrong first observation and had no teammate to interrupt it.

5.6 Visible scores overstate hidden ones

Figure 4. Visible guard score, hidden parse-plus-solve speedup and official hidden speedup. The visible guard overstated every keep-going episode, sometimes by more than half. Removing the Docker output overhead recovers part of the gap.

Visible scores are the agents' own guard runs on the exact frozen binary (median when repeated). Round 1 uses the host's visible check. Runs with heavy contention, such as goal, have noisier visible scores.

Four things widen the gap:

• the Docker output overhead described in section 4;
• output-formatting changes look large without Docker (wake run a gained about 0.33 visible from output alone and saved 16 ms hidden);
• band 1 is absent from the visible set;
• timing noise inside containers that shared a CPU with teammates.

Small pivot-count gains measured on six visible instances also did not carry over to the 24 hidden ones.

5.7 How the teams actually used three agents

Three of the six 30-minute team episodes had one agent doing essentially all of the implementing. Where all three were active, in goal, they competed for the same CPU and threw away good changes on unreliable measurements. Only policy's run a looks like real parallel division of labour, and policy's run b, with one active agent, scored about the same.

6. Scaling to ten agents

I then gave each team ten agents, 30 minutes and the keep-going rule. Policy and goal take any team size. The frozen wake mode allows exactly three seats, so I patched the runner on an experimental branch for two new variants. Ten ten-agent episodes ran on 22 October.

The VM was resized from 4 to 8 vCPUs, with the same CPU model, between the first six episodes and the last four; no run was interrupted. Baseline times on the new host are 3–7% shorter, but every speedup is measured against the baseline in its own grading bundle. The goal-10 rerun on the new host (1.330) falls between its two runs on the old one.

Figure 5. Hidden speedup against fresh tokens, every episode. Spending ten times the tokens on ten agents bought nothing: the best results sit at 0.3–1M tokens.

Fresh tokens are input plus output, excluding cached input, per episode; the x axis is logarithmic. Round 1 wake is missing because its tokens were not recorded. grow-10 is drawn as a ten-seat team although only its starter ever worked.

6.1 Ten agents never beat the best three-agent runs

The best ten-agent result is goal-10's first run at 1.387, below the three-agent policy runs (1.396, 1.408). Within goal mode, ten agents averaged 1.34 against three agents' 1.29, well inside the 0.28 by which repeats of one arm differed in section 3. On the 8-vCPU host, four different ten-agent designs landed within 0.008 of each other, at 1.329–1.337, which is what one good single agent reached (1.342, 1.349). Policy got worse at ten in both runs.

The five levers still explain the scores. policy-10's first run kept only the parser and the output change: it reverted a SIMD build at minute 27 because one guard scored 1.438 against 1.502, and that guard overlapped another agent's timing run. Its second run kept the parser and SIMD and nothing else. goal-10's first run and grow-10 both kept four levers.

6.2 Ten-agent teams collapse to one or two implementers

• policy-10. codex-6 wrote 23 of the 25 code hunks in one 29-minute turn. The other agents did read-only review; codex-1 alone was woken 127 times, mostly to read a message and post a review.
• wake-10. The coordinator gave nine helpers a read-only task each in the first 20 seconds. All nine answered within two minutes, several with the same profile, and none was given more work. The coordinator wrote every change itself from minute 2.4 to minute 30.
• grow-10. Given the right to recruit, the starter never did. It worked alone for 30 minutes, posted DONE twice (the runner answered with the time left), and scored 1.337 on 0.30M tokens, a twentieth of routing-10's spend.

6.3 Shared resources take over the conversation

When all ten agents edit and time one shared binary on one host, most messages are about that sharing. A message can count in several rows.

The agents built a mutual-exclusion protocol for the CPU out of chat messages, and maintaining it cost a third of the conversation and still let contaminated timings through.

6.4 Routing only works if the agents address each other

routing-10 wakes an agent only when a message @mentions it. The agents mostly wrote "codex-8, please …" without the @, which the router does not count, so 90% of messages still went to everyone and the run behaved like goal: 628 messages, 6.47M tokens, 1.333. In policy-10, 43% of messages carried an @mention, because the starter's prompt said to delegate with an exact @name.

6.5 Sharing findings spreads them fast, and herds the team

indep-10 removed the shared binary: every agent coded in its own copy and the Room carried only findings. Conversation fell to 87 messages and nobody negotiated CPU time. Findings spread within minutes: the timer finding, posted at minute 5.1, was in all ten copies by the end. I graded all ten private copies on the hidden set.

• The copies scored 1.237–1.354. The one promoted on its visible score (1.903, the highest) ranked seventh on the hidden set at 1.311; visible and hidden ranks correlated at ρ = 0.35.
• All ten converged on the same recipe by about minute 17, and none built an output writer, so even the best copy stopped at 1.354.
• The repeat run herded harder. At minute 22, most agents replaced their copy with codex-1's after one guard reported 1.882; seven of ten final copies were byte-identical. Run again, that binary measured 1.25–1.62 visible. No agent found the timer overhead, and the run scored 1.117.

Sharing without independent confirmation turns ten searches into one, aimed wherever the noisiest early measurement points.

7. A new protocol: explore, split, merge

The failures above are specific: herding onto one early recipe, levers nobody covered, timing runs that overlap, single noisy measurements copied by everyone, and picking one copy instead of combining several. I wrote a protocol that targets each of them using only generic coordination rules, with no hints about this task's levers.

1. Explore alone first. Each agent profiles its own copy and posts a ranked HOTSPOTS: list before reading anyone else's.
2. Split the work. codex-1, the integrator, posts a PLAN: giving each explorer a different area from the combined lists, at most two per area, covering as many areas as possible.
3. Measure through a queue. All timing goes through /workspace/bin/eval, which lets two evaluations run at once and makes the rest wait.
4. Report only replicated results. A FINDING: needs two runs that agree.
5. Merge, do not pick. The integrator applies every non-conflicting finding to the shared build and keeps it only if the combined build is not slower.

Results. It fixed what it targeted and exposed a new failure.

• What worked. Chatter and herding disappeared: 10–18 messages, no CPU negotiation, and every null result was reported as "do not merge" instead of being copied.
• A new failure: the plan followed the profiler. Both integrators split the work along the five solve phases the solver prints. The per-pivot timers (overhead spread across every phase), block factor 2 (a parameter) and output (outside the solve) belonged to no phase, so nobody owned them. merge-10's seven other explorers reported only null or negative micro-optimisations.
• The queue became the bottleneck. With two slots for ten agents, an evaluation waited 3.4 minutes on average and up to 12; explorers completed 2–7 evaluations each.

merge v2 asks the hotspot lists and the plan to cover the whole wall time, including time the breakdown does not explain and build settings, rotates explorers after a result or eight minutes, and widens the queue to four slots with one-repetition screens. With ten agents it scored 1.244, up from 1.157. The whole-wall-time rule recovered the per-pivot timer lever that both v1 runs missed, but output and block factor 2 were still not kept, so it landed where the lever model puts three levers.

8. A scaling model

The flow results suggest one bottleneck model. A team's score is set by how many of the task's levers it finds, verifies and integrates before the deadline, and the narrowest of those three stages sets the count:

levers kept K ≈ min( D, V, M ),   score ≈ g(K)

D  discovery    = Σᵢ [1 − (1 − pᵢ)^N_eff]
                  N_eff = N / (1 + (N − 1)·h)     h: how alike teammates search
                  pᵢ measured over T_eff = T − time spent coordinating
V  verification = C·T / (m·τ)
                  m ≈ (z·σ / δᵢ)²                 runs needed to see lever i through timing noise
M  integration  = merges per minute × T           union of everyone's findings
                  picking one copy keeps the best copy's levers, not the union

Here pᵢ is the chance one agent finds and keeps lever i in the episode, N is the team size, h is the correlation between teammates' searches (0 for fully independent, 1 for identical), C is the number of clean timing slots, τ is minutes per timing run, σ is the timing noise and δᵢ is the lever's effect. Two familiar intuitions are the two ends of the discovery term. While pᵢ is small, D ≈ N·Σpᵢ, so the team scales with "agents × search speed". Once pᵢ is large, D approaches L, the number of levers the task has. The verification term adds a square law: a lever half as large needs four times as many runs to confirm, so small levers are the first to go when timing is scarce. Each term can be measured from the logs and each makes a prediction. The table checks them against the graded flow episodes with one, three, five and ten agents.

Figure 6. Lever keep rate: ideal independent search against observed teams. Drawn with pᵢ from the first four singles, three agents match the ideal curve and ten fall short on the levers that are hardest to see. A fifth single raises the ideal curve above both (see the table).

Ideal rates are 1 − (1 − p₁)^N with p₁ from four full-length single-agent episodes; observed rates are from the seven three-agent and eleven ten-agent 30-minute episodes. Block factor 2 was never kept by a single agent, so the ideal curve predicts zero for it; teams found it 29% (N = 3) and 18% (N = 10) of the time, the one place where a team beat independent search.

Read together, two things stack up beyond three agents on this task. Flow has only five levers, so even ideal independent search gains less than one lever from three agents to ten. And sharing findings drives h toward 1, so ten agents search like one or two and all miss the same hard levers, even in private copies. Meanwhile coordination grows with N². The same model says when more agents should help: a task with many levers that are each hard to find (low pᵢ, large L), low h, and integration that takes the union of findings. Section 9 tests that.

8.1 Experiments that test one term at a time

9. Do the findings generalise? Two more tasks

I picked two tasks that move the model's parameters away from flow and ran them under the same protocol: Codex gpt-6-luna in SharedNet goal mode, a fixed 30 minutes with no early finish, and hidden-set grading only while no other experiment ran on the host. Each cell was repeated two or three times.

• PolyBench/C 4.2.1. Twelve numerical kernels (syr2k, trmm, 2mm, lu, cholesky, doitgen, covariance, jacobi-2d, heat-3d, seidel-2d, floyd-warshall, nussinov) to speed up on one core under a fixed gcc -O3 -march=native, with no optimisation pragmas, OpenMP or precision changes. Many independent levers. The same code came in two layouts: one file per kernel (separable) or all kernels in one file (coupled). Each kernel was graded on two sizes the agents never saw, outputs checked against the reference to 1e-9; the score is the geometric mean speedup.
• AtCoder Heuristic Contest 046 (from ALE-Bench). One C++ file, a 2-second time-bound heuristic, scored on the contest's 150 private seeds as the mean score relative to a simple starting solution. Ideas interact inside one file and every measurement is noisy. On this scale the contest winner would score about 1.32.
• Flow with a clean timing service. The flow task again, with seats confined to six cores and a host service timing candidates on the other two.

Hidden-set scores; individual episodes in parentheses. Fresh tokens are cell means.

• Three agents beat one on both new tasks, so H3, not supported on flow (section 3), holds here. PolyBench rose by 0.63 (separable, Welch t = 2.4) and 0.76 (coupled). On the AtCoder problem three agents added 0.019, which moves the average contest-equivalent rank from about 190th to about 60th of 494 (t = 2.2).
• Ten beat three only on PolyBench. The gain was 0.42 in the separable layout and 0.24 in the coupled one; with three episodes per cell neither is significant on its own (t ≈ 1.3), but from one agent to ten the separable layout gained 1.04 (t = 5.3), the clearest team effect in this study. On the AtCoder problem ten agents scored 0.004 below three. On flow, ten agents gained 0.04–0.05 over three in both timing setups, inside run-to-run noise.
• Where ten agents gained. They did not touch more kernels: nine or ten kernels reached 2× in almost every run, whatever the team size. They went deeper on a few. In the separable layout 2mm rose from 5.6× (one agent) to 7.9× (ten), covariance from 9.3× to 15.0× and seidel-2d from 1.5× to 3.7×. Each kernel has several independent levers, so the task offers far more than three agents can cover in 30 minutes.
• The layout mattered less than predicted. One file per kernel was better at every team size, including a single agent (4.25 against 3.68), so most of the layout effect is about how easy the work is to navigate, not about merging. Its larger gain from three to ten (0.42 against 0.24) points the way the integration term predicts but is within noise.
• Clean timing did not change flow. 1.28 and 1.32 against 1.29 and 1.34 in plain goal mode: verification capacity is not what limits flow teams.
• Cost. Ten agents used 18–24 times the fresh tokens of one (roughly N^1.3), for gains of 1–27%.

9.1 Does the bottleneck model still hold?

In direction, yes; in its original three-stage form, no. Term by term:

What remains is a one-term discovery model, K ≈ Σᵢ [1 − (1 − pᵢ)^N_eff] with N_eff = N / (1 + (N − 1)·h). The number of independent levers L decides whether adding agents can help, which correctly ordered the three tasks. h is high as soon as findings are shared, which is why even PolyBench gains little from three to ten. Verification and integration bind only in special setups, such as a two-slot timing queue or a plan split along the wrong dimension. Two things the model misses: single-agent results vary widely (1.06–1.47 on flow, 3.22–4.14 on coupled PolyBench), so much of a team's value is avoiding a bad run, which needs many singles to estimate pᵢ well; and on PolyBench levers are not binary, since ten agents won by going deeper on the same kernels. The model so far has only been tested in direction. The next experiments control L and h directly.

10. So, do teams just try harder?

On flow, mostly yes, and a single agent can try just as hard. What moved hidden speedup was finding and keeping the five levers, and the biggest obstacle to that was stopping early. Under stop on pass, every arm stopped after one or two levers. Under keep going, every arm reached four levers at least once, including single, and the best flow run of all was a single agent that kept all five (1.470).

Three agents did not reliably add parallel work. Helpers went idle, agents disturbed each other's measurements, and at 30 minutes most team runs spent 1.7–4.2 times single's tokens (wake's run a spent slightly less) for scores in the same range. What a team did add, sometimes, was an interruption. Policy's profile-first opening exposed the parse cost early, and in goal a teammate's phase counters revealed the timer overhead. Single run a lost because nothing interrupted a bad first observation. That suggests a cheaper intervention than a team: periodically make one agent step back and re-profile.

Ten agents made this clearer. Every design either collapsed to one implementer (policy, wake, grow), spent a third of its messages sharing one CPU and one binary (goal, routing), or herded onto one recipe in private copies (indep). On the resized host the same score of about 1.33 cost anywhere from 0.30M to 6.5M fresh tokens. The protocol built against these failures (section 7) removed the chatter and the herding and still scored lower (1.16–1.24). On flow the extra agents added coordination, not search.

The band-level arithmetic puts flow's ceiling near 1.47–1.48, for example goal run a's solve changes combined with run b's output writer. The five-lever single run reached 1.470, so on flow one agent can get there alone.

On the other tasks, it depends on how much there is to find. Three agents beat one on PolyBench and on the AtCoder problem. Ten beat three only on PolyBench, where each kernel has several independent levers and ten agents went deeper on the same kernels. Even there, three to ten gained less than one to three, because teammates who share findings search alike from the start (section 9.1).

Rule of thumb. Before adding agents, make one agent keep working until the deadline. Go beyond three only for tasks with many separable improvements, and expect token cost to grow roughly as N^1.3 while the gains shrink.

11. Limits

• Sample size. One or two episodes per arm and condition on flow, two or three per cell on the new tasks. Enough to reject "only teams benefit" and to see the 1-to-3 gain, not enough to rank coordination designs or to establish the 3-to-10 gain on PolyBench.
• Runner limit at ten agents. In two ten-agent episodes (flow with clean timing, the first AtCoder run) the runner stopped one or two agents mid-episode because the room prompt it passes as a command-line argument grew past the operating-system limit (E2BIG).
• One crashed episode. A three-agent clean-timing flow episode stopped at minute 16 when the shared data disk filled; it is excluded and was re-run.
• Contamination. AHC046 ran in April 2025, probably before the model's training cutoff, and its editorials are public. Its scores are also compressed near the top, so small differences matter.
• Shared host. The new tasks' episodes alternated with another autonomous research loop and other users' jobs. Every grading ran with no other launcher or grader active, but the agents' own timing during episodes could see load.
• Run conditions. Mixed: run b and the 90-minute single ran in parallel lanes with more contention than the sequential runs. All grading was sequential and uncontended.
• The grader's own overhead. The hidden metric includes Docker output streaming, about half of baseline time. It is equal across arms but compresses every ratio and favours output changes on small instances.
• Post-hoc levers. The levers and their sizes were identified after seeing the results, from diffs and timing data rather than controlled ablations.
• Fixed seed for policy. The policy seed was the same each time, so codex-3 always started; I cannot separate the policy from that starter.
• Harness asymmetry. A single agent whose turn ends without DONE is never woken again, which a teammate's message would prevent in a team. One single episode lost 13 minutes this way.
• Host change. The last four ten-agent episodes ran on a host resized from 4 to 8 vCPUs. A calibration rerun suggests this did not change the outcome, but it is one run.
• Experimental runner. grow-10 and wake-10 used an experimental branch of the SharedNet CLI (exp/prometheus-scaling, 7d50ef1, all 507 tests passing). grow pre-provisions all ten containers and activates one per RECRUIT: message.

12. What I would run next

1. More repetitions where it matters. Five or more episodes for PolyBench at three and ten agents, to settle the 3-to-10 gain and the layout interaction.
2. Vary the number of levers directly. PolyBench with 4, 8 and 16 kernels at one, three and ten agents: the model predicts the team size where gains stop grows with the number of kernels.
3. Measure herding on the new tasks. Compare teammates' kernel-level changes the way section 6.5 compared flow copies.
4. Fix the measurement. Write solver output to a file inside the sandbox and report parse-plus-solve speedup next to end-to-end speedup, so the flow grader stops compressing gains.
5. Interrupt the single agent. Single plus a re-profile prompt every 5 minutes, against single and against a team, to test whether teams help mainly by interrupting each other.

Data and reproduction

Harness: prometheus-benchmarks, branch feat/benchmark-harness (grading at commits 6d08de0, e7374f6, d3224d6; report f4d184e). Episodes: runs/prombench-methods-20261021 and runs/prombench-methods-best-20261021. Hidden bundles: runs/prombench-hidden-flow-grading/…6ce6530d, runs/prombench-hidden-flow-best-grading-…/a2f51137, runs/prombench-hidden-flow-final-grading-…/1b46fdb0; all fully_reverified. Ten-agent bundles: prombench-hidden-flow-{ten,tenrep,indep,indeppriv,indeprep,routing10,grow10,wake10,goal10host8,merge10,merge3,merge2x10,policy5,goal5,single3,clean10,clean3b,clean10b,clean3c}-grading-*. Generality round (section 9): runs/prombench-generality-20261022 (episodes, PolyBench and AHC046 grades, analysis/table.json), code in experiments/2026-10-22-generality (commits fad2c3d, bdbc237, 28d8ca8), all fully_reverified. Model gpt-6-luna (medium) via Codex; SharedNet CLI goal runner at d5c245c, experimental runner at 7d50ef1. Notes for every run: runs/prombench-methods-best-20261021/NOTES.md.