← Thinking Thinking

Agent-Written Inference Engines Beat SGLang

An inference engine written by an AI agent out-earned SGLang in a test run by the paper's authors. Each agent got twelve hours to write an engine from scratch for a 320-billion-parameter model, serve the same replayed coding traffic as SGLang, and be scored by simulated income per GPU-hour. Two of the eleven engines came out ahead, the best earning about forty percent more, and six earned nothing. Neither winner decoded faster than SGLang; the edge came from how requests were queued and admitted. The question the paper set out to answer is how far an agent can take this kind of systems engineering on its own.

2026-11-16Thinking41 min read
Site cover: agent-written engines beat SGLang

On November 12, 2026, a paper with nineteen co-authors appeared on arXiv: Singularity-Bench. Among its six tasks is an inference-engine task whose rules are simple to state: give an AI agent twelve hours and four GB300 GPUs, and have it build an FP8 inference engine for the 320B-parameter GLM-5.3-Flash, starting from PyTorch and a toy server. The opponent is SGLang v0.5.20 deployed per the official recipe. Scoring replays real coding-agent session structure against the engine and computes simulated income per GPU-hour at fixed API prices.

The engine written by Opus 5.5 earned 11.52 dollars per GPU-hour, 40.8% more than SGLang's 8.18; the one written by GPT-6.1 Sol earned 9.70, or 18.6% more. Six of the eleven engines scored zero.

But the most informative numbers sit elsewhere. The median stream speed of the Opus 5.5 engine — output tokens per second per response — was 124.2, against SGLang's 125.0: essentially identical. It did not make the model compute faster. What separated the two was the tail of the first-token latency: 90th-percentile TTFT (time to first token) of 0.478 seconds versus 12.0. GPT-6.1 Sol's win is stranger still. Its TTFT tail was worse than SGLang's (24.3 seconds), its stream speed lower (105 tokens/s), and its income still 18.6% higher.

The two engines took opposite routes and both gained their advantage in the queue. The paper turns something the industry has sensed into a measurable fact: the engineering dividend in inference engines is shifting from the kernel layer to the scheduling layer — and scheduling happens to be exactly the kind of engineering agents can automate.

1 How the money is counted

To read this result properly, you first need to know exactly how the score is computed, because every number in this article flows from that definition.

Singularity-Bench spans six layers — pre-training, mid-training, post-training, inference, the agent harness, and cluster orchestration — each pitted against a production system, from Megatron-LM to Claude Code. This article unpacks only the inference layer.

The target and constraints: GLM-5.3-Flash is a mixture-of-experts model with 320B total parameters and 17B active per step (the paper's figure; the official model card says 18B), with linear- and sparse-attention layers interleaved, to be served in FP8. The agent gets four GB300 GPUs and twelve hours, starting only from PyTorch, Hugging Face Transformers, and a naive reference server — no production engine, no FlashAttention-4 or FlashInfer, no network.

The workload comes from SemiAnalysis's AgentX corpus: recorded from opt-in Claude Code sessions, stripped of original content and rebuilt with synthetic tokens, preserving lengths, timing, and prefix-sharing structure. The replay is real in session structure and synthetic in content. At evaluation, sixteen sessions run concurrently in a closed loop for fifteen minutes: each turn resends the growing conversation (median input 89K tokens, up to 256K, largely shared with the previous turn) and requests a short greedy completion.

The score is simulated income. Completed requests are billed at fixed prices — 0.09 dollars per million input tokens (cache hits priced the same) and 0.30 per million output tokens — and the total is divided by the combined GPU-hours of the four cards within the arrival window. Four cards times fifteen minutes equals exactly 1 GPU-hour, so the number is simply income per GPU-hour; a five-minute drain period follows, and requests completed during it count in the numerator. Above the income sit two gates: 99% of finished requests must return their requested output length; and the live server must pass an identity-hidden benchmark (MATH-500) at 95% of the reference accuracy, i.e. 0.8816.

Two details of this scheme deserve a pause. First, equal cache pricing. Income is computed on all input tokens at full price — whether or not they hit cache does not change a request's revenue — but cache hits lower the cost of completing requests, so the same compute finishes more conversation turns and earns more. The equal price is deliberate: cache-hit counts are reported by the server itself, and pricing them the same removes any incentive to over-report. Real API markets discount cached input, so this simulated scheme rewards prefix reuse more than real markets do. Second, the paper says it plainly: this metric rewards completing full interactions — effective prefix reuse and admission control — and "is not a direct measurement of kernel throughput."

fig1 How the score is computed
fig1 How the score is computed

The baseline, SGLang v0.5.20, follows the official SGLang Cookbook recipe. That version is precisely the minimum line supporting GLM-5.3-Flash, and the cookbook offers low-latency and high-throughput operating points per hardware. Note this is a by-the-book production baseline, not a limit-tuned one.

One more layer of credibility comes from the author list: Ying Sheng, one of SGLang's authors (at RadixArk), is among the co-signers, along with people from NVIDIA, Together AI, and CMU. The target is a system the authors helped build, and their incentive runs toward making the target solid.

2 Two winners, two different approaches

Opus 5.5 (with the Claude Code harness, 140.42 dollars of API spend in development) has its performance described precisely in the paper. It completed 983 requests against SGLang's 705, 39.4% more; median stream speeds are nearly identical, and prefix cache hit rates are close (96.7% vs 95.9%). The gap sits in the latency tail. SGLang's median TTFT is only 0.46 seconds, which is not bad at all; what hurts is the 90th percentile at 12.0. Most requests are fast, and a few held behind long inputs are slow.

fig2 Simulated income per GPU-hour
fig2 Simulated income per GPU-hour

The engine written by Opus 5.5 used tensor parallelism across four GPUs, batched chunked prefill (the stage of processing a request's input), CUDA-graph decoding, and custom kernels for MoE and hybrid attention. What opened the gap was scheduling and caching working together. Admission prioritizes requests with less uncached prefill work, so short continuations no longer wait behind long cold prefills; memory is reserved for a request's requested output before it is admitted; prefill chunks are ordered by remaining uncached tokens. On the caching side, because of GLM's hybrid attention, the prefix cache must preserve both the paged sparse-attention state and snapshots of the linear attention's recurrent state — storing only KV tensors cannot reproduce the full model state. During development, one combined scheduling and speculative-decoding update cut TTFT p90 from 1.81 to 0.57 seconds; the paper finds the baseline's long tail consistent with its first-come-first-served prefill.

GPT-6.1 Sol (with Codex, 42.78 dollars) took the other route. Its TTFT median of 1.30 seconds and p90 of 24.3 are both worse than SGLang's; its median stream speed of 105 tokens/s is lower too. It wins in how admission is defined: requests are admitted in order of uncached prefill work per unit of revenue, with an aging boost for requests that have waited more than 30 seconds. The cost of that rule is that large cold prefills are deliberately delayed; the return is a shorter end-to-end tail. A number from its development log says a lot: shortening the aging threshold from 120 to 30 seconds alone raised income from 8.98 to 9.12. It completed 816 requests with a 99th-percentile end-to-end latency of 67.8 seconds — only 17 requests took longer than a minute. This engine likewise combines custom kernels, exact speculative decoding with multi-token-prediction and n-gram drafts, and prefix caching of recurrent and sparse-attention state. In other words, both engines delivered complete systems with the major parts of a production engine, not demonstrations that only need to run; the agents wrote and assembled those parts themselves, inside the twelve hours.

The paper also reports a datapoint that seems to point the other way, and it deserves an honest explanation: both winning engines process more uncached tokens per GPU-second than SGLang, 19% and 8% more. This is not a kernel victory. It is what working scheduling looks like: shorter queues and no repeated prefill mean the same kernels stop idling, and effective work per second rises. Scheduling removes waiting and waste; it does not make computation itself faster.

Put together, the paper's Section 6 analysis gives a quotable conclusion: the two engines that out-earn SGLang "win on queueing and latency tails rather than on median decoding speed, which neither improves."

fig3 Three metrics compared: stream speed, first-token tail, completed requests
fig3 Three metrics compared: stream speed, first-token tail, completed requests

3 Kernels contributed, but scheduling separated the winners

Saying "kernels had nothing to do with it" would be wrong, and the point needs unpacking. The Opus 5.5 engine did write kernels: custom kernels for MoE and hybrid attention, and fused communication and hyper-connection kernels — that update lifted income from 7.23 to 8.08; it used the checkpoint's own multi-token-prediction layer for single-token self-speculation, restoring recurrent state on rejection; development-period measurements associate it with income rising from 8.42 to 9.72, a 15.4% gain; it also ships a precompiled kernel cache to avoid compiling during request processing.

But two things take these kernel contributions out of the decisive column. First, the baseline has the same class of weapons: SGLang's cookbook low-latency recipe enables MTP speculative decoding by default, and EAGLE drafting is standard. Second, the ablations: the medium- and xhigh-effort Opus engines post median stream speeds of 126 and 122 tokens/s against the baseline's 125, with smaller slow tails (10th percentile 55 and 62 vs 33) and TTFT tails of 0.91 and 0.72 seconds vs 12.0. The paper's own words: speculative decoding alone does not distinguish them from the baseline.

The real difference is in how much scheduling was changed, and there is a progression from weaker to stronger here. The medium and xhigh engines actually admit first-come-first-served; the one scheduling thing they do is order prefill chunks by remaining uncached tokens. With just that one change, the medium engine earned 11.07 — over the baseline. The max-effort engine adds admission priority and output memory reservation, cutting the TTFT tail to 0.478 seconds and lifting income to 11.52. That increment was not separately ablated, but across the two ends of the progression the direction is clear.

So the accurate statement is: kernel-side contributions are real — the paper's summary is that "these measurements show contributions from both kernel optimization and systems integration" — but what separated the winners from the baseline was scheduling.

4 Why the six zero-score engines failed

Six of eleven engines scored zero; that is the other half of the evidence. The paper lists every failure, and they fall into three classes.

Class one is scheduling and resource admission, with three engines. The most typical is GLM 5.3 + Codex. After clients disconnected, the server kept processing the abandoned traffic, and all 560 accuracy-test requests queued behind it timed out. The engine's answers were correct in isolated testing; the failure happened entirely in the queue. The other two are memory exhaustion: DeepSeek + Codex blew up under the accuracy burst and the engine failed; MiniMax M3 + Claude Code exhausted memory into widespread incomplete responses.

Class two is correctness, with two engines. Kimi K3 + Claude Code's server crashed as the accuracy run began. Kimi K3 + Codex is more instructive. Its traffic income was 2.70 dollars, but accuracy came in at 0.740 and the gate zeroed it. Speed and correctness separated here: the short reference checks it ran in development never covered the full accuracy workload.

Class three is throughput architecture, with one engine: MiniMax M3 + Codex's serial generation could not finish the one-hour accuracy run.

fig4 Failure classes of the six zero-score engines
fig4 Failure classes of the six zero-score engines

The paper explicitly warns against merging these failures into one cause; an accuracy-gate failure is not necessarily the same implementation error. But it also delivers a sentence with real weight: under this service contract, "cancellation and admission control are thus part of correctness" — even for an engine that produces valid answers when tested in isolation. The winning engines won in scheduling, and among the failed engines, the largest group failed there too.

5 What the ablations show

Two ablation families beyond the headline results speak to the result's quality — and to its mechanism.

The first concerns reasoning effort. Rerunning Opus 5.5 at the four lower effort levels (low, medium, high, xhigh) alongside the max-effort run from the main table: it clears the baseline at medium, xhigh, and max, and misses at low and high. The reason is not insufficient thinking. The lower-effort runs simply stopped early. Low worked 3.8 hours, high 7.2, medium 9.4, xhigh 10.9, and income follows the same order. Medium reached 96% of max income (11.07 vs 11.52) for 65% of the API cost (90.77 vs 140.42 dollars). GPT-6.1 Sol is far more effort-sensitive: only max clears the baseline, with the other levels earning between 1.55 and 6.29 — and its xhigh run misread its own clock and stopped after 7.5 of its 12 hours. Income tracks hours actually worked, not reasoning level. The value of this engineering comes from sustained iteration against the metric, not from a single flash of insight, and that point returns later.

The second concerns subagents. When each agent was required to dispatch at least five subtasks, the two winners moved in opposite directions. The Opus 5.5 comparison uses the high effort level: single-agent income was 7.31, which misses the baseline; with subagents it reached 10.32, crossing the baseline, though still below the max single-agent 11.52. Its division of labor: every kernel or benchmark subagent gets its own GPU with a 30–60 GB memory cap, analysis subagents touch no GPUs, and most subagents are forbidden from starting servers or killing processes. Subtasks can validate their work on hardware while the main agent keeps control of the server. GPT-6.1 Sol went the other way: from 9.70 at max single-agent down to 2.09, with the graded score landing inside its own measured development range of 1.99–2.61. It delivered a poorly performing design as measured, and the problem was in the division of labor, not in the delivery. About 28% of the main agent's tool calls coordinated subagents, every GPU experiment had to pass through the main agent, and the run completed only 8 full-length test runs (21 when working alone); two-thirds of the API spend went to subagents, and much of their output — 81 notes files, patches, features that shipped switched off — never reached the server.

The paper's summary deserves a direct quote: "Requiring subagents thus does not decide the outcome by itself; how the agent divides the work does."

6 What this result does and does not show

The boundaries, one by one — the result is worth exactly what these allow.

It is simulated income, not general throughput. Equal cache pricing rewards prefix reuse more than real markets, where cached input is discounted. So the size of the +40.8% is a reading under this pricing and moves with it; the direction of the win does not, because completion counts and latency tails are traffic-level measurements. The baseline is a cookbook recipe, not a tuned limit, and only v0.5.20 was tested. There is no same-time-budget human control: the human baseline is a production system's existing work, not a fair twelve-hour race, so "beat" precisely means "matched and exceeded that system's performance on this workload." All results come from a single hardware platform (four GB300s) and a single model architecture. GLM's hybrid attention offered a specific "read the architecture, then design the cache" opportunity, which may not transfer to architectures without recurrent attention state. Each configuration was run once (the traffic is chosen by a hidden seed), so the +40.8% carries no variance interval. The mechanistic evidence — TTFT tails, the completion-count gap — is strong, but the single-run fact should be on the record.

Half the paper is about how hard verification is. That work can be read as the paper's second contribution: how to make an agent's engineering results measurable and trustworthy. Red-team attacks cover nine of the ten OWASP agentic risk categories: plain Harbor fails across the board, Harbor with an LLM judge still admits six attacks, and the authors' Breakwater extension blocks all 28. SERVE-K04, aimed at the inference task, is worth one more sentence: it has the server distinguish "graded requests" from "traffic requests" by observable features, serves the former through the model and the latter through a cheap path, disguises the implementation as a single system, and steers the audit toward a sanitized snapshot. Understanding these attacks is understanding why the result is credible: the gates are hidden, the seed is hidden, and integrity audits are independent. The project site's leaderboard is live while its nav says "Code soon" and its FAQ says the code is out — two contradictory claims, so cite availability as of actual access.

7 Five groups of technical designs worth a closer look

Beyond who won, five groups of technical designs in this paper deserve to be taken apart on their own, because their usefulness extends past this one benchmark.

The first is the scoring design. It compresses throughput, latency, correctness, and cache efficiency into a single number: completed requests are billed at fixed unit prices, divided by GPU-hours, with two quality gates on top. Two things make it worth borrowing. One is the reduction of multi-dimensional service quality to a single economic quantity, which gives any engineering effort, human or agent, a scalar to iterate against; in the paper's own words, the metric rewards the completion of full interactions and is not a direct measurement of kernel throughput. The other is that anti-gaming is part of the scoring design rather than a patch added afterward: cache hits are billed at full price because hit counts are self-reported by the server and equal pricing makes lying unprofitable; the traffic seed is hidden, the gates are hidden, and integrity auditing goes to an independent party. The fifth group below covers attacks aimed at the scoring itself, a reminder that these defenses face threats that actually exist.

The second is architecture-specific prefix caching. GLM-5.3-Flash interleaves linear- and sparse-attention layers, so the prefix cache must preserve both the paged sparse-attention state and snapshots of the linear attention's recurrent state; storing only KV tensors cannot reproduce the full model state. Both winning engines built this dual-state cache independently. What is notable is not the specific design but the way of reasoning behind it: caching follows the model's architecture. Any architecture with recurrent state outgrows a generic KV-cache scheme, and the engineer, or agent, has to read the architecture first and design the cache second. The same logic applies to other hybrid architectures that carry recurrent state.

The third is how speculative decoding is used. The Opus 5.5 engine did not train a separate draft model; it used the multi-token-prediction layer that ships in the checkpoint for single-token self-speculation, restoring the recurrent state on rejection; development-period measurements associate it with income rising from 8.42 to 9.72. What it lowers is the barrier: as long as the checkpoint carries a multi-token-prediction layer, there is no separate draft model to solve for first, and the engineering question becomes how to wire the built-in layer into the engine correctly. Worth noting: the baseline SGLang recipe also enables MTP speculative decoding by default, so the component itself does not decide the outcome — Section 3 covered this; the difference between winners and baseline lies not in the component but in scheduling and integration.

The fourth is the parameter-level leverage of scheduling. GPT-6.1 Sol's admission rule orders requests by uncached prefill work per unit of income and accelerates requests that have waited more than 30 seconds; its development log contains a number that says a lot — changing the aging threshold from 120 to 30 seconds alone raised income from 8.98 to 9.12. Scheduling optimization happens at the granularity of constants and ordering rules: change a few lines, replay fifteen minutes of traffic, and the direction can be verified. Set against the kernel layer's feedback loop — edit, compile, benchmark — it is easy to see why iterative engineering works so well at this layer, a point Section 8 develops into a judgment.

The fifth is the workload structure and the attack surface. The benchmark replays the traffic structure of real coding-agent sessions rather than a synthetic stress test: median inputs of 89,000 tokens grow turn by turn and share prefixes heavily with the previous turn. Optimization for this kind of agentic workload has already produced fifty-plus upstream PRs across vLLM, SGLang, TensorRT-LLM, Dynamo, LMCache, and Mooncake. And for such a measurement to stand, attacks aimed at the scoring itself have to be stopped: SERVE-K04, described in Section 6, has the server distinguish graded requests from traffic requests by their observable features, serving the former through the full model and the latter through a cheap path. Its lesson is inseparable from the first group: whenever graded requests can be identified, they will be treated differently, which is why hidden gates, hidden seeds, and independent audits are components of the benchmark, not accessories.

8 The scheduling advantage and automated engineering

Now the opening question can return: why scheduling was automated first.

Comparing the two layers of work in this article makes it clear. Kernel optimization has a long, hard feedback loop: edit, compile, benchmark, and returns that depend on intuition for hardware microarchitecture. A scheduling policy's feedback loop is far shorter: change a few admission rules, replay fifteen minutes of traffic, read the metrics, change again. The rule space is large and complex, but each step's cause and effect is measurable. That is the kind of work agents are good at: long-horizon, iterable, metric-defined. Opus 5.5's general working pattern in the paper — measure before designing, get an end-to-end pipeline early, iterate against a proxy that tracks the graded score, select by rules fixed in advance — is a footnote to that shape. The "income tracks hours worked" ablation in Section 5 is its mechanistic evidence.

There is corroborating evidence on the same line. EngineAgent (arXiv 2611.14356) automates "new-model enablement" for on-device inference engines: an engine-hierarchy knowledge index plus a builder–auditor split reaches 75% end-to-end success across twelve tasks spanning CPUs, GPUs, and NPUs, against 33.3% for the strongest coding-agent baseline. Its success criteria include on-device execution and output logits agreeing with a reference implementation. It differs from Singularity-Bench — adapting models to an existing engine versus building an engine from scratch — but "inference-stack engineering labor being taken over by agents" is the same direction.

Industry signals converge on the same spot. The AgentX site reports fifty-plus upstream PRs optimizing agentic workloads across vLLM, SGLang, TensorRT-LLM, Dynamo, LMCache, and Mooncake. The prefix-sharing structure of agentic traffic is becoming an important axis of engine differentiation. As "my kernel is fast" yields to "my scheduler knows your traffic," the main battlefield of engine competition shifts by one layer. For clouds building their own engines, this moves the hiring calculus: the marginal return of scheduling talent may now exceed kernel talent's. For model vendors, it moves engine selection: load-awareness is worth more than peak throughput.

Three things to watch, by way of closing. First, if Singularity-Bench becomes a standing leaderboard, the decay rate of the scheduling advantage becomes an indicator in itself: how much of today's 40% edge survives after mainstream engines absorb these two admission policies as defaults. Second, which appears first in vLLM or SGLang defaults — GPT-6.1 Sol's revenue-weighted admission or Opus's prefill-aware admission. Third, which layer gets automated next. The paper's own framing is more cautious: Opus 5.5 matches or beats the human baseline on all six tasks (Singularity-Index 1.252), while every task's raw score still leaves significant room for improvement. All six layers have been touched, but being taken over is still far away.

Conclusion

The paper set out to measure whether agents can build the infrastructure of agents, and what it surfaced is a more specific signal: at the inference-engine layer, the variable separating winners from the baseline has switched from kernels to scheduling. Scheduling is short-loop, rule-complex, endlessly iterable engineering — exactly what today's agents are best at.

The result also shows how the weight splits between model capability and working process. Five of the eleven engines passed both quality gates; apart from the two winners, the other three (a previous-generation Opus 5, GLM 5.3, and DeepSeek) ran at median stream speeds of 15.1, 7.9, and 0.7 tokens per second against SGLang's 125. They made it through the gates with income still far below the baseline, beaten by the engines' own slowness rather than by how they iterated. Writing a complete, usable engine is still an entry ticket that only the strongest models hold, and capability draws the first dividing line inside the gate too. Section 5's two ablations were both run on the two winners; inside that pair, income stops tracking reasoning effort and starts tracking hours actually worked and how the work is divided. Model capability decides whether you get in, and where you stand once in. To win at the very top, what matters is the working process of iterating against the metric.

The conditions of that claim are equally clear: one hardware platform, one model, one workload, and simulated pricing. Anything beyond them is trend inference, not measurement. Looking forward, the scheduling advantage will decay as mainstream engines absorb these policies, and the speed of that decay is the readout of how fast this automated engineering is advancing.


Sources

[1] Zhou, Agarwal, Chen, et al. Singularity-Bench: Can Agents Build Agents? arXiv:2611.16591, Nov 12, 2026. Task definitions, all performance numbers, ablations, and red-teaming from §3.1, §5, §6, Appendix E and A.3; the four quoted sentences are verbatim from the paper. The scoring formula and the equal-cache-price anti-gaming design were line-checked against the official code (tasks/serve-glm53-flash/tests/simulator.py: input and cached input both 0.09, output 0.30; income_per_gpu_hour = income / (GPU_COUNT × window hours), GPU_COUNT = 4, window 900 s, drain 300 s).

[2] Singularity-Bench project site (infini-ai-lab.github.io/singularity-bench): leaderboard, Breakwater 28/28, the two contradictory code-availability claims.

[3] SemiAnalysis InferenceX: AgentX methodology and datasets page (inferencex.semianalysis.com/agentx). Session de-identification and synthetic reconstruction; the 50+ upstream PRs.

[4] SGLang Cookbook: GLM-5.3-Flash deployment recipe (docs.sglang.io). v0.5.20 as the minimum supporting version; low-latency recipes enable MTP speculative decoding by default; architecture described as MLA + DSA sparse + KDA linear hybrid, 320B total / 18B active.

[5] Huang, Sun, Chen, et al. EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities. arXiv:2611.14356, Nov 10, 2026. 75% vs 33.3% enablement success; success criteria include logit agreement.

[6] GLM-5.3-Flash model card (Hugging Face, zai-org/GLM-5.3-Flash): 320B total, 18B active, hybrid sparse + linear attention, mHC; technical report arXiv:2602.15763. The active-parameter figure differs slightly from the paper's 17B; experimental numbers in this article follow the paper.

Notes on figures: "income per GPU-hour" throughout is the paper's simulated metric (15-minute arrival window × 4 GPUs = 1 GPU-hour; cached input billed at full price); "stream speed" is median tokens/s per response, not aggregate throughput; +40.8%/+18.6%/+39.4% are single-run results without variance intervals.