← Thinking Thinking

SGLang Summit: The Technical Roadmap and Future of Inference Engines

The first SGLang Summit disclosed a concrete technical roadmap: v1.0 bets the engine's architectural center on agent workloads — multi-type state and histories that outgrow GPU memory push the engine's core capability from throughput to state management, with hierarchical caching as the answer: median TTFT down 60% in a multi-turn benchmark, and iFlytek's production cache hit rate up from 30% to 95%. One engine stack now spans four computing platforms, RL weight-update pauses drop from 53 to 7.2 seconds, and most merged PRs are already AI-produced. The competition has changed axes: state, heterogeneity, and AI-written engines.

2026-11-18Thinking55 min read
Cover: The Technical Roadmap and Future of Inference Engines

Inference is taking over the bulk of AI compute. Deloitte projects that inference will account for roughly two-thirds of all AI compute in 2026, up from about half in 2025. What determines how fast and how cheaply that compute runs is the inference engine.

On November 12–13 at Fort Mason in San Francisco, more than two thousand people attended the first SGLang Summit. NVIDIA, AMD, Intel, Google, and Meta shared the opening stage; AMD CEO Lisa Su delivered a recorded message, AWS released its Trainium backend the same week, and more than ninety technical sessions filled two days. The inaugural summit of an open-source inference engine had pulled nearly every computing-platform camp to the same table.

The depth of the bets differs: NVIDIA, AMD, Google, and Meta are contributing at production or day-0 level, Intel announced a collaboration, and the Trainium backend is still in public preview. But the direction is uniform: no one is treating a proprietary engine as a moat anymore; every camp is adding weight to the same common stack. By the organizers' account, the engine now runs on hundreds of thousands of GPUs, with deployments at Meta, Microsoft, Google, Amazon, Thinking Machines Lab, Cloudflare, Reflection, and Modal.

The shared stage is the surface; the real substance of this summit lies in the technical roadmap it disclosed: v1.0 bets the engine's architectural center on agent workloads, the RL post-training framework Miles converges with the serving stack at the engine layer, and most of the engine's merged PRs are already produced by AI. This article first surveys what the summit disclosed, then takes the technical details apart layer by layer, and closes with four projections for the future of inference engines.

1 What the Summit Disclosed: Four Product Lines, One Roadmap

The announcements line up as four faces of a single roadmap.

SGLang v1.0. The official framing is one center plus three work items: the center is efficiency and production scalability; the work items are strengthening existing functionality, surfacing configuration problems clearly, and establishing compatibility expectations. This is not a leap in features. The release tracker states that until the release branch is cut, the core features (scheduling, KV cache management, parallelism, prefill–decode disaggregation (P/D), speculative decoding, and the server APIs) receive bug fixes only, no large refactors. It reads more like a productionization commitment: unsupported configurations are rejected at startup or with a 4xx at request time, with a message naming the setting — never silently downgraded. A known-issues list enumerates nine historical cases where the engine silently overrode a user's setting; each is being fixed.

The v1.0 release slide from the keynote lists the serving features of the 1.0 cycle together (Figure 1, photo from the summit).

Figure 1  SGLang v1.0 release slide (source: SGLang Summit 2026 / LMSYS)
Figure 1 SGLang v1.0 release slide (source: SGLang Summit 2026 / LMSYS)

Miles v0.2. The RL post-training framework goes cross-hardware and cross-workload: training runs on NVIDIA GPUs, AMD GPUs, and Google TPUs (Trainium landing soon), with a choice of Megatron-LM, FSDP, or TorchTitan as the training backend; workloads extend beyond text to image, video, and audio RL, Miles-Diffusion covers diffusion models, and Multi-LoRA customizes one base model for many tasks at once. The release merged 400 PRs from 20 contributors.

SGLang-Neuron v0.1.0. The Trainium backend co-developed with AWS Annapurna Labs is now open source: a PyTorch-native plugin, in public preview and limited to supported language and diffusion models. The preview's feature list is heavier than the word "port". On the engine core, a fully asynchronous runtime runs decode steps back to back with no idle time in between, and NKI megakernels run every decoder layer of a step in a single kernel call. On workloads, Kimi-K3 serves prefill–decode disaggregated across three Trainium3 nodes with KV moving over NIXL on EFA, and GLM-5.2 (MXFP4) serves a one-million-token context for agentic coding workloads.

NIM Certified. NVIDIA folded SGLang into its enterprise distribution line: validated containers, optimized configurations per supported model and GPU, ongoing security maintenance, with optional enterprise support available through an active NVIDIA AI Enterprise subscription. NIM's validation is curated: it covers more than 30 NVIDIA hardware SKUs and more than 37 popular models, and its performance objective is stated concretely: tokens per second per GPU, on the configurations users actually deploy.

Governance followed: nine founding members (NVIDIA, AMD, Intel, Google, Meta, Thinking Machines Lab, Cloudflare, Modal, and RadixArk; Thinking Machines Lab was added four days after the announcement) form the Consortium. Each member appoints a technical representative and joins two roadmap meetings a year; working groups may collaborate between meetings, with public updates on roadmap direction and progress; responsibility for technical decisions and code acceptance stays with the maintainers. This is open governance with a sponsor layer, not a transfer to a foundation.

The security thread ran through the same keynote: vulnerabilities reported by Cloudflare and NEAR AI have been fixed, ongoing work covers isolation between requests and protection of administrative access, and the community is publicly recruiting security partners.

The serving line is being rebuilt around agent workloads; hardware coverage spreads horizontally; post-training converges with serving at the engine layer; and the development process itself is becoming AI-driven. The next four sections take these faces one by one.

Figure 2 puts the structure in one picture: consolidation happened at the engine tier, competition sank into hardware efficiency and floated up to distribution and service, with governance running across every layer.

Figure 2  Competition changed layers: consolidation at the engine tier
Figure 2 Competition changed layers: consolidation at the engine tier

2 The Center of v1.0: Managing Agent State

The v1.0 announcement makes only two technical statements about agent workloads, but each points at a concrete constraint: hybrid model architectures require different types of state to be managed together; and long-running agents accumulate histories that can exceed GPU memory.

The first is about the model side. Attention layers interleaved with linear or sparse layers mean the state to preserve is more than KV tensors, as covered in our previous article ("Agent-Written Inference Engines Beat SGLang"), prefix caching for GLM-class hybrid-attention models must store both the paged state of sparse attention and the recurrent-state snapshots of linear attention; storing KV alone cannot reproduce the full model state. The second is about the workload side. As AI shifts from answering questions to getting things done, an agent resends its ever-growing history to the model at every step, and that history eventually outgrows what a single node can hold in GPU memory.

The answer is unified memory management plus hierarchical caching, in the official wording reuse across different types of state and tiers of storage. The layering follows the official blog post's own wording: the radix cache reuses prefix KV in GPU memory, and HiCache extends that reuse to host memory and external storage. The widely quoted "60% TTFT reduction" needs its full measurement context. The benchmark ran GLM-5.2-FP8, eight B300 GPUs per worker, TP8/EP8 with prefill and decode identically configured; 32 concurrent conversations over 86 rounds; four groups of eight conversations each share a 32K-token initial prefix, each round adds 1K input tokens and generates 512 output tokens, ending at 160K context per conversation. Against a baseline with decode-side prefix reuse disabled, enabling GPU and host reuse cut average time-to-first-token (TTFT) by 25.4% (2.07 s to 1.54 s) and the median by 60.3% (0.681 s to 0.271 s), while output throughput rose 2.9%. Note this is the gain from decode-side prefix reuse in a P/D disaggregated setup — not a generalized HiCache number.

Production offers a corresponding case: the iFlytek Astron MaaS team serves GLM-5.1/5.2 CodingPlan workloads with pipeline parallelism, P/D disaggregation, and MTP; requests typically carry 30K to 200K input tokens, some reaching 1M. Before adopting HiCache with external storage, the deployment's cache hit rate was around 30%; after, 95%, with median TTFT under 3 seconds. This is a team-reported production reading, not a benchmark, but its direction matches the benchmark above: history is no longer recomputed; a hit is reused.

One mechanism shows best how caching and scheduling cooperate: under P/D disaggregation, the decode side maintains its own radix tree and reports the reusable boundary to prefill (decode_prefix_len), so prefill sends only the missing suffix. For a 12K-token prompt, an 8K prefix is restored locally on the decode side, and the remote transfer drops from 12K to 4K. Cache reuse turns from "saving compute" into "saving network" — the shape of cache–scheduling cooperation under agent workloads, consistent with what we observed in the last match: the winning engine there also wrote kernels, but what separated the winners from the baseline was the cooperation of scheduling and caching, not kernel throughput itself. Figure 3 puts the three storage tiers and this reuse flow in one picture.

Figure 3  Where state lives, how it is reused: hierarchical caching and decode-side prefix reuse
Figure 3 Where state lives, how it is reused: hierarchical caching and decode-side prefix reuse

Explicit configuration and compatibility expectations are the other half of the roadmap. The tracker lists nine known silent-override cases, such as a request's seed ignored unless deterministic inference is enabled, max_new_tokens silently capped by KV cache capacity, and speculative-decoding steps quietly replaced with 1; each is to be fixed by either honoring the setting or rejecting it with a named error. The compatibility policy lists command-line flags, environment variables, HTTP routes, and the Python Engine API as stable interfaces across 1.x, with deprecation windows. None of this is glamorous, but it turns "can this engine go to production" from a matter of reputation into a matter of contract. The Rust-written sgl-router 0.1.0 shipped with the 1.0 cycle, replacing the old sgl-model-gateway.

The composition of engine competitiveness is changing. When Meta explained its choice, the engine core was described as three parts: scheduling, caching, kernels; v1.0 puts its weight on the first two. The next competitive axis of inference engines is state management: where state lives, how it is reused, how it moves across tiers.

3 On One Stack, How Hardware Pulls Ahead

Once the engine layer converges, the hardware camps' numbers can be read against each other for the first time, but two kinds of comparison must be separated first.

Cross-vendor comparability comes from a common load. At the launch of Kimi K3.1 Pro, AMD and AWS delivered day-0 support, and NVIDIA published performance figures on Vera Rubin. The same open-source model on the same engine stack constitutes the comparable condition itself. Open models have become the hardware camps' common test load; the weight of that fact is that in the past each proprietary stack tested its own way, while now the comparison conditions are ready-made.

The generational comparison appears in NVIDIA's own numbers: Kimi K3.1 Pro on Vera Rubin (Dynamo plus SGLang) reached up to 6× total token throughput per GPU versus GB300 NVL72 running AgentX, at matched interactivity. Three elements of the measurement are indispensable. This is a comparison of NVIDIA's two platform generations, Vera Rubin versus GB300, same vendor across generations. "Total throughput including cached input tokens" is the agent-workload scoring method: cache hits count as output at near-zero compute cost and cannot be converted into raw decode throughput. And the comparison load, AgentX, is the same corpus used in our last match, recorded from opt-in Claude Code sessions, stripped of original content and rebuilt with synthetic tokens, preserving length, timing, and prefix-sharing structure: real session structure, synthetic content. Vendor-reported, from a keynote; labeled accordingly.

The numbers come from Kapasi's keynote slide (Figure 4).

Figure 4  Kimi K3.1 Pro on Vera Rubin performance slide (source: SGLang Summit 2026 / LMSYS)
Figure 4 Kimi K3.1 Pro on Vera Rubin performance slide (source: SGLang Summit 2026 / LMSYS)

NVIDIA also disclosed three pieces of systems work. Dynamo integration provides cache-aware routing, with requests following cache placement, plus distributed serving; Snapshot and Model Express speed up replica startup, cutting cold-start and elastic-scaling time; and Vera CPUs are positioned as the substrate for agent sandboxes and RL workloads, so CPU compute enters the inference topology and the agent's execution environment sits on the same platform as model serving.

AWS's Trainium stack shows depth already in preview: beyond the asynchronous runtime and NKI megakernels covered in Section 1, SGLang Diffusion now runs on Trainium2, covering FLUX.1-dev, FLUX.3-Action, MiniMax-H3, Wan2.2 A14B, and LingBot-World-V2. Here two megakernels must be distinguished: the Neuron NKI megakernel is a kernel-fusion technique, one kernel per step; the "AI-written GPU megakernels that made decoding up to 2× faster" from the technical sessions are agent output. These two words do not name the same thing; the latter belongs to Section 5. On AMD's side, a time-accounting analysis of agentic RL on MI355X followed by a series of optimizations lifted throughput 3.5× (a workshop figure); collaboration on MI455 and Miles continues. Google's TorchTPU works with SGLang and Miles, in the direction of workload-specific TPU design.

Cerebras's pairing pushes P/D disaggregation to the device-class level: GPUs do prefill, the wafer-scale system does decode. Prefill is compute-heavy, decode is memory-bandwidth-bound; the logic of splitting load by hardware trait is unchanged, but disaggregation no longer requires both ends to be the same kind of hardware. At the kernel layer, FlashInfer keeps supplying improved kernels to SGLang. NVIDIA's Luis Ceze described how AI-assisted kernel optimization is accepted, through compiler feedback plus independent correctness checks; details follow in Section 5.

Meta's production evidence gives the competition a sense of scale: SGLang supports Muse App, Muse Code, Model API, the Meta family of apps, and image and video workloads, on infrastructure spanning NVIDIA GPUs, AMD GPUs, TPUs, and MTIA; the migration from prior inference solutions saved more than 2× GPUs, validated at large scale across multiple hardware platforms. This number is Meta's own; the benchmark and validation details were not disclosed, but this is the largest-scale multi-hardware production migration reading publicly available.

Every efficiency figure in this section is self-reported by the camps on the common stack, each labeled with its measurement context. Horizontal comparison in the consolidation era runs on measurement discipline.

4 Post-Training Converges at the Engine Layer

The relationship between RL post-training and serving is usually told as two stacks. Miles pulls the interface into the engine layer: rollouts run on SGLang, the training side picks one of Megatron-LM, FSDP, or TorchTitan, and the synchronization cost between them is the hard problem.

The trillion-parameter Kimi-K2 (FP8, roughly 1TB of weights) makes this cost concrete: its weight-update pause dropped from 53 seconds to 7.2 seconds, across 32 training nodes and 32 rollout nodes. The number matters because during weight transfer the entire RL run stalls, with trainer and inference engines both waiting, and weight synchronization sits on the critical path. NCCL broadcast exposes three problems here: the redundancy of sending identical data multiple times across the network; most ranks idling while a few participate; and rigidity, since an NCCL communication group, once built, is fixed and dynamically scaling inference instances means rebuilding the group. Under lock-step semantics, a single slow-starting receiver can hang the entire group.

The new mechanism is that on the source side, each training rank keeps an engine replica in CPU memory (about 32 GB per rank as the cost), transferring weights point-to-point over RDMA via the Mooncake TransferEngine, in the correct shape, to the SGLang ranks, and rollout resumes significantly faster. Put the four options side by side and the picture is clear: disk loading takes minutes, NCCL broadcast runs about 50 seconds, Perplexity's fabric-lib reaches 1.2 seconds but is not open source and supports only FSDP2 DTensor, and this design lands at 7.2 seconds fully open source. It is still an engineering trade-off, with the cost and the gain both on the table.

Three things on the Miles line matter to RL engineers: multimodal RL in token space, with a single media processor so rollout and trainer see the same tokens; score centering (since v0.1.2) removing the drift stale samples add to the gradient; and experimental fault tolerance, where a failed cell no longer ends the run. NIXL becomes a second transport for P2P weight transfer, the same library the Trainium side uses to move KV over EFA. The transport-layer abstraction is converging, and that is a signal to keep watching.

One more piece of corroboration loops back to Section 2: Miles v0.2 fixed Mamba state resets at packed-sample boundaries for Nemotron-H / Nemotron 3 (hybrid Mamba architectures). The state-management problem of hybrid architectures is not confined to the serving stack; it is equally real in post-training. RL rollouts are themselves long-context, multi-state, agent-shaped workloads, and unified memory management and hierarchical caching are efficiency sources for them too.

5 AI Is Starting to Write the Engine Itself

The most unusual official sentence of the summit sits inside the v1.0 announcement: agents now produce most of the project's merged PRs; the next-generation toolchain will write code, review PRs, build tests, manage compatibility, and propose roadmaps.

The technical sessions put numbers on it. GPU megakernels written by AI made decoding up to 2× faster, with outputs verified against the original kernels. Two engineers working with AI agents ported SGLang to another AI accelerator (unnamed) in under a week. The source says "within 1% of production performance"; the reference frame was not spelled out. The acceptance mode for kernel optimization is compiler feedback plus independent correctness checks. And a symbolic scene from the workshops: a fully local coding agent powered by Qwen3.8-27B on a single RTX PRO 6000 debugged and repaired optimizations in Mini-SGLang — local inference repairing the engine itself.

The conclusion of the last match connects here. In that Singularity-Bench inference-engine contest, a complete working engine was an entry ticket only top models could obtain: capability decides whether you get in and which tier you stand in; and within the top tier's two winners, income stopped tracking model tier and tracked only actual work hours and division of labor, and the winners' gains came from the cooperation of scheduling and caching. The open-source mainline has now itself become the production site: AI output enters the mainline while human maintainers keep acceptance authority. That structure is isomorphic to the Consortium's governance: technical decisions and code acceptance stay in human hands; capacity comes from machines.

The acceptance mode is what makes this tenable. AI output is not checked by humans line by line but by a machine loop: compiler feedback says where it went wrong; independent correctness checks say whether it is right. That is the precondition for AI engineering to scale, and the mechanism that makes "AI writes the engine" trustworthy. Without it, "most merged PRs" would just be a euphemism for quality incidents.

The flywheel should be stated with restraint: AI writes the engine; the engine runs AI; more AI output follows. This is not vision rhetoric but an extrapolation of what is already in the merge records. The bottlenecks are here too: the summit's public call for security partners is itself evidence that security-review pace has not caught up with production pace.

6 The Future of Inference Engines: Four Projections

Each projection below is labeled with its evidence strength and observation points, showing which parts are direct extrapolations of established fact and which are judgment.

First, state management becomes the engine's core competitive axis. Workloads shift from question-answering to task completion; agents re-read history at every step; engine differentiation concentrates in the integration of caching, memory, and scheduling. Evidence strength: direct extrapolation of the official v1.0 roadmap, the HiCache benchmark, and the iFlytek production case. Watch: the tracker's compatibility policy and silent-downgrade cleanup, hierarchical-caching numbers from more production cases, and how state-management complexity moves as hybrid-architecture models spread.

Second, engine-choice weight falls; hardware efficiency and service distribution rise. Same engine plus same common load makes the comparison possible. The cross-vendor evidence is K3.1 Pro: day-0 support from AMD and AWS, Rubin performance figures from NVIDIA, all on the same model and the same engine stack. The same-stack generational comparison has Rubin 6× as its example. On the enterprise side, the risk of choosing "which engine" gives way to "whose hardware runs it well and whose service catches it": NIM subscriptions and the Dynamo system stack are ready-made commercial forms. Evidence strength: consolidation fact plus NIM's commercial motion, with the direction clear and the magnitude to be observed. Watch: the Consortium's first roadmap decisions, third-party same-stack hardware re-tests, and the evolution of the vLLM–SGLang dual-stack landscape.

Third, heterogeneous compute becomes the inference norm. One stack covers NVIDIA, AMD, and Intel GPUs, TPUs, Trainium, MTIA, and wafer-scale systems, with CPUs entering the topology as agent sandboxes; Cerebras-style cross-device P/D pairing pushes "assign hardware by load trait" to the device-class level. Discount the evidence strength here: most backends are at public-preview, day-0, or announcement level, with the direction set and the production scale unproven. Watch: production-scale landings of the Trainium and MTIA backends, actual deployments of wafer-scale decode pairing, and the tracker's open decision on K3.1 support scope for AMD MI35x.

Fourth, the flywheel of AI-written engines has started turning. Most merged PRs are already AI output, AI kernels are entering the mainline, and ports take a week, so the engine's own pace of evolution becomes a new variable. OpenAI's Lilian Weng says researchers are becoming team leads directing agents through implementation; the infrastructure-project counterpart has already appeared. Evidence strength: present fact, with uncertainty in the extrapolation, and acceptance and security review are the bottlenecks. Watch: the AI share of merged PRs over time, the frequency of AI kernels entering the mainline, and whether security-audit pace matches production pace.

Governance and ownership are the horizontal variable behind all four: open governance with maintainers controlling the roadmap, and chipmakers deeply embedded in open-source stacks, are already market-structure questions in a regulatory lens. NVIDIA's $12.93 billion purchase of Hugging Face is, reportedly, under Justice Department antitrust inquiries, and the question asked is precisely whether the chipmaker would thereby control a key distribution channel for AI models. Over the next two years, the Consortium's governance-disclosure cadence and the outcome of that review matter more than any benchmark run.

7 Boundaries and How to Read the Numbers

This article's conclusions come with three exceptions. What consolidated is the serving stack: training and post-training stacks have not: Megatron, FSDP, and TorchTitan coexist, and Miles is an engine-side extension, not a unified training stack. The dual stack persists: vLLM and SGLang are both alive, each has gone commercial (the vLLM team formed Inferact; RadixArk stands behind SGLang), and the Consortium covers only one of them. Proprietary stacks are not dead: TensorRT-LLM remains NVIDIA's compilation-based high-throughput option, and closed-source enterprise deployments remain common.

Every number's reading method is condensed into this table:

Figure Context Attribution
Rubin 6× Total token throughput per GPU incl. cached input tokens; AgentX load; matched interactivity; NVIDIA generational comparison (Vera Rubin vs GB300) Kapasi keynote, self-reported
Meta >2× GPU savings Large scale, multi-hardware validation Meta self-reported; benchmark and validation details not disclosed
Median TTFT −60.3% Multi-turn benchmark; decode-side reuse in P/D; GLM-5.2-FP8, 8×B300, 32 conversations × 86 rounds, 160K final context Official SGLang blog benchmark
Hit rate 30%→95% Production case; 30K–200K inputs; PP + P/D + MTP iFlytek team self-reported
1,800 tok/s single user 1.8× stock recipe; reasoning accuracy unchanged Recap's retelling; reporter and model unnamed
Weight update 53s→7.2s 1T Kimi-K2 FP8; 32 + 32 nodes; ~32 GB CPU replica per rank Official SGLang engineering report
Megakernel 2× Decoding; verified against original kernels Technical session disclosure
One-week port Two engineers + AI; another accelerator; "within 1% of production performance" (reference frame not specified) Technical session disclosure
500 days, zero regression Eight major upgrades Single production team's experience
MI355X 3.5× Throughput after a series of agentic-RL optimizations Workshop demo, AMD self-reported
332 organizations Grown from early adopters within a year Organizer self-reported

Every efficiency figure of the consolidation era is self-reported on the common stack, or retold without attribution. Horizontal comparability is not automatic; it is bought with measurement discipline. This table is how this article reads them.

Closing

Three judgments close the piece.

The engine has gone from competitive moat to public utility. The risk of choosing "which engine" is falling; the weight of choosing "whose hardware runs it well and whose service catches it" is rising. Competition has not disappeared; it has moved down into kernels, interconnect, and whole machines, and up into distribution and enterprise service.

Agent workloads are rewriting engine architecture. Multi-type state in hybrid architectures and histories that outgrow GPU memory extend the engine's core capability from throughput to state management: where state lives, how it is reused, how it moves across tiers. That is where v1.0 has placed its bet, and where engines will actually compete next.

The flywheel of AI-written engines has started turning. Most merged PRs produced by agents, AI kernels in the mainline, and week-long ports are present facts, not predictions. The pace at which the open-source stack evolves is itself becoming a variable.

The conditions and observation points were given in Section 6, covering the tracker's landing, third-party re-tests, the production scale of the backends, and how well the pace of security audits matches the growth of AI-produced output. Looking back in two or three years, this summit will likely be remembered not as "the first conference with two thousand attendees" but as the point where engine competition changed axes — from throughput to state, from stacks to hardware, from humans writing engines to engines participating in writing themselves.