In 2024, Meta added eight new datacenter network types in a single year, with 800 to 2,000 network changes rolling through the pipeline every week. Before this era, the mainstream way to handle such work was senior engineers hand-writing Methods of Procedure (MoP), supplemented by scripts only their authors could read. The gap sits in between: how exactly does a network design blueprint become configuration running on thousands of switches? Industry research concentrated on topology and hardware deployment, from Google's Jupiter to Meta's F16 — describing what networks look like and how they get installed. Academic research concentrated on post-deployment configuration checking, with the Batfish line as the representative. The step from blueprint to switch configuration had long gone unexamined as a systems problem.
Matryoshka, published at NSDI '26, fills that layer. It has run inside Meta for six years: 400 thousand lines of code, four major code revisions, five generations of datacenter networks, supporting nearly 900 networks across 18 types; today Meta's DCN infrastructure is 100% Matryoshka-based, and its latest work is the 100K-GPU supercluster that carries LLaMA training. The paper calls itself "the first production-grade DCN design realization system made accessible to academia." This article takes its operation apart: how a network compiler turns network delivery into software engineering.
1. The Problem: Network Delivery Becomes Continuous Delivery
Matryoshka's reason to exist is written in three sets of Meta production numbers.
The first is scale. Meta's networks span more than 70 regions, with thousands of switches per building; the largest F16 fabric holds 5,664. Among the nearly 900 networks of 2025, the new-generation front-end dcP FE takes 47.3%, fabric aggregate FA takes 18.5%, RoCE training networks take 12.2%, back-end dcP BE takes 11.8%, experimental DSF takes 5.0%, and other types take 5.2%.
The second is iteration speed. From 2020 to 2025, network types grew from 2 to 18 and deployed instances from 18 to nearly 900, both curves accelerating after 2023. The back-end AI network has already accumulated 109 cluster versions across 26 device roles. The heaviest change pressure actually sits on the front-end network, absorbing business migrations and dynamic adjustments; the back-end's structure is comparatively stable while its versions keep forking.
The third is heterogeneity. In-house FBOSS white-box switches coexist with third-party gear; NVIDIA GPUs and in-house MTIA accelerators place different demands on the network; different vendors' configuration syntaxes are mutually unintelligible.
The three sets of numbers point to the same structural change: the network is no longer build-once-stable infrastructure but a continuously delivered product iterating weekly. Change frequency multiplied by network diversity is the load the delivery toolchain must bear. AI sends both soaring simultaneously, and at Meta's scale the experts-plus-scripts model has reached its limit.
2. The System at a Glance: A Network Compiler
The paper's positioning is blunt: a software system acts as a compiler, translating high-level network design intent into runnable switch configuration. The pipeline runs in seven steps (Fig. 1).
Step one: engineers write a Network Spec in Thrift (Meta's interface definition language), declaring what they want rather than how to configure it. The spec is a set of structs. The Switch Entity sorts switches into roles and generates device names through templates with tokens and value ranges. The Hardware Platform records chassis model, line-card model, and switch operating system; translated into FBNet entities, it generates bills of materials (BOMs) for procurement and deployment. Provisioning State marks each device's position in its lifecycle: PROVISIONING during bootstrapping and quality assurance testing, IN_USE once ready for production traffic. The remainder covers topology patterns and counts, port-mapping templates, IP supernets, BGP templates.
Two kinds of content are deliberately kept out of the spec: static, rarely changing settings such as QoS enter generation directly as templates, and real-time status such as port up/down or device drain and undrain stays out entirely; intent and current state are fully separated. Spec engineering is managed with the same discipline as code: versioned with retrievable histories, each change automatically sanity-checked and then human-reviewed, with a company-wide standardized tooling stack running integration, regression, and validation on every change, blocking merges on failure unless formally approved.
Step two is topology synthesis; step three, port mapping. Topology is first generated in abstract form: vertices are real device names and edges are logical interface names, serving early planning and procurement; port templates then map it onto physical port names, producing the concrete topology that directly guides building cabling. Port templates went through a consolidation: the initial per-network version hard-coded switch names, and across hundreds of production networks the templates proliferated beyond easy management; the rebuild replaced specific switch names with generic device identifiers, letting a single template be reused across network instances and cutting the template count sharply.
Step four is layered IP allocation. Two prefix families are managed separately: infrastructure prefixes carry routing and control traffic, including loopbacks and device interconnect addresses; workload prefixes carry actual data traffic, including cluster dynamic addresses, virtual IPs, and IP-per-task addresses. Prefixes are sliced hierarchically from supernets down to per-device per-port, kept stable across network renovations where possible.
Step five is route generation. OpenR handles intra-domain routing; BGP handles inter-domain and policy routing. BGP neighbor groups follow predefined peer-group templates by device-role pairs (spine to aggregate, for instance); route templates allocate prefixes programmatically by prefix category, each carrying a network domain config for specific switches that controls route distribution. Routing policy can be injected as external input.
Step six generates the Generic Switch Configuration (GSC), a platform-neutral form; step seven has each vendor backend translate GSC into native configuration, which passes validation gates before being written into FBNet, the authoritative network database, and distributed to switches.
The structure resembles a chip-design flow: the Network Spec plays RTL, topology synthesis and port mapping correspond to logic synthesis and place-and-route, GSC is the intermediate representation (IR), vendor translation is backend code generation, and the validation gates are the pre-tape-out checks. To be clear, "compiler" is the paper's own framing; the EDA analogy is this article's extension.
The name comes from the nesting doll, and the paper gives it two meanings: Clos is a modular, reusable building block, replicated and adapted like nesting dolls into hyperscale networks; and configuration generation advances at the same rhythm, progressively concretizing layer by layer from design intent to physical ports. The name is also a promise: however large the scale, delivery remains the same act of nesting one layer inside another.
3. Four Key Design Choices
Four graphlets compose every topology. Matryoshka decomposes any datacenter network into combinations of four standard connection patterns: complete bipartite, ring, full mesh, and circulant bipartite. Traditional Clos is a stack of layered bipartites; AI clusters add full-mesh topology for bandwidth density. Composition works by top-down decomposition: inter- and intra-layer connections are partitioned into repeating blocks, each matched to a predefined graphlet. The value of this design is component isolation: a new topology touches only the topology and port modules, while IP allocation, BGP templates, and device modeling are fully reused.
Both of Meta's experimental directions plug in this way. One is rail topology: GPUs of the same rank across machines connect to the same leaf switch, so all-reduce collective traffic completes within-rail exchange at the first hop; the AI racks and leaf switches participating in this scheme form a scaling unit (SU), and a higher-level spine tier interconnects rails within an SU and links multiple SUs together. The other is DSF (Distributed Scheduled Fabric): leaf switches break packets into cells, spray them across fabric uplinks for load balancing, and reassemble at the receiver, scheduled throughout by virtual output queues with credit-based flow control. DSF builds routing adjacencies leaf-to-leaf without traversing spines, and IP allocation is disabled on fabric nodes: each of these differences lands in its own module without entangling the rest (Fig. 2).
Deterministic and stateless: recompile from scratch, every time. The same input always yields the same output; every run is a complete recompile that never depends on the previous output, with all run state kept in FBNet. This property yields a counterintuitive benefit: retrofitting a live network is equivalent to deploying a new one. The compile result is diffed against FBNet's current state; switches outside the diff can be proven byte-identical, giving online upgrades a safety boundary (Fig. 3). The cautionary tale dates to 2020: a code defect during an online migration deleted every cross-fabric IPv6 prefix and nearly misconfigured an entire region's fabric switches. Meta then invested heavily in compile-time validation, from which the scorecard below follows.
GSC: a platform-neutral intermediate representation. Whether in-house FBOSS or third-party devices, every switch first compiles into the same Thrift-defined platform-neutral configuration, which vendor-specific backends then translate into native formats. Validation is split in two: between design intent and GSC, and between GSC and native configuration, each independent. Adding a hardware type means adding a translation backend; the layers above stay put. That is how heterogeneity is kept under control.
The concrete validation runs on two levels. The database level checks network topology statistics against the spec: switch count, link count, IP prefix count, each cross-checked. The configuration level checks completeness and consistency of the generated results: routing configuration cross-checks external routing policy definitions against generated peer information, and a routing emulation system verifies network-wide behavior against criteria such as no routing loops or black holes. These validations are modular interfaces that partner teams can extend with their own checks.
Rollout is likewise two steps: the network's full set of GSCs is staged in a distributed repository, and upon successful validation moved to a production-ready directory; network operators then arrange a deployment event, the converter picks up the GSCs, translates them into native configuration, and stores them on the switches. On detecting adverse impact, fleet-wide rollout halts: offending configs are removed from the production repository or replaced with fixed versions, and on devices, buggy configs roll back to a previously validated version or get overridden with a hotfix.
Diff-based updates that touch only the write layer. The most painful part of full recompilation is the database write. The early version rebuilt FBNet by wiping and recreating; a large fabric expansion's transaction could take 34.6 minutes. What forced the redesign was an incident: an external maintenance tool read an inconsistent intermediate state mid-transaction, mistook it for device failure, and triggered an alarm that shut devices down. The new algorithm, deployed in April 2022, updates by diff. The compiler keeps a local in-process copy of all database row entities, compares the old and new network models in memory, and generates transactions proportional to the change scope, in six steps: compare, drop foreign keys, delete, update static fields (Update-static), create, rebuild foreign keys. Fabric expansion transactions fell from 34.6 minutes to 46 seconds; grid migration from 34.84 minutes to 13.19 seconds. Note the compilation semantics did not change: it remains stateless full recompile, and only the database write layer changed. Greenfield builds still take the full-write path, which is why monitoring spikes dropped sharply but did not vanish: before the new algorithm shipped, transaction times frequently exceeded 2,000 seconds, and the remaining spikes are precisely the greenfield full writes.
One tradeoff runs through the whole system and deserves its own note: the generic model has a boundary. Topology and scale are now captured in the generic input model, while IP allocation still lives in compiler code sections written per network type. The paper's criterion is reuse prospects: capabilities likely to be reused by future networks go into the generic model, one-off specials go into dedicated code sections, which is how 400 thousand lines stay maintainable across 18 network types. Where that boundary sits is a continuously apportioned engineering decision as designs evolve.
4. Case Study: Delivering the 100K-GPU Supercluster
The compiler's largest work to date is the AI supercluster built over 2024 with more than 100 thousand GPUs, where the latest LLaMA training runs.
Two networks, front and back. Every training rack is dual-attached: the front-end network carries data ingestion, checkpointing, and logs; the back-end network is a dedicated RoCEv2 training fabric. The back-end's building block is the AI Zone: rack training switches connect the rack's GPUs and uplink to cluster training switches, non-blocking within the Zone; in the current supercluster, AI racks and rack training switches are mapped one-to-one, with all GPUs in a given rack uplinking to the same rack training switch. Across Zones, aggregator training switches interconnect in full mesh, with bandwidth intentionally oversubscribed to maximize port utilization supporting GPU connectivity density (Fig. 4). The paper uses scale-up and scale-out for intra-rack and cross-rack connection tiers (not the same thing as NVLink-style inter-die domains); readers should keep them apart.
Delivery combined brownfield retrofit and greenfield build. Three dcT buildings previously running general-purpose workloads, each with four RoCE storage clusters, had the original RoCE clusters expanded into eight AI Zones during the retrofit; storage moved to separate buildings, and as cross-building traffic surged, the FA aggregation layer carrying inter-fabric traffic received a matching bandwidth upgrade. Front-end traffic never went down during the retrofit: stateless deterministic compilation delivered its promise here, verifying switch-by-switch that unaffected devices kept their exact configuration while only the needed changes shipped. Two new buildings took the greenfield path: although the back-end is a training fabric, its switch roles mirror the front-end one-to-one, with rack, cluster, and aggregator training switches mirroring rack, fabric, and spine. Clos and full-mesh are existing graphlets, so the build reused spec templates and added only AI-specific configuration.
The entire regional supercluster is modeled as a single network entity to support cross-building training jobs; cabling accommodates both in-rack direct cabling and End-of-Row, the latter concentrating rack training switches into separate racks connected to GPUs via optical modules over single-mode fiber.
5. The Scorecard and Its Ceilings
The production numbers cut both ways (Fig. 5).
On the gate side: of 12,689 compilation checks in the past year, 589 were blocked, a 95.4% pass rate. Blocked configurations stop at Configuration Mover Service and never touch production, averaging roughly 11 blocked compilations needing human handling per week. Daily end-to-end behavioral tests have run at 96.4% success since late October 2024: of 46 failures, 25 were non-system factors; excluding them, 568 passes against 21 system failures gives 568/(568+21) = 96.4%. On the speed side: diff-based updates compressed large-expansion database transactions from minutes to seconds, with the paper's four scenarios showing reductions from 70% to 99.4%; new-network-product introduction (NPI) dropped from a year to four weeks, a 13× compression. The speedup came from three moves: making network specifications more adaptable, enforcing common design standards to avoid exceptions, and developing shared algorithms across DC types, IP address allocation being one such shared algorithm.
The ceilings are equally clear. First, the bottleneck did not disappear; it moved from hand-written configuration to handling blocked compilations, and roughly 11 human interventions a week say the pipeline is far from unmanned. Second, processing time scales near-linearly with switch count. The paper breaks end-to-end time into three components: GSC generation and upload run per switch and grow strictly proportionally with switch count, the largest share; database queries and design-intent compilation operate primarily per network and grow far more moderately. About 90% of CPU is spent on GSC serialization and deserialization — the intermediate representation itself has become the scale ceiling. The largest F16 fabric (5,664 switches) takes close to 200 minutes end-to-end (read from the axis magnitude of the paper's Figure 9); an order of magnitude beyond that likely demands a new intermediate representation or sharded compilation. Third, the shared code path has costs: adding port-mapping support for the AI back-end once broke migration logic for the traditional front-end, after which Meta instituted end-to-end non-regression testing standards covering all network types. The same theme has a routing-policy case: a BGP policy added to block certain prefixes unintentionally caused prefix mis-filtering in other DCs through the shared code path. Exact validation rules are hard to write in advance, because the prefixes are themselves generated by the compiler; the fix is to define rules more abstractly, using naming conventions or prefix mask length heuristics, which breaks the interdependency between Matryoshka and validation.
One more cut is necessary: the correctness numbers above are all self-reported, and their coverage is the generation pipeline. Configuration drift on devices, hardware defects, and syntactically correct but wrong-intent configurations all sit outside what these gates intercept.
6. Summary and Judgment
Matryoshka is worth reading for a different reason than novel network design. The paper states plainly in its related-work section that it proposes no new network design and is orthogonal to prior topology-focused efforts. Its real contribution is turning network delivery into software engineering: versioned design specs, daily regression behavioral tests, deterministic reproducible builds, gates and rollback. That a toolchain paper reaches NSDI reflects the AI era in which change frequency and network diversity are both surging: the load on the toolchain equals their product, and at Meta's scale the experts-plus-scripts model has reached its limit.
Three judgments. First, deterministic stateless compilation, graphlet algebra, and a platform-neutral intermediate representation: none of these depends on Meta's particular circumstances, and any team with network delivery at comparable scale can check against them item by item. Second, the claim of being "the first production-grade DCN design realization system made accessible to academia" is itself a signal: the pre-deployment configuration-generation stage has formally entered research view. Google's Jupiter series has published only up to the topology and management layers; peers publishing at the same layer is a matter of time. Third, place Matryoshka beside Meta's published MetaRoCE (runtime transport) and RDMATracer (runtime observability), and the software-defined picture of a network's full lifecycle is complete at Meta.
Three things to watch next: whether rail and DSF enter the next production generation; whether Google, Amazon, or Alibaba publish at the same layer; and if full-pipeline compile time for 100K-plus back-end networks approaches half a day, intermediate-representation replacement will be on the agenda.
Note: This article is based on the preprint of "Matryoshka: Realizing Hyperscale Data Center Network Design for the AI Era" (Yan Cai et al., Meta, May 2026), USENIX NSDI '26, cross-checked against the official USENIX abstract. Not investment advice. Written September 10, 2026, revised September 11 with additional implementation detail from the paper; all figures follow the paper's published numbers.