MetaRoCE in Depth: RDMA Designed for Loss, an Open Standard for Million-GPU Scale
MetaRoCE is Meta's clean-sheet RDMA transport protocol, opened through OCP. This analysis works through six questions: why it appeared, what it actually is, why Meta opened it, how the protocol works inside, what it is worth and to whom, and a closing peer review of ByteDance MegaScale, Alibaba HPN, Tencent Xingmai, AWS SRD, Google Falcon, and the UEC/MRC alliances.
On August 24, 2026, Meta's engineering blog published two posts. One introduced MetaRoCE, an RDMA transport protocol designed from scratch for AI workloads on commodity Ethernet, with the specification, reference implementation, and conformance test suite to be opened through OCP at the October Global Summit. The other introduced MTIA 300, Meta's first training chip with built-in NICs and communication-offload engines. The MetaRoCE post compresses its design philosophy into a single line: The fabric sees packets, but the NIC sees intent. The two pieces came from the same hand: Rajiv Krishnamurthy is one of MetaRoCE's four co-authors and the first author of the MTIA 300 post.
Why would a company that sells no networking gear hand its in-house transport protocol, specification and implementation alike, to the open community? That fact alone needs explaining.
1. Why MetaRoCE Exists: The Three Premises and a Decade of Evolution
RDMA's original purpose is to bypass the kernel network stack and let the NIC read and write application memory directly, pushing latency down into microseconds. It was born inside InfiniBand. That is a dedicated network custom-built for high-performance computing, naturally ordered and nearly lossless, and RDMA's simplicity takes the perfection of the network as a premise. To run RDMA on much cheaper commodity Ethernet, a best-effort network that drops packets has to impersonate InfiniBand. RoCE's three premises are the cost of that imitation project.
The first premise is losslessness. RDMA pays far more for a lost packet than ordinary Ethernet traffic does: the NIC's retransmission timer is measured in milliseconds while transit across the fabric takes only microseconds, and a single timeout can blow through tail latency. So RoCE guarantees no loss with priority flow control (PFC): as a switch buffer nears capacity, it sends a pause frame upstream to stop the other side from sending, buying a "lossless" network. The cost of that trade is written into the mechanism. Pause frames propagate hop by hop upstream, and the backpressure from one slow link can spread across an entire fabric; wherever it reaches, flows in the same queue that have nothing to do with the offending failure are frozen along with it, and head-of-line blocking arrives at millisecond scale; and to keep the mechanism from misfiring, every switch's buffer watermarks must be tuned unit by unit. The lossless network thus becomes the hardest network to operate.
The second premise is ordering. The transport assumes packets arrive in order so that the NIC's hardware implementation stays simple, at the cost of binding each flow to a single path. The third premise is one path per flow. ECMP hashes the five-tuple, and all packets of a flow land on the same path from establishment to teardown. The ordering assumption needs exactly this binding, and the two are mutually causal. In clusters of a few thousand GPUs this carefully tuned engineering works well: training traffic is dominated by periodic large flows, few of them concurrent, and the cost of hash collisions is still absorbable.
As scale climbs, the three premises fail one by one, and Meta has a first-hand record of every crack. Meta's SIGCOMM 2024 paper, RDMA over Ethernet for Distributed AI Training at Meta Scale, retraces the limits of this engineering. On routing, default ECMP's distribution was uneven, and the routing scheme went through several iterations before approaching the ideal. On congestion control, the DCQCN they started with hit vendor-side implementation defects when the network moved to 400G, and Meta simply shut DCQCN off, running PFC-only for 18 months. How was congestion managed during those 18 months? The answer had already left the network layer: congestion management moved up into the collective library, where every transfer begins with a receiver-issued clear-to-send handshake and no data is injected until the receiver is ready. The two control loops that had operated from the sender's perspective without accounting for each other became one, and system behavior grew more predictable. Even the ordering premise was relaxed: the paper lists relaxed ordering alongside NIC PCIe credit tuning among its enhancements.
The production-side numbers come from Meta's Llama 3 paper: 419 unexpected interruptions over the 54-day training of Llama 3 405B on 16,384 H100 GPUs, on a RoCE fabric built from Arista 7800 and Minipack2 switches at 400G. The direction is worth noting: inside the lossless frame, Meta kept moving its control points up from the network layer into software. That was already the first step of moving intelligence back to the endpoint, only still in software, and still with PFC. Two years later, the same company's answer was to stop patching the three premises and overturn them.
The industry has spent a decade on routes past these premises, in two branches. One repairs the network and keeps the premises standing: better congestion control, reworked topologies, tuned ECMP, with ByteDance, Alibaba, and Tencent pushing the engineering to its limit on this road. The other replaces the premises and moves transport responsibility back to the endpoints. That road has four problems to solve, and for each the industry already has its answer: multipath utilization, through per-packet spraying; reordering, through receiver-side reassembly, or by designing it away entirely; loss tolerance, through selective-acknowledgment bitmaps; many-to-one convergence, through receiver-side rate allocation. Together, those four answers are MetaRoCE's mechanism list.
The pioneers of the endpoint route are none of them paper designs. AWS's SRD proved the route viable first, with endpoints handling reordering and retransmission themselves so that PFC could disappear entirely; Google's Falcon landed privately in its own clusters; NVIDIA's Spectrum-X takes the middle road of switch-NIC coordination; UEC 1.0 and the MRC (Multipath Reliable Connection) wrote the endpoint route into open standards. The proprietary camp proved feasibility and the alliance standards laid out specifications; what was missing was a complete implementation led personally by the largest buyer, with production validation behind it.
Meta's run-up is traceable. From the production RoCEv2 retrospective in August 2024, to the ESUN network architecture initiative in October 2025, to the RCCLX collaboration with AMD in February 2026 and the same-day protocol-plus-chip release in August, this line was laid down over two years, with AMD the partner through the back half. Meta's cluster scale is several hundred thousand GPUs by its own account, more than a million at H100 equivalence by press estimates, with the Prometheus generation (late 2026, 1GW) already under way. The cracks in RoCEv2, a decade of endpoint intelligence, and Meta's own two-year run-up converged in August 2026.
2. What MetaRoCE Is: Definition, Deliverables, and Form
In one sentence: MetaRoCE is an RDMA transport protocol that runs over UDP on commodity Ethernet, designed from scratch for AI training and inference workloads, and it preserves the RDMA Verbs programming interface. It is three things at once: a transport protocol, an OCP open project, and the next transport foundation of Meta's own production network. The third identity is our path judgment; no announcement contains that promise.
Four deliverables come through OCP. The full protocol specification, from which any vendor can build interoperable hardware. A software reference implementation, libsoftmetaroce: a fully functional transport stack that runs on commodity Linux over standard UDP sockets with no specialized hardware, positioned as the authoritative behavioral model for silicon development and the foundation of the unified conformance framework. A production-grade conformance test suite, the tool that lets hardware vendors prove their implementations match the spec. And a DPDK-optimized reference implementation shipping at the October summit, the reference stack's step onto the performance path. Together, the four deliverables are what make an open standard procurable: the specification defines right from wrong, the reference implementation provides an executable behavioral baseline, and the conformance suite supplies the acceptance yardstick. Missing any one of the three, openness stays on paper.
What the NIC has to take on: path management and per-path telemetry, packet spraying and entropy adjustment, bitmap handling and precise retransmission, two-way congestion control, and the three-tier stream-path-connection state model. The protocol lands in three forms. The software form is libsoftmetaroce itself, runnable on any Linux server for development, verification, and behavioral comparison. The programmable-NIC form has AMD Pensando as its validated path: Meta worked with AMD to implement the full protocol on Pensando processors, and a 64-node AMD GPU cluster running the RCCL collective library completed a direct comparison against RoCEv2; per AMD, the protocol was implemented on the Pollara 400 NIC and is on a path to Vulcano 800 (800G) deployment. The fixed-function form is within the design target: the protocol is meant to run on programmable and fixed-function NICs alike, and implementations from other vendors are underway.
What Meta itself has disclosed of its own use falls on the validation side and the chip side. The validation side is the 64-node cluster above; the chip side is MTIA 300, released the same day. The training workloads this chip targets are themselves communication-hungry: the embedding tables of recommendation and ranking models can hold over 99% of the model's parameters, training relies on hybrid parallelism, and collectives like AllReduce, AllToAll, and AllGather spread across hundreds of accelerators, with communication and computation contending for the same hardware.
MTIA 300's answer is to make communication a first-class citizen of the chip. Per the original Meta engineering blog, the chip carries two network chiplets, each containing 6 in-house 800Gbps RDMA NICs for a total of 1.2TB/s. The communication engines bypass PCIe entirely, sitting right next to the compute so that data movement skips the hop through the host bus, and the same set of NICs can switch dynamically between scale-up (16 nodes, 1TB/s) and scale-out (200GB/s). The compute array is a 12×6 grid of processing elements, and communication gets 16 dedicated message engines: each pairs a RISC-V core for orchestration with a near-memory compute block that performs reductions at 128 bytes per cycle, for a combined reduction throughput of 2.8TB/s, more than twice the I/O bandwidth. With a large GEMM and collective communication running concurrently, compute throughput degrades by less than 0.5%; a conventional GPU cluster degrades by more than 20% in the same scenario.
The companion communication library is HCCL, co-designed with the chip from scratch (it shares its acronym with Huawei's Ascend-ecosystem HCCL; the two are unrelated). Its approach is compiled communication: each collective is compiled into a complete set of subgraphs and handed to the message engines for autonomous execution; once work reaches the device, the host is no longer involved. To minimize per-transaction latency, there is also a fast signaling path: the work request write itself serves as the doorbell, eliminating an additional memory read and saving roughly 800 nanoseconds per operation. The scorecard: HCCL reaches up to 940GB/s within a single rack, and a 150B-parameter production recommendation model running across 40 accelerators completes total communication 3.9 times faster than the equivalent GPU cluster. These numbers come from the ISCA'26 chip paper and the SC26 HCCL paper, the latter available as preprint arXiv 2608.00358. Two qualifications are needed: the original text never says these NICs run MetaRoCE, and the chip targets training workloads for recommendation and ranking models, not LLMs. That the protocol lands in first-party silicon is our path judgment; no announcement says so.
3. Why Open It: What Is Given and What Is Kept
What is opened is a three-piece set: the full specification, the reference implementation source, and the conformance suite. Any NIC vendor can implement the specification and sell compatible hardware; any buyer can run the conformance suite as acceptance; any researcher can use libsoftmetaroce as a behavioral model. What is opened is not just protocol text but the acceptance machinery.
What Meta keeps is equally clear. The lead on spec evolution stays with the Meta team; the four co-authors are the gatekeepers of the roadmap. Production tuning and operational experience are not in the specification; that is the asset of a team that has run a 16,384-GPU RoCEv2 cluster. Integration details with its own silicon are not in the OCP package, and the engineering detail of the Pensando implementation sits with AMD. In other words, the industry receives an implementable protocol and acceptance tooling, while Meta retains the knowledge that makes it run best.
Each party takes what it needs. NIC vendors gain a new track: beyond the traditional RDMA NIC market there is now a product category with open specifications and clear acceptance criteria, where AMD has moved first and Broadcom, Intel, and white-box vendors start from the same line. Cloud and cluster buyers gain a procurement path: purchasing shifts from buying integrated stacks to bidding against a spec, with the conformance suite as the yardstick for cross-vendor comparison, and the tender can be written in three lines: implement per the OCP specification, pass the conformance suite, quote a unit price. Researchers gain a behavior model open to comparison, the first public reference isomorphic to a hyperscaler production environment.
The division of contribution and adaptation has taken shape. Governance sits with OCP's ESUN initiative, which established the open multi-vendor philosophy at the fabric layer; MetaRoCE extends it to the transport layer. On the NIC side, AMD has implemented and others are in progress. On the switch side there is essentially nothing to do; ECMP and ECN (Explicit Congestion Notification) are existing features. On the collective-library side, RCCL was adapted along with the validation, while NVIDIA's NCCL and the other mainstream stacks await the specification's landing and demand.
Openness is not philanthropy; it is procurement strategy. Meta sells no networking gear, so turning the transport protocol into a public good costs it no revenue. What it buys back: AMD, Broadcom, Intel, and white-box NIC vendors can all implement MetaRoCE against the OCP specification and conformance suite, and procurement shifts from buying integrated stacks to bidding against a spec. The buyer-defined protocol model was demonstrated once by OpenAI with MRC; MetaRoCE is the second. The difference is that Meta simultaneously holds its own training silicon, hyperscale self-built clusters, and one of the world's largest network procurement budgets. Protocol, chip, and procurement: three cards in the same buyer's hand.
4. Deep Dive: Mechanisms, Flows, Validation, and Deployment
Three mechanism pairs, overturning the three premises one by one.
The first pair: out-of-order delivery and multipathing. MetaRoCE splits each connection into many logical paths and sprays packets across them; out-of-order arrival is the designed norm. The spraying does not disturb a single corner of the network: the ECMP hash input at the switches includes the UDP five-tuple, and the source port is a field the NIC may rewrite freely. In a traditional stack the five-tuple stays fixed for the life of the flow, one path per flow forever; the MetaRoCE NIC adjusts the source port packet by packet, which amounts to changing paths packet by packet, and the switches keep hashing and forwarding, entirely unaware. That is what entropy adjustment means: the right of path selection moves out of the network's hash tables and back into the NIC.
At the destination, every packet carries its own destination and writes directly to its final memory location, with no reorder buffer and therefore no head-of-line blocking. The two verb semantics are handled separately: Writes carry the destination address in every packet; Sends carry the match to a posted receive buffer, so a message needs no round trip to learn where its data goes, and whatever arrives first lands first. This detail is what preserves two-sided messaging: collective libraries need not demote all their communication into writes.
Each path independently maintains real-time telemetry of round-trip time, ECN marking, and utilization, which lets congestion be told apart from failure: congestion shows up as rising round-trip time plus more ECN marks; link failure shows up as collapsing utilization plus stalled transfers. The transport rebalances explicitly on this basis, moving traffic off degraded paths. Because each path maintains its own window and round-trip estimate, a bad link slows only one path, and the connection continues to be carried by the remaining paths. On multiplane networks, plane selection belongs entirely to the NIC, and the network is used no better than the NIC sprays; the network holds no transport state.
The second pair: loss tolerance and two-way congestion control. MetaRoCE treats Ethernet as a lossy network and does not ask it to become lossless: no PFC, no pause frames. Loss detection rests on per-path independent sequence numbers, the most critical step in the design: a traditional transport has a single global sequence space, where an acknowledgment gap may be nothing more than disorder that has not arrived yet, loss cannot be judged immediately, and only a timeout can settle it; in MetaRoCE every path has its own sequence space, a gap is no longer confusable with disorder, and a gap in the 256-bit selective-acknowledgment (SACK) bitmap is direct evidence of loss.
The semantics invert tradition here: in a traditional protocol the SACK's main job is to avoid resending data that already arrived; in MetaRoCE the bitmap gap directly triggers precise retransmission — only the missing packet is retransmitted, on the path that lost it, the moment the gap appears.
Congestion control is performed by both ends. The sender runs ECN-driven AIMD, additive increase and multiplicative decrease on every marked packet. The receiver does the other half: it knows the capacity of its inbound link and the number of active senders, and it returns the allocated bandwidth share in every acknowledgment, so the sender approaches the right rate directly instead of probing repeatedly. Many-to-one convergence traffic (incast) settles within one to two round trips for this reason: probing algorithms have to search back and forth among the senders and burn many more round trips before stabilizing, and direct share allocation does away with that process. AMD's account completes what the sender does once it has the rate: cap the transmission window and schedule traffic across the available paths. Windows are maintained per path and per connection: a congestion mark trims only the path that saw the congestion and steers subsequent packets toward idle paths. Fairness and tail latency improve together, and the price is that the state on the NIC grows from one copy per connection to one copy per path, part of the wholesale trade of NIC state for network simplicity.
The third pair: topology independence and connection structure. Everything the protocol asks of the network reduces to two features every switch already has: ECN marking and ECMP. It requires no packet trimming, no in-network telemetry, no credit-based flow control, and no switch-side spraying, and it does not conflict when switches offer these. Fat trees, multiplane fabrics, deep and shallow buffers, vendor clouds whose configuration you do not control: the same protocol covers them all, and the network is left free to optimize purely for cost and cabling. Set against the tuning route, the economic meaning of this point emerges: ByteDance and Alibaba rebuilt whole topologies for uniform distribution, and Meta itself iterated its routing through multiple rounds toward uniformity; MetaRoCE turns uniformity into a byproduct of spraying, and the cheapest switches will do.
The connection structure solves state explosion. A traditional RDMA Queue Pair (QP) carries message streams and bandwidth at once: a single QP is one ordered stream, one congestion window, one ECMP path; a single QP cannot fill the bandwidth, more parallelism means opening more QPs, dozens per node pair, each with its own congestion window, mutually invisible, hard to keep fair and hard to ramp together, and the state swells on the NIC with the degree of parallelism. MetaRoCE separates the two: a single connection carries many independent ordered streams above (one per communicator or collective operation) and many paths below, managed by one congestion controller. Connection state stops growing with the workload's parallelism. The application layer needs essentially no changes; existing RDMA Verbs APIs and software stacks keep working, with multiplane and other enhancements exposed through extension APIs.
The complete path of one collective operation. The application establishes a connection through the existing Verbs interface, and the collective library assigns one independent ordered stream inside the connection for each communicator or collective operation. Once a collective's data enters its stream, the transport segments it and sprays packet by packet along paths chosen by live telemetry; the UDP source port is the ECMP entropy, switches forward by standard hashing, and no special state is needed anywhere. Packets arrive out of order on standard Ethernet, and the receiving NIC handles each packet by the information the packet itself carries: Writes go directly to their final memory locations, Sends land in posted receive buffers by match information, and whatever arrives first becomes usable first.
Each path's acknowledgments carry the 256-bit bitmap and a bandwidth share, and the sender completes every anomaly handling from them. Packet loss: a gap appears in the bitmap, and the missing packet is retransmitted precisely and immediately on the path that lost it. Path congestion: an ECN mark trims only that path's window and steers subsequent packets toward idle paths. Link or plane failure: patterns of change in round-trip time and utilization distinguish congestion from failure, and the transport explicitly moves the traffic away; in live testing, traffic redistributed itself autonomously during simulated plane failures, with no application involvement and no operator intervention. Incast: the receiver tells each sender its available rate directly by share, and the burst settles within one or two round trips. By the end of this path, the switches have done exactly two things throughout: ECN marking and ECMP forwarding, with all the transport intelligence sitting in the NICs at both ends.
Validation results and boundaries. Hardware validation ran on AMD Pensando NICs, in a 64-node AMD GPU cluster, using the RCCL collective library, in direct comparison against RoCEv2. Four groups of results: all-reduce and all-to-all operations consistently achieved higher throughput and lower flow-completion times. At a 1% packet-loss rate the protocol sustained roughly 86% throughput (the original text does not state the baseline; quoted literally). The number has to be read against its premise: in RoCEv2's design a 1%-loss network simply does not exist, because stopping such a network from appearing is PFC's job; MetaRoCE treats that same network as a normal operating condition and still delivers close to nine-tenths of the throughput. Under extreme 10% loss it still delivered usable bandwidth and converged gracefully. Across 4-plane and 8-plane topologies with up to 4,000 concurrent connections, throughput grew linearly with the number of planes. During simulated plane failures, traffic redistributed itself autonomously. The boundaries deserve equally clear wording: a 64-node cluster is more than three orders of magnitude short of the million-GPU design target; the GPU count per node is undisclosed, as is the NIC the baseline RoCEv2 ran on. The software reference implementation serves strictly as a behavioral model; so far only one NIC family, AMD Pensando, has produced performance numbers, and the DPDK-optimized version does not arrive until the October summit. Fairness of receiver-side rate hints under thousands of concurrent senders, cross-region round trips at millisecond scale, and nanosecond-scale short messages within a rack all remain roadmap directions, with no measured numbers yet.
Three distance regimes and deployment requirements. In-rack (scale-up), accelerators exchange small messages where every nanosecond counts; MetaRoCE has already removed two latency sources (the reorder buffer and PFC), and a fast signaling path for short memory operations, sent directly from one processing element to another, is in progress. Within the data center (multiplane), plane selection belongs entirely to the NIC and the network needs only standard ECMP and ECN; the deployment list is very short: switches need essentially no changes, PFC can retire, and multiplane enhancements are opt-in through extension APIs. Across buildings (scale-across), a single job spans buildings thousands of kilometers apart, round trips stretch into milliseconds, and small differences between paths accumulate. Paths as first-class entities are the basis of adaptation: the transport prefers uncongested paths and seeks fairness at every level. Fair sharing of long-haul links remains an officially unsolved roadmap item. For storage and KV cache, distributed storage brings incast by nature: a single read fans out to a large number of servers and the replies surge back together. Receiver-side rate hints let whichever side is receiving, whether the storage server taking write requests or the client taking read results, directly moderate the inbound rate, whether the request went to ten servers or a thousand. The new difficulty is keeping those rate hints accurate in a network of varying speeds and requests of varying sizes. In one sentence: applications and switches both need essentially no changes; the change concentrates in the NIC. Once the October specification lands, deploying means adopting NICs that support the protocol, with the conformance suite as the acceptance basis. Whether existing programmable NICs can follow via firmware awaits vendor roadmaps; that is our path judgment.
5. A Multi-Angle Evaluation
The technical value concentrates in two design advances. The SACK bitmap inverts from "avoid resending" to "trigger precise retransmission," which, with per-path independent sequences, makes loss tolerance a first principle of the transport rather than a patch; the single connection carrying many streams over many paths under one congestion controller ends the QP-stacking state explosion. Zero new requirements on the network matter just as much: ECMP and ECN are features every switch already has, so deployment is not tied to any switch vendor's roadmap. The way the validation boundaries were disclosed deserves note: the gap between 64 nodes and a million GPUs, the undisclosed baselines, and the separation of roadmap items from measured numbers are all stated plainly, which is rare in new-protocol announcements.
The market value lies in establishing the pattern. Buyer-defined protocols move from OpenAI's single instance to a repeatable model; the largest buyers have discovered they can bypass vendor alliances and specify directly to the shape of their own clusters. A new product category with open specifications and clear acceptance criteria has appeared on the NIC side, while the switch side's differentiation space is further compressed. NVIDIA is squeezed from both directions: InfiniBand's lossless narrative faces head-on competition, and Spectrum-X's privately-paired story faces an open specification.
The practical value splits by audience. Those who can use it now run AMD GPU clusters with RCCL, the only validated hardware path, which per AMD is already moving toward 800G deployment. Those who wait: NVIDIA-cluster users (NCCL adaptation and NIC support have not happened), the October specification and DPDK release, and vendor productization pace. Who it suits: CSPs and large-cluster buyers should track it most closely, and even without adopting the protocol itself, the open pattern of buyer definition plus conformance-gated acceptance is directly borrowable; NIC vendors face a new track, and the earlier, the better; enterprise clusters can afford to wait for the ecosystem to mature; researchers already have a behavior model open to comparison. China's domestic compute ecosystem is a particular beneficiary: a directly usable specification, where commodity switches plus domestic NICs implementing the spec are enough to build a fabric whose transport layer depends on no single vendor.
6. Peer Review: The Transport-Layer Answer Sheets of Other CSPs
Put the peers on the same table and two routes emerge.
Tuning within the RoCE frame, represented by ByteDance, Alibaba, and Tencent. ByteDance's MegaScale system (NSDI'24) disclosed the full engineering of its ten-thousand-GPU cluster: 12,288 GPUs on a network built from Broadcom Tomahawk 4 switch silicon (25.6Tbps per chip, 64 400Gbps ports) in a three-layer CLOS-like topology with 1:1 downlink-to-uplink bandwidth. The transport is still RoCE; the optimization concentrates on four things: custom topology, reduced ECMP hash collisions, customized congestion control, and tuned retransmission timeouts, backed by observability and diagnostic tooling deep in the protocol stack to keep long training runs stable.
Alibaba's HPN (SIGCOMM'24) goes a step further but stays on RoCE: a two-tier, dual-plane architecture interconnecting 15K GPUs in a single pod, built on 51.2Tbps single-chip switches, with primary-backup dual ToRs removing single points of failure and the dual plane dissolving ECMP hash polarization under heavy traffic: LLM training puts only a few periodic large flows on each host, and ordinary ECMP's load imbalance is amplified sharply at ten-thousand-GPU scale. On top of that, per the paper's own account, the HPN 7.0 production architecture adds self-developed Solar-RDMA and the ACCL collective library, deployed at scale since September 2023, with Tongyi Qianwen 2.5 trained on it.
Tencent's Xingmai network, fully disclosed in June 2023: self-developed white-box switches and a network operating system, 3.2Tbps interconnect bandwidth, and a target of 100K-GPU clusters. On the transport side, the self-developed TiTa protocol coordinates endpoints and network, monitoring and regulating network congestion in real time; per Tencent's official claims it improves bandwidth load by 40% with zero loss under high load. The TCCL collective library, per official claims, cuts communication latency by 40%. The system as a whole, per official claims, improves GPU utilization by 40%. TiTa's endpoint-network coordination is directionally adjacent to MetaRoCE's endpoint intelligence; the fork is the frame: Xingmai still targets a lossless network, while MetaRoCE discards that premise.
Microsoft has no comparable self-developed transport layer disclosed; Azure's AI cluster networks run on commercial solutions. All three keep the lossless premise in place — protocol-layer innovation (custom congestion control, Solar-RDMA, TiTa) serves the lossless framework; what MetaRoCE changes is the premise itself. The operating cost of this route is one Meta's own SIGCOMM paper has already recorded once: multiple rounds of routing iteration, congestion control lifted into software, ordering relaxed, all of it inside the lossless frame, with the frame itself left untouched.
Rebuilding the transport, represented by SRD, Falcon, and MetaRoCE. The three are solving the same set of problems; their answers differ on two things: where reordering gets absorbed, and at what granularity packet loss is evidenced.
AWS's SRD is the route's earliest large-scale specimen, running privately on Elastic Fabric Adapter since 2019: reliable-datagram semantics, out-of-order delivery permitted, retransmission and congestion control performed by Nitro hardware, the Ethernet left in its natural lossy state, PFC no longer needed. AWS's own driver documentation states the motive plainly: reliable connected transport limits the number of outstanding messages, and all-to-all connectivity demands piles of queue pairs; SRD removes both constraints at once. Elastic Network Adapter (ENA) Express later carried it into ordinary instance networking, with official figures of up to 25Gbps for a single TCP flow and up to an 85% reduction in P99.9 tail latency.
Google's Falcon serves the company's own clusters, runs privately, and has had its design disclosed through OCP (October 2023): one connection splits into multiple subflows, the sending NIC rewrites the IPv6 flow label to choose a subflow for each packet while switches hash as usual, a path granularity one step coarser than per-packet spraying. The receiving NIC uses an on-chip buffer of one to two square millimeters to reorder out-of-sequence packets back into order, with a 128-bit acknowledgment bitmap; congestion control runs on delay signals. RDMA and NVMe map onto a unified push-pull transaction interface as upper-layer protocols.
UEC 1.0 (June 2025) and the MRC write this route into alliance standards: UEC rebuilds from scratch, swaps the API stack, and waits for its hardware ecosystem to ramp through 2026 and 2027; the MRC, led by OpenAI, uses SRv6 source routing to pull path control into the endpoints, requiring corresponding forwarding support in switches, whereas UEC and MetaRoCE share the same minimal switch requirement, ECMP plus basic ECN. Meta is simultaneously a founding member of UEC and the prime mover of MetaRoCE.
The two routes carry different bets. Tuning wins on certainty and zero migration: the ecosystem stays as is, and the risks are known. Rebuilding has the higher ceiling, betting that lossless-network operating costs rise super-linearly with scale.
| Approach | Openness | Transport-layer change | Where reordering is absorbed | Validation or production scale |
|---|---|---|---|---|
| AWS SRD | Private | Rebuilt from scratch | Endpoint, passed to upper layers | In production for years |
| Google Falcon | Private (design disclosed via OCP) | Rebuilt from scratch | On-NIC reorder buffer | TPU-cluster production |
| ByteDance MegaScale network | Paper public, protocol private | No protocol change; tunes topology/ECMP/congestion/timeouts | Not absorbed (ordering premise kept) | 12,288 GPUs in production |
| Alibaba HPN / Solar-RDMA | Paper public, protocol private | Self-developed transport stack, lossless frame kept | Not absorbed (dual-plane load balancing) | 15K GPUs in production, Tongyi Qianwen training |
| Tencent Xingmai / TiTa | System publicly disclosed, protocol private | Endpoint-network-coordinated protocol, lossless frame kept | Not absorbed (lossless plus end-network coordination) | 3.2Tbps, 100K-GPU design target |
| UEC 1.0 / MRC | Open specifications | Rebuilt from scratch | Receiver-side handling defined by spec | Hardware ecosystem ramping 2026-27 |
| MetaRoCE | Spec + reference implementation + acceptance suite fully open | Rebuilt from scratch | Direct memory writes; reordering eliminated | 64-node validation, path to 800G deployment, specification lands in October |
MetaRoCE's unique position in this table: the only implementation that combines rebuilding the transport with full openness and an acceptance suite, defined personally by a million-GPU-scale buyer. At the mechanism level its divergence from Falcon is the most representative: Falcon reorders inside the NIC, paying with an on-chip buffer plus a 128-bit acknowledgment bitmap; MetaRoCE deletes the reordering step itself through direct memory writes, with a 256-bit bitmap maintained independently per path. Its divergence from SRD is openness: the same endpoint intelligence, one locked inside Nitro hardware, the other handed over with specification and acceptance tooling alike. For Meta itself it is a bridge and a hedge: before UEC hardware reaches volume, MetaRoCE first takes on the demand of the Prometheus cluster generation.
The risk lives here too: three open lines with overlapping engineering effort could fragment the camp first. Meta's priorities are not hard to read — beat InfiniBand first, then settle the internal contest. The list of those forced to respond is long. AMD has already bet on multiple lines, with the same programmable-NIC route serving MRC, UEC, and MetaRoCE. NVIDIA itself participates in MRC, betting both sides, and whether it will open up equivalent Spectrum-X capabilities remains unresolved. Broadcom and Intel must each take a position across both MRC and MetaRoCE.
7. Summary and Judgments
MetaRoCE overturns RoCEv2's three premises with three mechanism pairs, validates the direction on 64 nodes and multiplane topologies, and opens the specification, reference implementation, and conformance suite at the October OCP summit. The scale is real, the openness is real, and so are the boundaries of the validation.
Judgment one: endpoint intelligence has completed its industry convergence. SRD, Falcon, Spectrum-X, UEC, MRC, and MetaRoCE point the same way, and the countdown for PFC in new deployments has begun. The network used to bend to the protocol; now the protocol bends to the network.
Judgment two: the buyer-defined protocol is now a repeatable model. MetaRoCE's decisive factor is the ecosystem time window: within the window before UEC hardware matures, can it pull in enough NIC implementations? The same-day release of MTIA 300 shows the path from protocol into first-party silicon, but the protocol's endpoints are not limited to Meta's own chips.
Judgment three: the biggest beneficiaries of an open transport layer are all those building AI clusters on commodity switches. The network's differentiated value retreats further into NICs and silicon, the switch industry accelerates down the commodity curve, and the NIC becomes the new contested high ground.
Four things to watch next: the launch and follower list at the October OCP Global Summit; production data from the Prometheus cluster, queryable from 2027; NVIDIA's countermove; and the first signal of MRC, UEC, and MetaRoCE converging or forking.
Related Reading