The SubQ Three-Month Audit: The Technology Is Real, the Narrative Ran Ahead

On May 5, Subquadratic released a model, SubQ, claiming a 12-million-token context window and attention computation cut nearly a thousandfold versus dense architectures. The most-circulated verdict in the Chinese community: "This is either the biggest architectural breakthrough since the Transformer, or the AI version of Theranos. There is no middle ground."

Three months later, on August 20, SubQ's technical report landed on Techmeme as a sponsored post, and the distance between the claims and the facts could be measured for the first time. This piece proceeds in five steps: first, the claims as made and the skepticism that met them; second, what actually happened (what the company did); third, where SSA sits among the long-context technical routes; fourth, an item-by-item evidence check; fifth, technical assessment and prediction. The conclusion up front: the technology is real, the narrative ran ahead of it, and the publicly verifiable parts are still owed.

First, the objects of the audit. What shipped in May was generation one (SubQ 1M-Preview: the production model, plus its 12M research result); what shipped on June 16 was generation two, SubQ 1.1 Small, the subject of that 22-page technical report: four named authors (Saul Ramirez, Alex Whedon, Ashmal Vayani, Phong Vo), in model-card form, about six weeks late. The third-party evaluation firm Appen tested this generation as well. May's skepticism hit generation one; August's evidence comes from generation two. Cross-generation comparisons are recorded as "switched tracks," never as "disappeared."

1. The Claims

The launch post was signed by CEO Justin Dangel. Unpacked, the claims are seven:

• RULER 128K (RULER: a multi-task long-context retrieval benchmark suite) at 95.6%, third-party verified, ahead of Opus 4.6's 94.8%;
• "attention 52× faster than FlashAttention (an architecture-level comparison), while using 63% less compute" — two numbers bound in a single sentence;
• MRCR v2 (MRCR: a multi-round coreference-resolution long-context benchmark) with two scores: 83 for the research result, 65.9 for the third-party-verified production model;
• SWE-Bench Verified 81.8, above Opus 4.6 and DeepSeek 4.0 Pro (comparison scores per the official site's original text; the media retold a version with 81.42, which doesn't match — no explanation);
• 12M as a "research result," not a product capability;
• the architecture narrative ran on the words "ground-up redesign, designed from scratch, from first principles";
• a $29 million seed round, and a team of 11 PhD researchers.

The skepticism arrived in full within three days (compiled from Chinese-community retellings): the technical report and the model card both said "next week," with a prior record of slipped dates; the official About page listed only institution names — Meta, Google, Oxford, Cambridge, and the like — "not a single name attached"; and $29 million was nowhere near enough to train an Opus-class model from scratch. The most specific inference came from an OpenAI engineer: most likely not a new model trained on a new architecture, but an existing Transformer given an attention replacement, on a path close to DeepSeek's sparse attention. He added a rider: "It very likely doesn't work at all, and it might just be fake." Another outside researcher made a joke of it: "The probability that this is a scam is rising subquadratically."

2. What Actually Happened

The report was delivered: June 16, 22 pages, in model-card form, about six weeks late. But the most important content in the report is not the scorecard; it is a methodological confession:

"Rather than training a new model from scratch, we converted an existing open-weight frontier model by replacing its dense attention with SSA."

In plain terms: rather than training a new model from scratch, they converted an existing open-weight frontier model, replacing its dense attention wholesale with SSA (Subquadratic Sparse Attention, a content-dependent sparse attention).

The OpenAI engineer's inference landed on every count. The $29 million question resolved with it: they never took the retraining route — conversion is far cheaper than training.

But the collision sits here too. May wrote "ground-up, designed from scratch"; June admitted a conversion of an existing open-weight model — two descriptions of the same thing, six weeks apart. Which base model, the report does not say.

The recipe, in four steps:

1. Take an open-weight frontier model as the base (which one is not disclosed);
2. Replace dense attention with SSA. Mechanism: for each query, attention positions are selected by content relevance, and exact computation happens only at the selected positions; selection, retrieval, and attention are each linear in sequence length;
3. Extend the context in stages — 262K→512K→1M→2M — with YaRN (a positional-encoding rescaling method) adapting the model to each new length, and long-text continued pretraining interleaved between extensions;
4. Roughly 1T tokens of continued pretraining, on a corpus of naturally long text (books, documents, repo-scale code).

After that: 100-plus experiments, six to seven generations of model iteration.

The engineering bill is not small. Beyond attention itself, multi-million-token training remains a memory problem: activations, optimizer states, and the sequences themselves all have to fit the GPU memory budget. The report's solution is a "memory ladder": hybrid-sharded data parallelism to start, then sequence parallelism, CPU offload (nested included), and multi-node sequence parallelism deployed in stages, with Ring Attention brought in above 2M. When packing long sequences, cross-document attention boundaries are not masked — the same practice as DeepSeek-V3 and UltraLong; in post-training, the team also explored aggregating the loss per sample, reducing the dominance of extremely long examples in gradient updates. The key sentence: no off-the-shelf distributed technique can carry SSA efficiently as-is; every component had to be adapted to the irregular memory-access patterns introduced by content-dependent selection. That is the real engineering weight the retrofit route hides behind its recipe.

3. Route Analysis and Technical Analysis

3.1 Quantifying the Cost

Dense attention has computational complexity O(n²): per-layer FLOPs grow with the square of sequence length. The report gives per-tier numbers: 8.6×10⁹ operations per layer at 128K, 5.49×10¹¹ at 1M, and 2.2×10¹² at 2M. These are self-consistent with the square law: from 128K to 1M the length grows ×8 and the computation ×64 (8.6×10⁹ × 64 ≈ 5.5×10¹¹).

FlashAttention needs precise placement: it is an IO-aware kernel optimization that compresses attention's memory footprint from O(n²) to O(n), but it does not change the quadratic complexity of the FLOPs. It made long-context experimentation feasible; it did not make long context cheap.

3.2 The Design Space: Comparing the Optimization Directions

Saving compute is the field's consensus; the disagreement is the selection mechanism. In the report's words: "The challenge has never been sparsity itself. The challenge has been deciding which interactions to keep." The published optimization directions fall into five families, each trading away something different for complexity:

• Fixed-pattern sparsity (sliding-window and strided attention, the Longformer/BigBird line): pruning by position meets the complexity target but is content-independent. A token can only attend to the positions the pattern allows; relevant information outside the window is invisible. Production models such as Gemma use it.
• KV latent compression (MLA, Multi-head Latent Attention): K/V are compressed into a learned latent representation; what it saves is inference-time KV-cache memory and bandwidth. But prefill's quadratic computation is left untouched. The report's verdict: it solves a different problem.
• Systems-level avoidance (RAG — retrieval-augmented generation — agentic orchestration, RLMs — recursive language models): retrieval is moved outside the model. These ship, but in the report's landscape table, the "retrieval from arbitrary positions" column counts as only partially satisfied.
• Linear attention / SSM (linear attention, Mamba, RetNet, RWKV): abandons the attention matrix for a fixed-size recurrent state, O(n) complexity. The Mamba-2 result established that SSMs and linear attention are two parameterizations of the same mathematical object. The cost: state compression is lossy — adequate for summarization-class tasks, not for exact reproduction.
• Hybrid architectures (Jamba, Kimi Linear, Qwen3-Next, Nemotron v3): mostly SSM/linear layers with a few full-attention layers inserted for retrieval. A constant-factor improvement that does not change the asymptotic shape, and the attention layers are load-bearing, so their share cannot be pushed down indefinitely. Footnote: MiniMax M1 shipped hybrid; its successor frontier model M2 returned to full attention.

A sixth family is the branch SubQ shares: learned sparsity (NSA/CSA, the DeepSeek line), where the model learns where to look — content-dependent, but NSA's selector (the Lightning Indexer) carries quadratic overhead of its own. SubQ's SSA sits on this branch; its radical bet is making the selection step itself linear, which is the subject of the next subsection and the divide with DeepSeek.

3.3 SSA's Mechanistic Claim

The report draws the entire field as one tradeoff table (Figure 4): eight routes against four properties (sub-quadratic scaling, content-dependent routing, retrieval from arbitrary positions, production maturity), with every existing family missing at least one column. SSA's claim takes all four columns: sparsity is decided by content (the relevant information can be at any position), and the cost of the selection step itself must also be linear — otherwise what attention saves is eaten back by the selection step.

The divide with DeepSeek is the core of this claim (note: this is a competitor's view). The report first concedes that NSA (Native Sparse Attention) with its Lightning Indexer for learned routing is an "important shift," then criticizes in specific terms: the Lightning Indexer is itself a distilled full-attention model; its scoring overhead is quadratic and overtakes the main attention it serves at around 52K tokens. Plugged into DeepSeek v3.2: at 1M tokens, the indexing overhead runs about 16.1× the main attention; at 12M, 190.4×. CSA (the route that moves retrieval onto a compressed representation) cannot get around the same indexer. The report's summary: the routing mechanism born to make long context affordable became the main cost of long context itself. SSA's corresponding claim is to make the "selection" step itself linear, introducing no indexer with quadratic overhead. The report also notes that SSA can run on the raw token stream or on a compressed representation (the latter only partially explored, listed as future work), so the divide is not raw versus summarized input but the cost of the selection step. The two are headed the same way: DeepSeek has already vouched for learned sparse attention with a product, and SubQ pushes the claim on selection-step complexity further.

The full verdict on the SSM/hybrid line (fixed-size state as lossy compression; the MiniMax M2 return-to-full-attention footnote) is in the direction comparison in 3.2; the report treats it the same as fixed-pattern sparsity: complexity target met, retrieval quality not guaranteed.

One boundary must go on the record: the report states plainly that "the mechanism by which SSA meets these requirements is outside the scope of this report." This is a model card accounting for motivation, recipe, and results — not an architecture paper disclosing the algorithm. How the selector achieves linearity cannot be verified from the report by anyone outside the company.

4. Evidence Check

The audit of the seven claims, verdict table first: two honored, three landing at half a step, one track switch, one unmet.

The report's self-reported, checkable results fall into two groups.

Retrieval generalization is the hardest set. Training ran mostly at 1M; at inference, retrieval held through 12M — six to twelve times beyond the training window. The price is also the mechanism: throughout, only 0.13% of token pairs are attended to, which works out to roughly 770× attention pruning (the full pairing at 12M ≈1.44×10¹⁴; 0.13% of that ≈1.9×10¹¹; the report's "near a thousandfold" is rounding — 0.1% would be exactly a thousandfold). NIAH (needle-in-a-haystack, single-needle retrieval) scores: 100% inside the training window, 98% outside. The 13-task composite RULER 128K came in at 99.12. The report attributes the generalization to the selection criterion depending on content relevance rather than positional patterns.

Compute cost is the second set. Table 1 lists attention FLOPs per tier: dense attention quadruples with every doubling of length (0.25→0.99→3.9→15.8→63→252 PFLOP), while SSA roughly doubles (0.12→0.25→0.49→0.99→2.0→3.9). Quadratic versus linear. In Figure 12, SSA crosses FlashAttention-2 at about 16K; beyond that, the ratio stretches from 2.1× at 32K to 64.5× at 1M: the advantage grows with length; it is not a constant speedup. At short context SSA is barely ahead; long context is the operative regime. The cost is that short-context capability gives ground: GPQA 85.4, self-assessed by the report as "below mid-tier frontier models" (GPT-5.5 at 93.2, Opus 4.8 at 92); LiveCodeBench 89.7, close to the frontier.

Cross-checking the plausibility of the results. Any one group of numbers can be doubted on its own; three independent measurements interlock:

First, the pruning ratio is consistent with the compute curve. The 0.13% figure is the 12M-tier reading, working out to roughly 770× pruning; 64.5× is the measured 1M-tier ratio. Dense is quadratic, SSA is linear, so the ratio itself grows linearly with length: extrapolated to 12M per Table 1, 64.5×12 ≈ 770 — the same order as the pruning ratio. The two numbers are readings of the same curve at different tiers. May's "thousandfold reduction" was effectively a 12M-tier claim; measured up to 1M, the ratio is 64.5×. How the total splits among selection, retrieval, and aggregation, the report does not break out.

Second, the pruning ratio is consistent with retrieval quality. After 770× pruning at 12M, NIAH still holds at 98%: if the selector were missing key positions at 12M, retrieval should fall off a cliff rather than ease from 100% to 98%. The gentle slope is consistent with the report's attribution — a position-pattern mechanism would show a cliff, not a gentle slope.

Third, the cost is symmetric with the gain. At 32K the Table 1 ratio is only 2.1× (barely ahead at short context), and GPQA 85.4 sits below mid-tier frontier models: the saved compute did not appear from nowhere — it was traded out of short-context precision. Gains growing monotonically with length while costs concentrate at short context is itself circumstantial evidence that the sparsity mechanism is real.

These three cross-checks establish plausibility, not truth: all three groups of numbers originate from a single report. The real test remains independent replication (see Chapter 5).

Third-party verification exists, but at a discount. Appen's methodology is fully disclosed: NIAH in its most basic single-needle variant, 50 samples per tier, key parameters public (temperature 0, for one), plus a coding benchmark with its setup disclosed; the model under test was subq-2m-preview-small. But Appen was engaged by Subquadratic, and a commissioned evaluation is not an unpaid replication. The MRCR v2 account is also entered in full: 83 for the research result, 65.9 for the third-party-verified production model — below GPT-5.5's 74, several times above Opus 4.7's 32.2. On this benchmark the frontier models have always spread wide; reading position off a single comparison score will misplace you.

Efficiency wording: half a step. The speed line holds: May said 52× (an architecture-level comparison), June narrowed it to 56× @1M, from a single attention-layer measurement on an H100 — SSA at 966 ms against FlashAttention-2's 54,164 ms; more qualifiers, if anything, make it more honest. The compute line hangs: May's "63% less compute" converts to about 2.7×; June's is "64.5× less compute" — the same fact, about 24× apart, with no explanation in the report.

What cannot be independently verified. The API remains in private preview, the SubQ Code plugin is waitlisted, and the official line is GA by year's end; product capabilities such as "12M context, linear cost" cannot yet be tested hands-on. SWE-Bench 81.8 no longer appears in generation two: May listed it, the June report does not, and Appen switched to LiveCodeBench, its rationale naming SWE-Bench as "easier to game." Small-size models do not routinely take this benchmark in the first place — it may be standard practice, or the May number may never have been stable; the evidence does not settle it. Recorded as a track switch.

The skeptics get an audit line too. "It doesn't work at all" missed: there is now a signed report, a commissioned evaluation, and a product iteration track. The "Theranos, pick one" framing asked the wrong question: most industry truths live in the middle state. "Not a single name attached" lands as half a miss: the May launch post itself carried the CEO's signature and a named bio for CTO Alex Whedon, plus an investor list with links; the report's signatures came after, and the only thing that never changed is the About page.

5. Assessment and Outlook

Economic assessment. The $29 million turned from a suspicion into an argument. If long-context capability can be obtained via "open-weight base + attention replacement + 1T continued pretraining," then the algorithm-layer attack on the memory wall is democratized and needs no frontier-lab budget. That matters far more than the title of "first sub-quadratic frontier model." May's "ground-up" was spin; June's admission of retrofit instead proved the route's economics: hype the wrong direction first, then do the thing right.

Technical assessment. The technology is real: retrieval generalization, the shape of the compute curve, and third-party verification interlock, and the recipe is self-consistent. The narrative ran ahead: the ground-up/retrofit collision, the roughly 24× wording jump between 63% and 64.5×, and the silent exit of SWE-Bench are two faces of the same character. The publicly verifiable is owed: mechanism details undisclosed, no hands-on product.

Four items outstanding. Public hands-on testing after year-end GA; unpaid third-party replication, above all of the 12M generalization; end-to-end benchmarks (56× is a single attention-layer number); and pricing made public ("cost below 5% of Opus" comes from May's Chinese retellings — the official site never claimed it; by price the figure is 10%, by compute 1.5%). The wording jump between 63% and 64.5× also needs an explanation.

Conditional prediction. If 12M retrieval is independently replicated, what takes the hit is not just the long-context model rankings but RAG's fragmentation paradigm across whole-repository settings: when a model can retrieve faithfully inside a single context, the architecture that chops first and stitches back together loses its reason to exist. That premise gets its answer at year's end.

Of the memory wall's four lines of attack, SSA is the cheapest (the algorithm layer). The other three sit at the silicon layer, the co-design layer, and the systems layer, mostly landing in 2027. That is the next piece.

Disclosure: This piece draws on the Subquadratic official site and the SubQ 1.1 Small technical report (June 16, 2026, 22 pages, read in full), the Appen third-party evaluation brief (June 16, 2026, noted as commissioned by Subquadratic), the original May 5 launch post (retrieved August 20, 2026), and the community skepticism record compiled from public Chinese-community retellings in early May (original posts not revisited item by item). The views of the OpenAI engineer and the outside researcher are retold through the community; the original posts were not accessed. Multipliers and percentages are transcribed per their original wording, with measurement boundaries as marked in the text. This is not investment advice. Data in this piece is current as of August 20, 2026.