Why Can “It Runs” Still Be Wrong?

Turning correctness verification into a reproducible path

Abstract: A migrated program that keeps producing output with normal latency is not necessarily semantically correct. This piece discusses two blind spots and gives one acceptance path:

• Structural errors: identical tensor shapes and normally returning functions can still pair every component wrongly (the public RoPE case);
• Runtime identity: the code you think you are running and the code actually running may not be the same (the compile-cache signature chain);
• Acceptance method: first separate normal error from semantic error, numerical comparisons with tolerance and semantic invariants checked exactly; then three layers of checks (operator numerics, fixed token sequences, task quality) and a five-step path (fix the baseline, compare operators, localize by sequence, regress by task, verify the environment).

Only when the whole chain is walked can something be called “verified correct.”

The first four parts were about performance: resources, layouts, communication, attention. This one is about a quieter problem: when everything appears to run normally, how do you discover that the migration has already gone wrong? And the reverse: when two implementations’ outputs are not exactly equal, how do you tell normal error from real mistakes?

Deployment failures come in two kinds. One is easy to find: the program errors out, and the stack points at the problem. The other is hard to find: the program keeps producing output at normal latency, but some data layout, position encoding, or scaling handling is wrong. The model is “working while sick”: output still looks plausible while quality quietly drifts. This kind of failure is dangerous precisely because routine monitoring watches latency and throughput; those numbers stay normal, and only the semantics are wrong.

---

1. Blind Spot One: Same RoPE, Different Pairing

This is not hypothetical. DeepSeek-V3.2-Exp’s November 17, 2025 update note disclosed an implementation difference in the inference examples: the indexer requires the non-interleaved RoPE layout while the MLA module uses the interleaved layout; the issue could affect model behavior and was subsequently fixed (DeepSeek-V3.2-Exp repository).

A hand-computable example shows what “different pairing” means. Under both layouts the RoPE math is identical: components of the q and k vectors are paired two by two and rotated by position. The difference is only which components form a pair.

Take the 4-component vector [1, 2, 3, 4] and, for demonstration, rotate every pair by 90°. This demonstrates the pairing rule only, not the model’s real RoPE frequencies.

The two outputs have the same shape, both squared sums are 30, and every component differs. Feed interleaved-arranged data to a RoPE implemented as non-interleaved, and the function still returns normally, the output still a finite value that “looks position-encoded.” But every pair rotates the wrong components.

Shape checks, NaN checks, and simple Q&A can all still pass. A constructed contrast on fixed inputs, like the one above, catches this structural misalignment directly. When constructing the contrast, avoid degenerate inputs such as all zeros, all equal, or zero positions: they happen to mask the difference.

The case’s value is not “the project once had a bug”; every active project has fixed bugs. Its value is marking a blind spot of migration acceptance: identical tensor shapes and normally returning functions do not prove semantic correctness. Constructed contrasts on fixed inputs are exactly what acceptance can use to catch this class. The official note covers the issue, its impact, and the fix; how it was discovered is not in public materials.

---

2. First Separate What You Are Looking For: Normal Error versus Semantic Error

Numerical differences between pre- and post-migration results are certain; not every difference is an error. Two sources of difference are expected:

• Precision configuration. The same operator on an FP8/FP4 path, or with different accumulation precision, carries a corresponding error magnitude. The backfill matrix multiplication in Part 4 that turns HF32 off for precision is an example of an explicit trade-off: the precision requirement changed both the execution path and its cost.
• Reduction and scheduling order. Different orders of distributed reduction or tiled accumulation change the low-order bits of floating-point results. DeepGEMM even ships a determinism switch; the project itself treats “do same inputs produce same outputs” as a configurable property.

Opposite them are semantic errors, for example:

• wrong pairing rules (Section 1’s case);
• reversed layout conversion direction (the MN-major direction is in Part 2);
• scaling factors applied at the wrong place or the wrong number of times;
• wrong offsets when reading indices.

Raising precision cannot fix a wrong pairing rule: at any precision the computation is still the same misalignment. Separating the two therefore relies on structural checks and inputs that expose differences, not on “does the error shrink as precision rises.”

At the same time, an explainable source is not automatically acceptable. Deviations from FP8/FP4 and reduction order must still fall within the task’s agreed numerical and quality tolerances. “Low precision is inherently lossy” does not license accepting deviations of any size.

With the two separated, the acceptance methods differ accordingly:

• Numerical comparisons carry tolerance, determined by the precision configuration. The concrete values come from the measured spread between the precision path and the reference implementation; no universal threshold fits everywhere.
• Semantic invariants are checked exactly. Pairing conventions, layout direction, index offsets, scaling counts: right or wrong, nothing in between.

This release happens to provide ready references: DeepGEMM’s tests cross-validate the corresponding GEMMs against aclnn reference implementations; DeepEP’s tests cover expert alignments 1 and 128 and check expert intervals and padding regions. DeepEP’s two checks examine exactly structural invariants, not just numbers.

---

3. Three Layers of Checks: What Each Covers

On top of that distinction, correctness checking has three layers, each covering a different scope.

Layer one: operator numerics. With fixed inputs, compare absolute and relative error between the reference and target implementations, across shapes, lengths, and boundary conditions. Deviations at a few positions can hide under averages, so read where the extremes fall and how they distribute.

Layer two: model computation on a fixed token sequence. Feed both implementations exactly the same token sequence and compare intermediate results and logits step by step, layer by layer. Fixing the input separates the problem from sampling randomness; the normal fluctuation of temperature sampling and top-p no longer drowns the signal. The layer where the first clear deviation appears narrows the search from “the whole model” to that layer and its input handling. Section 1’s RoPE difference is exactly the kind that surfaces here, because it is always wrong at a specific module’s specific positions.

Layer three: task quality. Score changes on real tasks. It tells you “something may be wrong,” never “where.” An unchanged score does not mean nothing is wrong; some benchmarks never exercise behaviors that would expose such position-encoding errors, so the score stays put.

---

4. Blind Spot Two: The Compile Cache’s Signature Chain

Correctness has one more hidden source: the code you think you are running and the code actually running may not be the same.

DeepJIT’s runtime path. It can be verified directly against source (commit 3732a3b9, runtime.hpp):

1. construct the runtime (Runtime): collect the dependency signatures and compiler information the application provides;
2. compile: merge compile options and compute the cache key;
3. look up the cache: first the loaded in-memory cache, then the disk cache; on a miss, call the backend compiler, store the artifact, and load it;
4. launch: run with the merged launch options.

CUDA and Ascend share this template, with different device sources and backend options.

What the cache key covers. Source, tracked headers, compiler version, effective compile options, and dependency signatures. The operative words are “tracked”:

• the header parser tracks only angle-bracket includes whose paths match a configured prefix; quoted includes are rejected.
• Implicit inputs do not enter the signature automatically, such as NVCC_PREPEND_FLAGS, CPATH, and the host compiler choice under CUDA. These must be provided by the caller as explicit dependency signatures.

What genuinely slips through silently is the third case: an angle-bracket include that matches no configured prefix is accepted by the compiler but ignored by the parser. Hence a silent failure of the same family as Section 1’s: you modify such a header, it never enters the signature, and the cache still hits the pre-modification binary. The modification effectively never happened.

The cache also has an in-memory layer. The README is explicit: digests of tracked headers are cached for the lifetime of the runtime. Modify a tracked header and only a runtime rebuild recomputes the digest; clearing the disk cache cannot replace already-loaded kernels and parser caches.

Configuration is snapshotted when the runtime is constructed. Later changes enter by reconstructing the runtime, updating the tracking set, or adding dependency signatures, not by editing configuration in place. This layer of state follows the runtime object, not the process boundary: rebuilding the runtime within the same process refreshes it, and absent configuration changes there is no need to clear caches routinely.

Hence four questions for verifying binary identity:

1. Which compiler was used?
2. Which compile options were in effect?
3. Was this a fresh compile or a cache hit?
4. Did the modified dependencies enter the signature?

---

5. Closing into a Reproducible Verification Path

Combining the sections, for this batch of components one can write a step-by-step acceptance path, not just a checklist of cautions.

Step one: fix the baseline. Choose the reference implementation and version, whether the aclnn reference, upstream reference kernels, or the pre-migration implementation. Record the environment: compiler and options, cache hit state, the determinism switch setting, and each library’s actually effective environment-variable values.

Step two: compare at the operator layer.

• Verify conversion outputs item by item under Part 2’s layout agreements; whether the scaling-factor layout conversion is correct is itself a thing to test.
• Following DeepEP’s test pattern, check dispatch intervals and padding regions.
• Set tolerances for attention and matrix multiplication by precision configuration, with extra tests at boundary shapes.
• Compare semantic invariants (pairing, direction, offset, count) exactly, item by item; judge numerical items against the measured spread of that precision path, within tolerance or not.

Step three: localize at the sequence layer. Fix the token sequence and compare intermediate results and logits layer by layer. The first clear deviation narrows the search to that layer and its input handling; return to step two and test both. Note: the layer where the deviation appears is not necessarily where the root cause lives; its input may already be wrong.

Step four: regress at the task layer. Run a fixed evaluation set as a regression indicator, not a localization tool. Fixed tasks still fluctuate with evaluation and sampling, so record the spread of repeated measurements and allowed error:

• differences explainable by a known precision path are classified “explained”;
• differences that cannot be explained and affect quality stay open; do not force a classification.

Step five: verify environment identity. On every upgrade or dependency change, answer Section 4’s four questions again. After configuration changes, besides isolating or clearing the cache directory, confirm the runtime has been rebuilt, because clearing the disk cannot swap out in-memory kernels and digest caches. DeepGEMM and DeepEP each carry their own environment-variable prefixes (DG_JIT_, EP_JIT_, falling back to DJ_JIT_ when unset; both READMEs state this fallback rule). With multiple libraries coexisting, confirm each component’s cache state separately.

Every step of this path uses resources the open-source batch itself ships: reference implementations, structured tests, cache metadata; the user need not invent acceptance from zero. But the path will not run itself; running it through is the migrating side’s own work.

Classify the error, finish the three layers, verify the signatures, record the environment; only when the whole chain is walked does “appears to run” become “verified correct.” The next part turns to economics: throughput up 20%, so why might total cost rise?

---

Sources and computation notes: This piece is based on the projects’ public repositories and documents, fact-checked as of October 7, 2026.

• The indexer RoPE layout difference (non-interleaved required, MLA interleaved; November 17, 2025 update note; fixed) comes from V3.2-Exp’s official update record; how it was discovered is not in public materials.
• DeepJIT’s runtime path follows commit 3732a3b9 (runtime.hpp). The header-parsing rules (angle-bracket and prefix-matched to track; quoted includes rejected; prefix-mismatched angle includes accepted by the compiler but ignored by the parser), implicit inputs not auto-discovered, and “header digests cached for the runtime’s lifetime; modifying tracked headers requires a runtime rebuild” come from the DeepJIT README.
• DeepGEMM’s aclnn cross-validation and determinism switch, and DeepEP’s checks on expert intervals and padding regions, come from their respective READMEs; the DG_JIT_/EP_JIT_ fallback to DJ_JIT_ likewise comes from both READMEs.
• The 4-component 90° rotation contrast ([1,2,3,4] → [−2,1,−4,3] / [−3,−4,1,2], squared sums both 30) is this piece’s constructed illustration, demonstrating only the pairing rule.
• The three-layer check, the four questions, and the five-step path are this piece’s methodological framework, not the projects’ own statements. Tolerances must be determined from the precision configuration and the reference implementation’s measured spread.

No hardware measurements were made. This is Part 5 of the DeepSeek Ascend open-source series.