Follow the vendor docs, and you’ll be three times slower

You have a DGX Spark. You download Qwen3.6-35B-A3B in NVFP4, read the model card, paste in the launch command. Done.

Except there are three NVFP4 quantizations of the same model, from two different organizations, and the two cards give contradictory launch commands for the very same hardware. Not a nuance: one explicitly forbids exactly what the other prescribes.

We measured it. On this machine, the wrong choice is a 2.0–3.3× slowdown. The full backend measurement matrix on GB10 is public.

The situation

NVIDIA’s own NVFP4 quantization model card has a dedicated DGX Spark section. The gist of the recommended launch:

--attention-backend flashinfer --moe-backend marlin

For its own NVFP4 quantization of the same model — also naming the DGX Spark explicitly — Unsloth says do not use the Marlin backend, because it’s twice as slow. Instead:

--moe-backend flashinfer_b12x --linear-backend flashinfer_b12x

Same GB10 chip. Same base model. Same quantization format. Opposite instruction.

One of them is wrong. The only question is which one — and by how much.

What we measured

Three NVFP4 checkpoints and the production FP8 reference, on one machine (GB10, 128 GB unified), vLLM v0.24.0, with a pinned image digest:

The only difference between uns and uns-fast: in the former, the experts of the last eight layers stayed in FP8, +2.65 GiB. But nv and uns-fast are byte-for-byte twins — 30841 vs 30840 packed FP4 tensors, the same 180 weights left at high precision. We’ll come back to this, because it’s the most uncomfortable number in this article.

The backend-sweep configuration: one model (uns-fast), one instance, GPU_UTIL=0.85, 262144 context, max_num_seqs=16, speculative decoding off, 60-second measurements at c=1, 2, 4, 8, 16 concurrency, on both backends. After every run we verified from the server log that the backend that actually loaded was the one we asked for — because vLLM silently falls back to another one, and then you’re not measuring what the header claims you are.

The result

Ten measurement points, and not a single one where b12x wins. No crossover point, no load range where Unsloth’s recommendation would be worth it.

I didn’t expect this. In the earlier vendor-recommended rounds, b12x once looked better on a single stream — but that was per-token decode speed, not sustained throughput. As soon as we measure sustained throughput, marlin is already twice as fast even at c=1. The “it wins on single-stream after all” life raft doesn’t exist either.

A side note: b12x is also unstable. The first sweep gave 44 tok/s at c=8, the repeated run gave 86 — a 2× spread on the same configuration. marlin at the same point gave 276 and 283, i.e. within 2.5%. Not just slower, but less reliable too.

Why? — the hypothesis that failed

We had an appealing explanation. At startup, the FlashInfer autotuner tunes “buckets” for certain tensor shapes, and if the batch shapes that arise at runtime fall outside the tuned range, it falls back to a spare kernel. The startup log of the NVIDIA checkpoint even shows such a warning, literally mentioning a perf cliff and recommending you widen the tuning range.

It was obvious: at c=1 we’re still in the tuned range, at c≥4 we’re not, hence the cliff.

That’s not it. Across the server logs saved during the full sweep, the number of occurrences of the cliff pattern was zero on both backends. The count of JIT-compilation warnings is also identical (5 and 5), and those belong to the shared DeltaNet kernels, not the MoE path.

That leaves the more prosaic explanation: the b12x kernel was built for the B200’s tensor cores, and it simply doesn’t map well onto the GB10’s sm_121a architecture. Not a configuration bug, not a tuning matter — wrong kernel on the wrong chip.

That’s an important distinction. If it were a tuning problem, it could be fixed. As it is, Unsloth’s recommendation isn’t “untuned” but invalid on this hardware.

And then comes the second surprise

The lesson you’d draw from this: “use marlin”. But that’s a false generalization, and it’s exactly the most important sentence in this article.

We ran the same backend round on the production FP8 checkpoint too. In principle vLLM offers six FP8 MoE backends. The reality on GB10:

Four of six won’t even start. And the sixth, the one that would start by default, dies with an assert — on GB10 the FP8 model only comes up at all if you explicitly turn off deep_gemm.

But the point is the remaining two: on NVFP4 marlin wins, on FP8 marlin loses. Same machine, same vLLM version, same model family. The right backend isn’t a property of the hardware, but of the hardware and the quantization format together.

This is what no model card states, and what you can’t carry over from one model to another.

The vendor documentation as a minefield

While we’re at it, a few more things the cards don’t mention:

Dead environment variables. To the two environment variables in NVIDIA’s DGX Spark command, vLLM v0.24.0 replies that it doesn’t recognize them: VLLM_FP8_MOE_BACKEND and VLLM_USE_FLASHINFER_MOE_FP4. You set them, export them, nothing happens. The command still runs, it just doesn’t do what you think.

The draft model for speculative decoding wants a separate backend. All three NVFP4 checkpoints leave the MTP module out of the quantization — the draft head stays BF16. But neither marlin nor b12x can load that, because they’re meant for quantized weights. The solution is a separate switch in the speculative config:

--speculative-config={"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}

On NVIDIA’s card this triton is there. Why, nobody explains. If you leave it out, the server won’t even start, and it isn’t obvious from the error message that it’s about the draft model.

The unquantized draft head is a measurable gain, by the way. All three NVFP4 variants reach a 73–75% speculative acceptance rate, the production FP8 reaches 71%. We looked at why: in the FP8 checkpoint the 775 tensors of the MTP module are quantized (F8_E4M3), in the NVFP4 variants they aren’t. The full-precision draft head predicts better. This kind of information is on none of the cards — it reads out of the weight files’ headers, without a GPU, in two minutes.

What to actually run

If you have a DGX Spark and you serve this model family:

For NVFP4 checkpoints (all three, regardless of vendor):

--moe-backend=marlin
--speculative-config={"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}

Environment: CUTE_DSL_ARCH=sm_121a, FLASHINFER_DISABLE_VERSION_CHECK=1. VLLM_USE_FLASHINFER_MOE_FP4 and VLLM_FP8_MOE_BACKEND can go in the bin.

For block-FP8 checkpoints:

VLLM_USE_DEEP_GEMM=0

… and let vLLM choose — it’ll pick triton, and that’s the right one. Don’t force marlin here, you lose 9% with it.

And in both cases: after startup, check the log for what actually loaded. vLLM prints it. If it isn’t what you asked for, then your measurement — or your production — isn’t about what you think it is.

The full picture

The final result measured on the same backend (marlin) and the same speculative setting, under concurrent load:

So NVFP4 is +40% under concurrent load and +69% on single-thread extraction versus the production FP8. This comparison is clean in machine, vLLM version and speculative setting: both ran on the same machine, on the same engine, and each with its own best backend.

This number is real. What’s behind it, though, isn’t entirely what it looks like.

What we can’t explain

Two data points the usual model doesn’t describe, and which are better said than left unsaid.

The byte-twins aren’t identical. The weight mass of nv and uns-fast is identical to within 1%, the packed FP4 tensor count is 30841 and 30840. Same kernel, no speculation, yet the difference is 7.9% in single-thread decode. Bandwidth can’t explain it: there’s nothing to explain, the amount of data to read is identical. The most likely candidate is the serving path: nv uses static activation scales, both Unsloth variants compute them at runtime. We didn’t isolate this.

The reverse of the same: uns’s active expert path is 15.4% more bytes per layer, so the bandwidth model would predict a penalty of about −15%. The measured difference is −3.0%. Off by a factor of four.

In other words: the cross-format jump (FP8 → NVFP4, 35 → 22 GiB) is properly explained by the weight reduction. The 3–8% spread among the three NVFP4s is not. Something else decides there, and this measurement series couldn’t pin it down.

The mixed-load measurement is unstable. Three repeats of the mixed profile gave 239.9, 271.2 and 292.6 tok/s — a ±10% spread on the same configuration. The uniform profile is stable by comparison. This means that between any two adjacent values in the mixed column you cannot rank, and every claim based on it carries an error bar. That’s why the table above shows the uniform profile.

In summary

If you follow your own vendor’s documentation, your machine with your model can be up to three and a half times slower than it could be. Not because the hardware is bad, and not because you configured it wrong — but because the recommendation was made for a different chip, and nobody measured it on yours.

Four things worth taking home:

1. Backend choice is a 2–3.3× stake. Larger than anything else we found in this measurement series — larger than the difference between quantization variants, larger than the gain from speculative decoding.
2. The right backend isn’t portable across formats. marlin on NVFP4, triton on block-FP8, on the same machine.
3. A vendor recommendation applies to the vendor’s hardware, even when it names yours. Unsloth’s numbers were made on B200, at high concurrency. There they may be right. On yours they aren’t.
4. After startup, check the log. That’s the only place where it becomes clear whether the system is doing what you asked.

Over this measurement series I predicted the outcome three times in advance, and was wrong all three times: I expected there to be a crossover point between the two backends (there isn’t), that the tuning bucket would explain the cliff (it doesn’t), and that the more cautious quantization would speculate better (it’s worse). That’s why we measure.

---

The measurement series was done on a GB10-based development machine, on vLLM v0.24.0, with a pinned image digest. Every number comes from error-free runs; failed runs were discarded. During the perf measurements a single model always ran on the machine. The backend sweep consists of 60-second measurements; the medians for single-thread measurements come from 1 warmup + 3 measured runs. The repeat spread of the mixed-profile measurements is 10%, marked in the text where relevant.