Three flags from a vendor recipe, not one of them faster

A hardware vendor publishes an “agent-ready” serving recipe for a model in the same family as ours. It differs from our configuration in a handful of flags. The question is the obvious one: do those flags help our model, on our hardware, under our load — or is the recipe tuned for a setup that isn’t ours?

We measured it. Not one flag made it in. This article is about why that isn’t a failure but the ordinary outcome of a measurement — and about the one lever that wasn’t simply bad, but workload-dependent.

Why measure a published recipe at all

This is the most boring kind of experiment, and precisely for that reason it saves the most time. Adopting a published recipe is cheap: you paste the flags into the launch command and you’re done. Finding out six months later that one of them costs 3% throughput is not cheap — especially once five other changes have grown around that command and you can no longer tell which one does what.

A vendor recommendation isn’t suspect because it’s from a vendor. It just wasn’t written for your load.

What we measured

A cumulative ladder starting from the production baseline. Each variant needs its own engine start, so each arm is a cold boot plus a measurement pass — roughly 20 minutes apiece.

Model: Qwen3.6-35B-A3B-FP8 on a DGX Spark, GB10 chip (sm_121a). Load: six single-stream workloads from 29 to roughly 159,000 prompt tokens, plus two concurrent mixes at concurrency 8, 60 seconds per mix. No document corpus involved — this is a serving measurement, not an accuracy one.

The primary decision metric is aggregate throughput at 8 concurrent requests. Secondary: single-stream decode per workload, TTFT, and speculative acceptance from the engine metrics endpoint.

We fixed the decision rule before the run: a flag is adopted only if aggregate throughput improves materially and the boot log is clean of kernel fallback warnings. That order isn’t a formality. Decide the rule afterwards and you’ll keep adjusting it until some arm wins.

What we deliberately did not test

We left out three things, and it’s worth saying so out loud, because a quietly skipped arm biases a sweep just as much as a badly measured one:

• 4-bit weights with a Marlin MoE backend — the recipe’s headline pairing. It belongs to a different checkpoint format; the FP8 model stays on the more mature kernel path. This got its own measurement later.
• The fastsafetensors load format — known to OOM on this platform during a large model load without GDS.
• A different tool-call parser — a functional change, not a performance one. Our agent’s streaming path is coupled to the current parser, and swapping it blind would break tool-argument streaming. That needs a functional test, not a benchmark.

The result: nothing survived

The numbers are relative: we publish them in aggregated form in the repository too, without absolute tok/s. For the purpose of this measurement that is enough — the question wasn’t how fast our machine is, but whether a flag moves it.

For the first two arms the story ends there. --attention-backend flashinfer brings nothing because the default already selects an equivalent path for this model — an explicit override can only make it worse. And --async-scheduling tries to exploit an overlap that isn’t the bottleneck on this hardware.

The third arm: this is where it got interesting

v3_spec3 isn’t a flat loss. It’s a textbook speculative-decoding trade-off, and the fault line runs exactly along baseline acceptance rate.

JSON extraction +19%, long RAG 32k +17%, code +13%, short Q&A −9%, short Hungarian chat −3%. Where acceptance is already high (JSON 99.5%→98.3%), the longer draft is nearly free speed; where it is low (Q&A 64.6%→43.8%, Hungarian chat 58.2%→43.2%), the extra work is discarded. Aggregate throughput at 8 concurrent requests is −3.6%, with overall acceptance falling from 70.6% to 61.2%." class="blog-post-image">

Where the model already predicts its own next tokens well — strict JSON, long-context continuation — a longer draft is nearly free extra speed. Where it doesn’t, the extra draft token is discarded work, and under concurrency it competes with real requests for the same compute.

That split is the whole point. On JSON extraction, acceptance drops only from 99.5% to 98.3% — the third draft token is almost always good. On short chat it goes from 64.6% to 43.8%: more than half the time the third token is thrown away, and we paid to compute it anyway.

Our production model serves chat and extraction, so the mixed profile decides: draft length stays at 2.

The lesson we keep

Tuning speculative decoding is workload tuning. A single num_speculative_tokens value is right for exactly one acceptance regime, and a model serving both strict-JSON extraction (99.5% acceptance) and short chat (58–65%) doesn’t have one.

That points at an open direction, too: splitting extraction and chat onto separate engine instances would allow a per-instance draft length — 3 for extraction, 2 for chat. It doesn’t justify the operational overhead today, but the measurement showed where the line is beyond which it would.

What the measurement said about itself

The primary metric was aggregate throughput at 8 concurrent requests. Nobody had checked what production concurrency actually is.

It’s between 1 and 3. Zero percent of the time above 8.

That doesn’t change a single rejection — no arm was positive at any concurrency. But it does mean the decision-relevant metric for this system is time to first token, not saturated throughput. And the reasoning about v3_spec3 deserved a low-concurrency crossover sweep that never ran.

It’s easy to call that obvious in hindsight. In practice it’s the most expensive class of mistake: measuring at an operating point the system never reaches. This time it rejected the right things for the wrong reason. Next time it could accept the wrong ones — with exactly the same confidence.

In summary

Not one flag from a published vendor recipe improved our production baseline, so we changed nothing. The value of the measurement isn’t the change; it’s that this question won’t need re-litigating for the next six months — and that along the way it turned out speculative draft length isn’t a global setting.

1. A vendor recipe doesn’t transfer across a hardware and quantisation boundary. Its gains come from kernel paths that either don’t exist on this chip or aren’t reachable for a block-scaled FP8 checkpoint. Copying flags across that boundary isn’t a shortcut; it’s an untested change with a vendor’s name on it.
2. Tuning speculative decoding is workload tuning. Where acceptance is already high, a longer draft is worth +13–19%; where it is low, −9%. On a mixed profile the worse case decides.
3. Measure the load before you choose the metric. A benchmark at an operating point you never reach can reject the right things for the wrong reasons.
4. A negative result is a result. Four cold boots, about an hour and a half of machine time, zero changes — and one closed question that used to resurface in every configuration argument.

---

Measured on 1 July 2026 on a GB10-based development machine, on a single engine build, with one run per variant (n=1). The per-arm differences of −2 to −4% sit near the noise floor individually — the robust finding isn’t any single delta but that no arm came out positive. Runs that ended in errors were discarded. The measurement says nothing about whether the recipe works on the hardware and quantisation format it was written for.