40% faster, and we still didn’t switch

We had a working production model: a Qwen3.6-35B in block-FP8, extracting structured data from Hungarian invoices. Then the same model showed up in NVFP4 — three different quantizations from two vendors. The measurement was unambiguous: +40% concurrent throughput, +69% single-stream extraction. Smaller weight files, faster decode, same model.

We didn’t switch.

This article is about how we got there — and about the fact that the decision was ultimately made by a metric other than the one we set out to measure.

The first metric lied

The natural first question: does extraction accuracy degrade? We have an established measurement for that — a 25-document annotated corpus with field-level F1, and the FP8 vs NVFP4 quality data is public.

At first glance the result looked clear:

FP8 wins, quantization costs accuracy. We could have stopped there.

Except that looking through some of the errors, it turned out they weren’t errors. The model wrote 1.0 where the ground truth said 1; a line break landed in a different place inside a longer source text; a missing numeric field came back as 0.0 instead of null. Formatting differences, not wrong content.

Here comes the uncomfortable part. Our first move was to introduce a “cleaned F1” rule after the fact — once we had already seen who won. That is a methodological error even when it comes from good intentions: if you shape the rule while knowing the result, you will keep shaping it until you like the output. So we started over: we wrote the normalizer, pinned it with a version number, and only then re-scored all four checkpoints with the same code.

The gap between FP8 and the Unsloth variant: 0.0001. A tie.

And notice what happened to the other two rows in the meantime: normalization swapped their order. What we saw as a 0.5-point gap in the raw numbers was mostly formatting noise, not quality.

So this measurement did not say what it appeared to say at first. It did not say “FP8 is more accurate”. It said that on this corpus it cannot decide.

Our own benchmark said nothing either

We have a second measurement too: a 16-scenario conversation eval covering tool calls, multi-step reasoning and refusals — “what are my receivables”, “compare the suppliers”, “what about a company that doesn’t exist”.

It gave us nothing. 7 of the 16 failed on all four models for the same reason: the test tenant’s data is incomplete, so the correct answer everywhere is “no data”, which the evaluator counts as an error. Only 3 scenarios differentiated at all — and on two of those the two Unsloth variants contradicted each other.

That’s noise. You cannot pick a model from 3 fluctuating results on a 16-item sample, and anyone who does is calling the variance of their own test set “quality”.

Two measurements in, and neither could decide. Then came the one that could.

The gate that mattered

Our invoice extraction has one decision that outranks all others: which company is us, and which is the counterparty. Both appear on the invoice, with a name, a tax number and an address. If the model swaps them, an entry lands in the books facing the wrong way, and the system cannot discover that error on its own later.

We didn’t measure this on the curated corpus, but on 100 real business documents. Not pretty, not hand-picked, not annotated — the stuff that actually goes through the pipeline in production. The methodology and the aggregate results are documented separately: the counterparty-identification gate, with McNemar testing.

All three NVFP4 variants roughly double the one error that matters most to us.

Same input, temperature=0, same prompt. The only difference is the quantization of the weights.

The nature of the error — where it got interesting

We looked at how it breaks. We expected name matching to be the shaky part: similar company names, abbreviations, accented characters.

That is not what happened. There was not a single partial case. In all 18 faulty documents the model copied the entire identity of our own company into the partner field — name and tax number — and in all 18 the internal flag that identifies our own company dropped to zero as well.

So it isn’t that the model is unsure about names. It’s that it knows two companies appear on the invoice, and it puts them on the wrong sides. This is a role-assignment failure, not a string-matching one.

Which has a very practical consequence: the error is total, therefore fully detectable. We wrote a three-line deterministic check for it — if the digits of the partner’s tax number match ours, or the partner name contains our own company name, it’s suspicious. That rule caught 18 of the 18 errors. Not 16, not 17. All of them.

There is also a hint about the mechanism. The rule flagged two additional documents that our tax-number check could not confirm — both were foreign partners, one with a SWIFT code where the tax number should be, the other with nothing at all. On those the model also fell back to our own company. This raises the possibility that the error surfaces when the partner’s tax number is missing or in an unfamiliar format — but two documents are not proof, only a hypothesis for the next round.

What the statistics allow, and what they don’t

7 versus 18 is striking, but striking is not an argument. A paired test is required, since we ran the same documents through every model.

McNemar’s test against production FP8:

The first two are significant, the third isn’t. And since we ran three comparisons against the same baseline, once corrected for multiple testing none of them individually clears the 5% threshold.

Then we compared the three NVFP4 variants against each other:

The three NVFP4 checkpoints are indistinguishable from one another on this sample. The 12–16–18 ordering is noise.

This matters, because it kills an obvious narrative. We had a hypothesis that more aggressive quantization degrades reasoning more — it’s logical, and the literature suggests it too. The measurement ranked things the other way round: the least aggressive variant made the most errors, the most aggressive the fewest. Except that this reversed ordering isn’t significant either.

So what we can honestly claim is: all three NVFP4 variants roughly double this error relative to FP8 (12–18 versus 7), and the differences among the three cannot be resolved on this sample. No more, no less.

The decision

FP8 stays.

Not because NVFP4 is bad. The +40% is real, the measurement is clean — same machine, same engine, each with its own best backend.

But because of this:

A hundred samples is too few to understand the error. We know it exists; we know it’s total; we know it’s detectable; we have a hypothesis about its trigger. What we don’t know: how big it actually is, on which document types it surfaces, and whether any NVFP4 variant is meaningfully better than the others. As long as the three candidates are statistically indistinguishable, we cannot even choose between them — it isn’t a question of which one to take, it’s that the data doesn’t permit a choice.

And because accuracy matters more than speed — at this load. The pipeline is not throughput-bound. That +40% solves no existing problem: we aren’t queueing, we aren’t missing deadlines, we aren’t paying more. In exchange we would take on an error class whose size we don’t know. That’s a bad trade, no matter how pretty the number.

If we ever become throughput-bound, this decision reopens — the measurement is there for it, and so is the deterministic check.

What we learned

The metric that decides is rarely the one on the model card. Both vendors published benchmarks: MMLU-Pro, GPQA, AIME. One variant scores 85.85, the other 85.58 — citable, comparable, and entirely irrelevant to our decision. Neither of them measured our number, and neither could have: it exists only on our documents, with our schema, under our business rule.

Your own benchmark can be blind too. We had two established measurements, and neither could decide. The decision came from a measurement we built specifically because we knew which error hurts the business most. Without that gate we would be running NVFP4 today, and would probably have spent weeks not noticing why the number of manual corrections had gone up.

Pin the rule before you look at the result. The after-the-fact “cleaned F1” was well-intentioned, and still invalid. With the pre-registered normalizer, the same data produced a tie where we had previously seen a win.

A total error is good news. If a model is partially wrong, that’s hard to catch. If it’s totally wrong, it’s easy. We wrote three lines of code and they catch 100% of the errors. Which raises the question that will be the topic of our next round: if a deterministic rule can verify a decision exhaustively, does a language model need to make it at all? Possibly not. Telling apart the two companies on an invoice isn’t a language task, it’s identifier matching — and we have a database for that.

What’s left

A bigger corpus. A hundred documents was enough to see the difference between FP8 and the NVFP4 class, but too few to characterize the error. The foreign-partner hypothesis needs its own measurement, on a purposely collected sample.

And building the deterministic pre-filter — not so that we can switch afterwards, but because FP8 also errs in 7 cases out of a hundred. If there is a rule that catches all of them, the question is no longer which model we use, but why we let the model decide this at all.

In the end, this quantization test wasn’t about quantization. It was about finding an error class in our own production that had been there all along — we just hadn’t looked at it.

---

The measurements were made on a GB10-based development machine, on vLLM v0.24.0, with an identical image digest, each checkpoint running its own best backend. The accuracy measurements ran at temperature=0. The 100-document gate used real production business data; only aggregate numbers appear in this article. The statistical tests are paired McNemar tests, with p-values from exact binomial computation.