When the eval lies

A simple question: is it worth moving to a bigger model in our document processing engine? The model in production today is Qwen3.6-35B-A3B — a 35-billion-parameter MoE (Mixture of Experts) architecture, of which only ~3 billion parameters activate per token. The challenger is Llama-3.3-70B-Instruct: a classic dense model, twice the size on paper, moving its entire weight set for every single token.

What matters here is what we expected — because the measurement got interesting somewhere else entirely.

We expected nothing on speed. A dense 70B will be slower on this hardware than a 3B-active MoE; that needs no experiment, just a division (which we do further down). We knew it going in and we would have accepted it: if accuracy were meaningfully better, you can design an architecture around a slower but more reliable model — batched, run overnight, in a two-stage pipeline.

On accuracy, though, we expected it to at least match Qwen. And there was an unspoken assumption baked into that expectation: that the flagship model of a large American lab would be at least as at home in a European language — Hungarian invoices, Hungarian company names, Hungarian tax-number formats — as a Chinese one. Llama’s training corpus is weighted far more heavily towards the West; if anywhere, this is where it should have pulled ahead.

It didn’t. It fell thirty F1 points short.

America versus China on Hungarian ground — and China won again. That is not what this article is about. There are two paragraphs on that sentence at the end, because drawing that kind of conclusion from a single measurement is exactly the mistake this whole piece argues against. The article is about the two things we found along the way: an error detector that confidently reported the opposite of reality, and a piece of arithmetic we should have done up front.

Customer and partner names are anonymised — “our company”, “partner A”, “partner B” — and the documents are numbered. The model names and the numbers are real.

What we measured, and how

The task is KIE (Key Information Extraction): pulling structured JSON out of Hungarian business documents — invoice number, partner name, tax number, bank account number, line items, and a handful of “whose document is this” fields.

• Corpus: 100 real Hungarian documents, split into 119 evaluation units (100 headers + 19 line-item blocks)
• Models: Qwen3.6-35B-A3B FP8 (current production) vs Llama-3.3-70B-Instruct FP8 (nvidia/Llama-3.3-70B-Instruct-FP8, ModelOpt quantisation)
• Metric: field-level F1 behind a versioned normaliser (whitespace, casing, tax-number format and so on unified)
• Significance: McNemar test, paired at document level — the same document went into both models with the same prompt, so the paired test is the right instrument
• Hardware: GB10-based development machine, 121 GB unified memory, vLLM 0.25.1
• Quantisation: FP8 on both sides, so the same bit class

One important detail about the splits. Qwen is multimodal: alongside the text it also receives a 200 DPI page image. Llama does not. That makes the whole-corpus number not a clean model comparison — Qwen got extra input. For the decision we therefore treated only the 52 units where both models received exactly the same thing as authoritative: text only.

This kind of self-restraint isn’t modesty. If your own measurement doesn’t separate out where the models received different input, you aren’t comparing models, you’re comparing pipelines — and later you’ll be puzzled that the “winning” model doesn’t deliver in a different environment.

The result

¹ In these rows Qwen received extra input (the page image) — see above.

Thirty F1 points. That is not a nuance, and it is not measurement noise either: the paired test gives p=0.0043.

Two things are worth highlighting, because both contradict the usual expectation:

Llama held the format flawlessly. JSON validity: 119 out of 119, for both models. Zero invalid outputs. The “big model, but it can’t hold the schema” fear simply didn’t materialise — the failure is one of content, not syntax. Llama returned exactly the JSON we asked for; it just filled it with wrong values.

The errors are not omissions but mistakes. The breakdown of Llama’s field-level outputs: 746 hits, 108 wrong values, 43 omissions, 26 spurious fills. The dominant error type is mismatch: the model confidently writes something that isn’t right. Operationally that is considerably worse than an omission — a validation system catches an empty field, but not a plausible-looking wrong tax number.

And where does it go wrong? Broken down by field, the picture is anything but even:

The errors cluster precisely around partner identification and document ownership. In records management these are the two most expensive field groups: partner matching is built on them, and so is the decision whether an invoice belongs on the incoming or the outgoing side. An average F1 of 0.90 can look reassuring — until you notice that the missing 10% isn’t spread evenly but concentrated on exactly the fields the downstream logic depends on.

The trap: the error detector that inverted

Here is the heart of the article.

We have a small helper script written for an earlier series of measurements. Its logic fits in one sentence: if our own company ends up in the partner field, that’s an error — flag it. In an earlier quantisation comparison it earned its keep: models with worse quantisation tended to copy the company printed at the top of the header into the partner slot, and this script caught exactly that.

We ran it on this measurement too. The result:

Qwen3.6-35B-A3B    doc=100   flagged=7
Llama-3.3-70B      doc=100   flagged=1     McNemar p = 0.070

Seven to one. In Llama’s favour. A trend at the edge of significance.

This would have been the single dimension in the entire measurement where Llama came out ahead. And it fitted our expectation perfectly: “there we go, the bigger model is more reliable at something after all.” It would have been the obvious move to write “Llama, however, is more accurate at deciding ownership” into the report and move on.

Instead we checked those seven documents against ground truth. This is what we found:

In six of the seven flagged documents the correct answer really was our own company. These are supplier-side documents: invoices where we are the buyer, so from the issuer’s point of view we are the partner. Qwen got all of them right — and that is precisely why the detector flagged them.

Turn the numbers around: genuine ownership errors are 1 for Qwen and 6 for Llama. The detector’s raw output (“1 vs 7”) suggested the literal reverse.

Why did it invert?

The script wasn’t buggy. The assumption baked into it drifted out from under the corpus.

The detector’s implicit premise was: “in this corpus, our own company is never a legitimate partner.” On that earlier corpus this was true — it contained outgoing invoices only. The new corpus mixes incoming and outgoing documents, and there the statement simply doesn’t hold. Same code, same logic, opposite meaning.

And this is the bad kind of failure: it doesn’t blow up. No exception, no empty output, nothing out of range. It runs, and it returns a sensible-looking number whose sign is flipped. You notice a crashed script in five minutes. You notice a quietly inverted metric only if you go back and check ground truth by hand — which is exactly the work the metric was supposed to save you.

What we took away

Proxy metrics have an expiry date, and it isn’t the metric you need to version, it’s its assumption. A heuristic error detector doesn’t measure the error; it measures a corpus-dependent fingerprint of the error. When the corpus changes, the validity of that fingerprint changes with it — silently.

For us this turned into three concrete rules:

1. Every heuristic detector gets its assumption written out in one sentence — there, at the top of the file, not in our heads. If you can’t state it in one sentence, you don’t know what it measures.
2. This script now only runs together with ground-truth verification. The raw flag count never goes into a report on its own.
3. If exactly one dimension in a measurement disagrees with all the others, that’s not an interesting exception, it’s a suspect. We check the metric first and only then write it up as a conclusion.

The third is the most important and the hardest to stick to — because that one outlier always looks more exciting than the twenty numbers pointing the same way. Especially when it happens to agree with what you were hoping for.

The second lesson: the arithmetic we should have done up front

The speed gap is brutal:

A 26× slowdown. We expected it — but that isn’t the point. The point is that this could have been calculated in advance, with a single division, and then on day one of the measurement we would have known that accuracy didn’t just have to match Qwen but had to beat it substantially to even be in the conversation.

The GB10’s memory bandwidth is 273 GB/s. On a dense model, every generated token requires reading the entire weight set out of memory. 70 GB of weights, 273 GB/s:

273 GB/s ÷ 70 GB = 3.9 tokens/s

That is the theoretical ceiling. Not the measured value, but the number physics permits. We measured 2.5 — so reality sits at 64% of the ceiling, which is entirely normal for a dense model.

From which follows the most important sentence in the whole exercise: this is not a tuning problem. No batch size, quantisation or kernel optimisation takes this above 3.9 on the same machine. Tuning can turn 2.5 into 3.2 — it cannot turn 214 seconds per document into 8.2.

Qwen wins the same formula from the other side: with MoE, only the weights of the selected experts have to be read per token, so only a fraction of the ~35 GB moves. That is where the order-of-magnitude difference comes from — the model isn’t “smarter”, it reads fewer bytes.

Run the numbers on your own setup

You need two figures, and both take five minutes to find:

ceiling (tokens/s) ≈ memory bandwidth (GB/s) ÷ active weight size (GB)

Where active weight size is, for a dense model, the full checkpoint size, and for an MoE roughly the share implied by the active parameter ratio (plus the always-active layers). If the resulting number is below what your use case demands, the model is not viable on that hardware — no matter how good its benchmark scores are.

This calculation reframes the usual question. It isn’t “does it fit in memory” — it fitted, it loaded, it ran. It is “how many times per second can I read through it”.

If you want to run your own document processing on-premise, this formula is worth more than any leaderboard: it tells you whether the model you have your eye on is even in the game in your box, before you download anything.

What this measurement does not prove

If a comparison has no limitations section, it isn’t a measurement, it’s marketing. Here are ours, all four:

1. We measured FP8, not BF16. The BF16 build of meta-llama/Llama-3.3-70B-Instruct (~141 GB) doesn’t fit in a 121 GB machine, so we measured the nvidia/Llama-3.3-70B-Instruct-FP8 checkpoint. Production Qwen is FP8 as well, so the comparison is fair within a bit class — but a BF16 Llama might be somewhat better.
2. The prompt is tuned for Qwen. We iterated the prompt set for months against the current production model. That is a systematic advantage for Qwen. A Llama-tuned prompt would narrow the gap — it would hardly erase 30 points, but we didn’t measure that, so we don’t claim it.
3. The cleanest split is small. On the header-only sample n=33 and p=0.118 — not significant. The broader 52-unit head-to-head sample is (p=0.0043), and the direction is the same in every breakdown, but on the strictest reading the cleanest slice shows a trend only.
4. We measured a single task. KIE on Hungarian business documents. That is our primary production metric; it isn’t anyone else’s. We claim nothing from this about Llama-3.3-70B’s general capabilities — chat, coding, tool calling and long context all went unmeasured.

A bonus: the operations lesson

Our first attempt to start Llama locked up the machine — the SSH connection dropped and it needed a manual reboot. The obvious explanation would have been that it ran out of memory: the kernel threw NVRM: Out of memory four times.

It wasn’t that. The model loaded fine. What got stuck afterwards was the FlashInfer autotuner: the first tuning round ran for 48 minutes, a single profile took 865 seconds, and then it started a second round with an estimate of over two hours. The symptom really was a memory error, but the cause was a background optimiser nobody had asked for.

The fix is a single switch: enable_flashinfer_autotune: false. As a bonus, the runai_streamer load format took model startup from 490 seconds to 16.

The lesson is the same as with the guard, wearing different clothes: there is no reliable mapping from symptom to cause. If we had stopped at the “out of memory” message, we would still believe the model doesn’t fit on the machine — and we would have ruled out a perfectly loadable checkpoint on the strength of a wrong inference.

So, America versus China?

Back to the unspoken assumption in the introduction. We expected Llama to at least match Qwen on Hungarian business text — a more Western training corpus, twice the parameters, the flagship of a large American lab. It didn’t, and it didn’t fall short narrowly either.

We allow ourselves only what the data can carry. This is one task, one language, one corpus, in FP8, with a Qwen-tuned prompt. Concluding “Chinese models are better” from that would be exactly the mistake the detector chapter argues against: elevating a single number because it fits a narrative.

What we will say without hesitation: anyone choosing an open-weight model for Hungarian document processing today has no basis for narrowing the field by country of origin. The intuition that “a big American model will surely be better at European languages” was not merely unsupported on this corpus — the opposite is measurable, and significant. Both are open weight, both are downloadable, both run on your own hardware, and in the reality of Hungarian data protection that is the only thing that truly counts.

So the lesson isn’t geopolitical but methodological: measure it on your own corpus. It is the same sentence as every other lesson in this article, only at a larger scale — don’t believe the leaderboard, and don’t believe your intuition either.

In summary

The model-swap decision itself is boring: we’re not switching. Llama-3.3-70B is thirty F1 points weaker head to head, its errors concentrate on exactly the most expensive field groups, and it is twenty-six times slower on this hardware. The three things that outlive this particular model pairing:

1. Your heuristic metrics are calibrated to a corpus, even if you never wrote that down. When the corpus changes, the metric doesn’t blow up — it quietly starts meaning something else. Write its assumption down in one sentence, next to the code.
2. If one dimension points the opposite way from all the others, suspect the metric first, not the model. For us this rule saved a report from a wrong sentence — in precisely the dimension where we would have been happy to write that sentence anyway.
3. The bandwidth ceiling can be computed before you download anything. Bandwidth divided by active weight size. If the number isn’t enough, the model isn’t in the game on that machine — and no amount of tuning changes that.

The last one is an attitude rather than a lesson: a measurement is worth something only if you are prepared to measure the instrument as well — and prepared for the result not to say what you expected. This article is essentially about the time we nearly failed to do both.

---

The measurement ran on 100 real Hungarian business documents split into 119 evaluation units, on a GB10-based development machine (121 GB unified memory, vLLM 0.25.1). Both models ran in FP8 with identical prompts; significance was computed with a McNemar test paired at document level. Only the 52 units where both models received identical input were treated as authoritative for the decision — in the multimodal and whole-corpus rows Qwen also received a 200 DPI page image, so those are not clean model comparisons. Customer and partner names are anonymised; the model names and the numbers are real.