When is a bigger AI model worth it — and when isn’t it?

Over the past two years, larger and larger AI models have shipped almost every month. The market narrative says bigger is always better: more parameters, more knowledge, better results. That assumption, however — as our recent comparative test at DocAI showed — is not at all universally true.

In this article I show how we decided, systematically, whether it’s worth switching to a four-times-larger AI model in our existing system. The answer is surprising: it isn’t worth it for every task, but there is one specific area where it pays off. The path we took to reach that conclusion can serve as a useful case study for any executive making AI investment decisions.

The context: what we’re building, and why the question matters

DocAI is a Hungarian-built document processing platform that automates invoice, contract and HR document handling — together with the related business analytics — for accounting firms and Hungarian SMEs. The core of the system is a large language model that extracts structured data from Hungarian-language documents, answers questions, and supports business decision-making.

The model currently running in production is a 35-billion-parameter system, fine-tuned on Hungarian documents, with strong, industry-recognised performance. The question was natural: if a roughly four-times-larger, 122-billion-parameter model exists in the same model family, is it worth switching? Would our customers see noticeably better quality in exchange for the higher operating cost?

We couldn’t leave the question unanswered. A wrong call in either direction is costly: if we switch to a larger model without justification, we pay the extra bill for months; if a switch was warranted but we skip it, our customers get worse results and eventually move to a competitor.

How do you objectively measure an AI model?

The AI industry pumps out an incredible volume of marketing-flavoured “benchmark” results. For enterprise decision-making, however, these are of very limited value, because the typical tests measure general-purpose English-language tasks — not what Hungarian business life actually demands.

That’s why at DocAI we started building our own evaluation framework in January this year, which over the last few weeks has finally reached a level where it’s reliably usable for model-selection decisions. We measure across two main dimensions:

The first dimension is structured data extraction: can the model correctly read out the relevant fields of a Hungarian invoice, contract or HR document? This is the foundation of daily DocAI use, and our current system already achieves essentially perfect accuracy here.

The second dimension is complex business analysis: solving multi-step problems that mimic human reasoning. For this we designed five new tasks taken from real Hungarian business life — tasks that typically land on a senior financial analyst’s or head of accounting’s desk and require hours of manual work:

Cashflow gap forecast: is a liquidity problem likely in the next 60 days, taking into account receivables due dates, customers’ historic payment patterns, supplier liabilities, and recurring fixed costs.
Supplier anomaly detection: which of our suppliers shows an unusual invoicing pattern over the past 12 months against a two-year baseline — sudden price jumps, frequency shifts, unusually large one-off items.
Customer concentration risk: how much risk our current customer-portfolio concentration represents, which customers are critical dependencies, and how we should diversify.
VAT position optimisation: breakdown of the quarterly VAT position with separate handling of reverse-charge items, plus timing optimisation recommendations.
Supplier external risk monitoring: stability check on the top 5 suppliers based on public news, risk classification, and action recommendations.

Each task has an objective evaluation rubric: how many data lookups the model performed, in what order, whether it reached the correct conclusion, and whether it formulated a concrete action recommendation. The results were also separately scored by a human expert — in this case me — with an “eyeball” quality assessment, so that the quantitative metrics are complemented by a qualitative view.

The first surprise: on the core use case, the larger model didn’t win

We put the large model through a full battery of tests. On structured data extraction — what DocAI customers use daily — the result was unambiguous: the larger model brought no improvement. The smaller model already operates at essentially perfect accuracy on Hungarian invoices and typical business documents, and there is nowhere to improve. The data behind the 35B vs 122B comparison is public.

On simpler chat-style questions — “who are our top 5 suppliers”, “what was our revenue last year”, “show me receivables over 10 million HUF” — we measured equivalent performance. Out of ten test tasks, both models succeeded on the same nine and failed on the same one.

Meanwhile the larger model was 2.6× slower. For an average user question where we currently deliver sub-second response time, users would have had to wait four to five seconds — not something any user would appreciate. On top of that, the larger model’s hardware and maintenance are significantly more expensive.

If we had only looked at these measurements, the decision would have been clear: stay with the current model. Fortunately, we kept investigating.

The second surprise: there’s one specific area where it is measurably better

The real test for the large model was the five complex business tasks above. There we saw entirely different results: the larger model gave a better answer in five out of six cases, or an equivalent one — it never performed worse on any single task.

The specific differences are often subtle, but practically significant:

On the cashflow forecast task, both models struggled, but the larger model at least produced a concrete action recommendation, while the smaller one just listed data without drawing any conclusion. From a business decision-making perspective this difference is substantive: an executive doesn’t want a data dump, they want a specific, reasoned proposal.

In another test — calculating monthly spending trends — the smaller model computed an incorrect growth percentage, while the larger model was accurate. This is especially concerning: it isn’t that the smaller model “leaves something out”, it’s that it confidently delivers a wrong answer. An executive who doesn’t double-check the numbers could make a flawed decision off the back of it.

The most interesting finding, though, was the efficiency pattern. The larger model typically reached the same correct answer with about five times fewer data lookups than the smaller one. Put differently: the smaller model attacked the complex tasks “by brute force” — firing off a long sequence of queries, often repeating the same ones — while the larger model worked with a more targeted strategy. A human analogy: it’s like a junior analyst opening fifteen different Excel files to answer one question, while a senior colleague solves the same thing from three reports.

The business lesson: not size, but the right model for the right task

What does this mean concretely in DocAI’s practice? We use both models, but for completely different purposes.

The current 35-billion-parameter model remains the workhorse of daily operations: structured document processing, interactive conversations with customers, simple queries. Here speed, well-established accuracy and cost-efficiency are what matter.

The larger 122-billion-parameter model runs as a dedicated background system that performs nightly and weekly complex analyses. These cover questions that would take a senior finance or accounting professional several hours of manual work — and the output arrives in the morning, in the form of a clear, reasoned report on the customer’s desk.

When is it worth using the larger model for such a background process? Based on our own experience, when the task meets at least three of the following criteria:

Information has to be assembled from several different data sources, and reaching a conclusion requires multi-step reasoning.
The answer needs numeric reasoning — trends, ratios, concentration measures — not just a listing of data.
The task is a comparative analysis across multiple periods (at least two years) or a large number of entities (at least five partners).
Long documents are involved — multi-year financial data, or multiple documents at once.
It runs as a background process where 2–10 minutes of response time is acceptable.
The result ultimately goes to human review, not directly to an end user.

This is a simple, easy-to-apply decision rule — and any organisation that’s currently weighing whether to switch to a larger or smarter AI system can use it.

The leadership takeaway

The dominant narrative in the AI market today is that the race is about the biggest model, the most parameters. The major players keep pushing new behemoths, and the news cycle suggests that anyone who doesn’t upgrade is falling behind.

Reality is more nuanced. The larger model is more expensive, slower, and often does not deliver a meaningfully better result on typical business tasks. But where there is genuinely a difference — complex, multi-step analysis, processing of longer documents, numeric reasoning — the larger model really does create value.

A truly good AI strategy therefore does not mean always picking the largest, most expensive model. It means measuring objectively, understanding which tool is right for your own tasks, and accepting slightly more complex infrastructure that orchestrates multiple models for different purposes.

This principle, incidentally, isn’t unique to AI. A well-run organisation doesn’t employ a single super-expert for every task either: it has a senior analyst for strategic questions, an accountant for the daily work, and a junior staff member for routine assignments. The right way to use AI systems follows similar logic.

At DocAI we apply this principle: two different models, two different purposes, each deployed where it can deliver maximum value. Our 122-billion-parameter model is now in production, and our customers receive the complex overnight analyses every morning.

And what did we learn from the process? Perhaps the most important thing is that going against the hype sometimes pays off. When everyone is sprinting under the banner of “bigger, better, faster”, deliberate measurement and disciplined thinking become a competitive advantage.

DocAI is a Hungarian-built document processing and business analytics AI platform. The methodology and models described in this article run in our own production system. For any question — whether about a concrete rollout or strategic consultation — I’m happy to help.

— Dániel Kiss, DocAI developer

Technical appendix — for engineers only

The section below isn’t aimed at business readers, but at engineers and ML practitioners who are working on language model rollouts themselves. The article above stands on its own; this appendix gives concrete handholds to those facing a similar project.

Hardware and model

The tests ran on an NVIDIA DGX Spark (GB10 chip, sm_121a architecture, 128 GB unified memory). The larger model: Sehyo/Qwen3.5-122B-A10B-NVFP4 — with 4-bit NVFP4 quantization, the 122-billion-parameter MoE model needs roughly 76 GB of disk and around 72 GB of runtime memory (alongside the main weights, a separate extra_weights.safetensors file holds the multi-token prediction draft layer weights in BF16 — more on that below). The smaller (production) model: Qwen3.6-35B-A3B-FP8. Runtime: vLLM (0.19.1rc1.dev328), based on the eugr/spark-vllm-docker community container image, which ships the NVFP4 MoE kernels built for the GB10 sm_121a architecture. The standard vLLM cu130-nightly image does not include these — this is the first pitfall every Spark user runs into.

Critical obstacles

--load-format=fastsafetensors OOM risk in unified memory. On a discrete GPU, the gpu-memory-utilization flag governs only VRAM, and the mmap peak during loading doesn’t count. On GB10 unified memory the two are the same — the 72 GB model’s mmap peak plus the vLLM allocation together trip the OOM killer even at 0.85. Fix: use the default safetensors loader and 0.75 gpu-memory-utilization.

FlashInfer NVFP4 MoE kernels are not supported on sm_121a. All four FlashInfer NVFP4 MoE backends (FLASHINFER_TRTLLM, FLASHINFER_CUTEDSL, FLASHINFER_CUTEDSL_BATCHED, FLASHINFER_CUTLASS) abort with NotImplementedError. Use VLLM_CUTLASS (default) or the MARLIN backend instead.

The Marlin MoE backend’s “no native FP4 support” warning is misleading. The warning suggests Marlin is running on some bad fallback — in reality, by design it always uses a weight-only path, and on sm_121a it is in fact faster than VLLM_CUTLASS: in our measurements +2.3% decode and −7% TTFT across the six typical workloads. Important caveat: Marlin only works with a fully quantized MoE model — if you also use multi-token prediction (MTP) speculative decoding (see below), the BF16 draft layer makes Marlin unusable, and vLLM auto-selects the VLLM_CUTLASS + TRITON combination instead.

Some quantization pipelines strip out the speculative decoding (MTP) weights. The RedHatAI/Qwen3.5-122B-A10B-NVFP4 model card documents multi-token prediction speculative decoding support — in practice, however, vLLM started at 0% acceptance rate (286 draft tokens → 0 accepted), because the quantization pipeline strips out the draft layer weights and vLLM silently random-initialises in their place. The Sehyo/Qwen3.5-122B-A10B-NVFP4 checkpoint, by contrast, keeps these in BF16 in a separate extra_weights.safetensors file (785 tensors, 4.4 GB) — here the acceptance rate is 74% on average (100% on JSON-structured output, 60% on Hungarian prose), which speeds up decode by 34–78% depending on workload. The price: the draft layer alone takes ~6 GiB out of the KV pool (lower concurrency), and the cold-start roughly doubles (~22 min). Both costs are acceptable for a cron job. Lesson: don’t trust the model card — measure.

KV-cache FP8 trade-off. The --kv-cache-dtype=fp8_e4m3 flag boosts KV pool capacity by +94% (296k tokens fit in a 12 GiB pool instead of 132k), which is critical for long-context use cases. In exchange, the per-token dynamic scaling overhead makes the TTFT of a 10K-token prompt +223% worse. Negligible for a cron-style background task, painful for interactive chat.

The KV pool does not scale linearly with max-model-len. vLLM’s max_cudagraph_capture_size autotune picks a smaller value at longer contexts (32 → 16 → 8) and adds the freed memory to the KV pool. The result: the 262K context config has more total KV tokens (1.06 million) than the 65K config (849k). The opposite of the naive expectation.

Final serving configuration

--max-model-len=262144
--max-num-seqs=4
--gpu-memory-utilization=0.75
--max-num-batched-tokens=16384
--attention-backend=FLASHINFER
--reasoning-parser=qwen3
--tool-call-parser=qwen3_coder
--enable-prefix-caching
--enable-auto-tool-choice
--kv-cache-dtype=fp8_e4m3
--speculative-config={"method":"mtp","num_speculative_tokens":2}

Deliberately omitted: --load-format=fastsafetensors (OOM risk in unified memory), --kernel-config={"moe_backend":"marlin"} (the Sehyo MTP draft layer is BF16, and Marlin only applies to a fully quantized MoE — vLLM auto-selects VLLM_CUTLASS for the main MoE and TRITON for the draft layer), VLLM_USE_FLASHINFER_MOE_FP4=1 (NotImplementedError on sm_121a).

Key measurement numbers

Concurrency at 262K context: 1.83× with MTP (4.00× without MTP — the draft layer takes ~6 GiB out of the KV pool)
Decode speed: 25–30 tokens/s on short structured output, 22 tokens/s on long Hungarian prose (with MTP)
MTP acceptance rate: 74% on average — 100% on the JSON-KIE workload, 60% on Hungarian prose
Cold-start time: ~22 min from a clean state, ~5 min cache-warm restart (the draft layer loads as a separate shard)
Latency ratio vs. 35B production: ~2.2× (the 2.6× figure in the article body is the snapshot before MTP was enabled — the MTP iteration brought it down)
Tool-calling efficiency: the 122B uses 4.7 tool calls per scenario on average, the 35B uses 8 — a 5× efficiency gap on the successful scenarios

The full bring-up log (14 phases, every measurement, every cause of failure) is in internal documentation. If you’re preparing a similar project and get stuck anywhere — I’m happy to share advice. After this much learning, it’s almost a moral duty.