November 1, 2026 · Models · ~12 min read

How to Choose an AI Model: Qwen vs Nemotron

Test run November 1, 2026 · ASUS Ascent GX10 (NVIDIA DGX Spark platform)

How to choose an AI model is the question every project asks, and it almost always gets the wrong answer: “the best one”. There is no best model. There is a model matched to the task, the hardware and the server configuration. We put that to the test with two models on a single desktop machine: Nemotron 3.5 Super VL (120B parameters, 12B active) and Qwen3.8-Flash-Next FP4-Local (125B, 6B active). Eleven tests, one expectation: 500 tok/s.

500 tok/s vs 37

NVIDIA's launch material carries a number that sticks: over 600 tokens per second on agentic tasks. It comes from a cloud endpoint (Baseten, Artificial Analysis) and states neither the hardware nor the number of parallel requests. For the DGX Spark the vendor gives only a multiplier: “up to 2.3× faster generation” with the LiLiCorr drafter. Qwen praises QSA (“up to 10.2× faster prefill and 6.6× faster decode kernels”), yet there is no official tok/s figure for FP4-Local on a DGX Spark anywhere.

We measured 24.62–67.86 tok/s in a single stream and 54.3–80.1 tok/s in aggregate with eight parallel requests. We never came close to 500. That is physics: decoding reads from memory as many bytes as the active parameters weigh per token. Nemotron has 12B active in NVFP4, roughly 7 GB per token; at 273 GB/s the ceiling is about 40 tok/s. Qwen with 6B active is roughly 3.4 GB, so a ceiling of about 80 tok/s. The measurements sit under that estimate: 24.62 and 37.02 tok/s on prose. And 63.19 tok/s on JSON is speculative decoding — code and JSON are predictable, prose is not.

What vendors promise — and what they leave out

Each claim next to the conditions it was published under, and next to our measurement.

Sources: the model cards and the NVIDIA launch blog (links in the annex).

What the documentation does not contain: neither card mentions Polish, the Qwen card makes no OCR claim, and the quality numbers come without reasoning mode or concurrency. Without those conditions a number is not comparable.

How we tested

Hardware: an ASUS Ascent GX10, i.e. the NVIDIA DGX Spark platform — GB10, 128 GB of unified memory, 273 GB/s and 1 PFLOP in FP4 (specification). One machine, one model at a time, no cloud. Both models ran in our own vLLM instances with the same max-num-seqs 4 and a 131,072-token limit; unavoidable differences: batch 8192 versus 4096 and speculative decoding LiLiCorr 7 versus MTP 4.

Speed: 3 warm-ups and 6 measurements per category, 900-token answers, median and spread. Quality automatically: every function ran in a separate process against assertions, JSON was validated field by field, invoice decisions were compared with ground truth. The writing tests were in Polish, our working language, and scored blind: 24 anonymised samples, two raters, rank agreement 0.783 (Spearman). The unit of analysis is an independent generation, not a single test case.

Speed: one user and four at once

Single stream: one user, one answer, median of six runs.

Median of 6 runs after 3 warm-ups, 900-token answers, reasoning off, temperature 0.6. Ranges in brackets are min–max.

Three takeaways. Prose is Nemotron's weakest spot — 24.62 tok/s, less than half of what it does on JSON. Code and JSON are a tie (1.05× and 1.07×), so “the faster model” is a property of the task, not of the model. Time to first token separates them more: 0.049 s against 0.296 s.

Many users at once — here server configuration enters and moves the result more than the model does.

Two rounds per level, answers of about 450 tokens, both models at max-num-seqs 4, temperature 0.6.

Qwen is 1.4× faster in aggregate and already saturated at eight requests — 79.7 and 80.1 tok/s are effectively the same number. Nemotron reaches 56.5 tok/s at four requests, does not grow at eight (54.3), and median time to first token climbs from 1.28 s to 13.26 s. The key lesson: in an earlier run Nemotron used the vendor recipe setting, max-num-seqs 1, and with four requests gave 26 tok/s at a median of 24.4 s. After changing that flag to 4 the same model delivered 56.5 tok/s. Doubling throughput took one number, not a different model.

Quality: code, JSON, tools, invoices

Code. Four tasks, five generations each, every function run in a separate process against assertions.

4 tasks × 5 generations = 20 generations per model; 10–11 test cases per task. Reasoning off, temperature 0.2, 1600-token limit.

Here is the trap most comparisons fall into: the gap of 94% against 77.5% of test cases looks dramatic, but those 200 cases come from 20 generations, so they are not independent. Counting generations properly: 18 of 20 for Nemotron and 15 of 20 for Qwen, p = 0.41. That is not evidence of superiority. Three of five Qwen samples in the amount-parsing task were cut off at the 1600-token limit — it writes more elaborate code and runs out of budget. This test does not show which model is better at code.

JSON and tools. Extracting invoice fields into a strict schema, 10 samples, json.loads and twelve field-by-field checks: 10/10 on both. Five scenarios in the native OpenAI tools API, two samples each, including “there is no tool, so it must not be called”: 10/10 on both; a prompt-protocol chain of two tools: 3/3 on both. The earlier server rejection on Qwen came from a missing tool-parser flag, not from the model.

Invoices. Six invoice types, two samples each, short loop: read the fields, check the tax ID with a tool, verify policy, decide.

6 types × 2 samples = 12 runs per model, temperature 0.2, reasoning off. p (Fisher) = 0.64.

The gap of 8/12 against 10/12 over twelve runs gives p = 0.64, i.e. within the range of chance. But one result here is not chance: both models approved an invoice whose line items sum to 4,320 rather than 4,520 — 0/2 on both. The mechanics were flawless (12/12 clean JSON); the arithmetic failed. An agent can drive the document flow, but the line-item sum and VAT must be checked by code.

The agentic document promise: what we tested and what we didn't

NVIDIA positions Nemotron for agentic document work: a single request may involve reading a document, checking information across sources, following company policy and calling several tools. That is a broad promise and our test does not settle it — it neither confirms nor refutes it. A day later we tested it separately — see the update at the end of this post.

What we tested: 12 short passes (6 invoice types × 2 samples) with simulated tools — tax-ID check, duplicate check, ledger write, flag for review — with reasoning off and temperature 0.2. Separately, long context: five hidden codes, three depths with three samples each, the deepest at 99,404 tokens for Nemotron and 88,197 for Qwen.

What we did not test: long, multi-hour agent runs with many turns; real document workflows, meaning real invoices, a real ledger and real policy documents instead of a simulator; the full one-million-token window — our deepest samples were around 100k tokens, and the configuration capped context at 131,072. We also did not test whether the model holds to company policy across dozens of documents in one session.

The conclusion: in this slice, the mechanics work on both models and arithmetic verification fails on both. That is enough to plan a validator in code, but not enough to declare the agentic document promise fulfilled or refuted.

Images and long context

Two images, five samples each: a bar chart with eleven months and an invoice as a scan. Five values sit above the threshold, so a fully correct answer needs five out of five and no false positives.

Chart: Nemotron 3/5 fully correct at recall 1.00, Qwen 0/5 at recall 0.80 (p = 0.167). Invoice scan: 5/5 on both.

2 images × 5 samples = 10 runs per model, temperature 0.2, reasoning off.

Nemotron found all five bars in every sample, but in two of five it added months below the threshold — hence 3/5 fully correct at recall 1.00. Qwen missed “Feb” in every one of its five samples, so recall fell to 0.80 and it was never fully correct. With five samples p = 0.167, so this is not proof of superiority, but the pattern repeats in every sample.

Long context. Five hidden codes, three depths with three samples each. Hits: 44/45 for Nemotron and 45/45 for Qwen. Cold prefill: Nemotron 2,844, 2,990 and 2,629 tok/s, Qwen 2,112, 2,388 and 2,392 tok/s, so Nemotron is faster by +35%, +25% and +10%. Caveat: one cold sample per depth and different batch settings (8192 versus 4096 tokens) — a comparison of configurations, not models.

Non-English prompts: instructions, not prose

Four prompts, three samples each, twelve samples per model: deterministic checks (sentence limit, required elements, forbidden phrases) and a blind style review.

Check results: Nemotron 7/12, Qwen 11/12. Breakdown: email 3/3 on both, API explainer 1/3 against 3/3, café copy 3/3 on both, correction 0/3 against 2/3.

4 prompts × 3 samples = 12 samples per model, temperature 0.6, top_p 0.95. p (Fisher) = 0.155.

The gap of 7/12 against 11/12 gives p = 0.155, so it is not statistically significant — but the pattern is consistent: Nemotron twice exceeded the sentence limit, and in the correction task it never fixed “wypaloną”. The blind style review found no significant difference (p = 1.00 and p = 0.51), and rank agreement was 0.783. The difference is in instruction-following, not in prose quality. Neither model card mentions Polish at all — everything we know about how these two models behave in that language comes from our own measurements.

Reasoning: 2–4 more correct answers, 23–39× the tokens

Five verifiable tasks (arithmetic, logic, units, a multi-step problem, a discount with VAT), two samples in each mode: 20 runs per model, 10 per mode.

Reasoning off: Nemotron 8/10, Qwen 6/10. Reasoning on: 10/10 on both. Median tokens 5 → 196 for Nemotron (39×) and 5 → 115 for Qwen (23×), time 0.48 s → 3.73 s and 0.35 s → 2.20 s, i.e. 6–8×.

5 tasks × 2 samples × 2 modes = 20 runs per model, 10 per mode, temperature 0.2.

Reasoning fixes multi-step errors on both models: Nemotron from 8/10 to 10/10, Qwen from 6/10 to 10/10. But the bill is brutal: median tokens go from 5 to 196 on Nemotron (39×) and from 5 to 115 on Qwen (23×), and response time by 6–8×. With ten runs per mode this is a small sample, so treat it as a direction. In practice: turn reasoning on where correctness matters and off in chat. On Qwen you have to do it explicitly — the model thinks by default with reasoning_effort=xhigh.

Which model for what

Code is a tie (18/20 against 15/20, p = 0.41), as are JSON and tools (10/10 and 3/3 on both). This table covers this one machine and this server configuration — on other hardware the numbers will change, and that is exactly the point of this article.

Nemotron 3.5 Super VL is a language model that writes answers. NVIDIA’s retrieval models from the same family, Nemotron 3.5 Embed and Rerank, do a different job: finding the right page first. We measured them in our multimodal RAG benchmark on construction drawings.

How to choose an AI model: seven steps

The order in which we make the decision ourselves. Each step maps to a number from this test.

1. Start from the task, not the model. Chat needs a fast reaction, so 0.049 s against 0.296 s matters. Document analysis can wait a few seconds.
2. Turn the task into a measurable test. Without a defined correct answer you have nothing to compare models with.
3. Work out the hardware ceiling. Active parameters times bytes per token divided by memory bandwidth: 12B active in NVFP4 at 273 GB/s is about 40 tok/s.
4. Equalise the configuration before comparing. One max-num-seqs flag doubled Nemotron's throughput from 26 to 56.5 tok/s.
5. Measure on your own data and language. Model cards do not mention Polish, and that is where the differences showed up: 11/12 against 7/12.
6. Separate mechanics from arithmetic. In short runs both models called the tools correctly and both let through an invoice with an inconsistent sum.
7. Price the reasoning mode. 23–39× more tokens and 6–8× longer responses, especially when a model thinks by default on every “good morning”.

And the main point: there is no best model. There is a model matched to the task, the hardware and the configuration — and matching it takes a few hours of measurement, not a week of reading benchmarks. To see how the same method played out across three other models, read DeepSeek V4 Flash, GLM 5.3 Flash or Qwen3.8-Flash-Next; for the narrow class of models built for decisions, see the decision model benchmark; and for when local inference beats the cloud, start with on-premise LLM vs cloud AI.

Limits of this test

• Small samples. Between 5 and 20 runs per test. At p = 0.41, 0.64, 0.167 and 0.155 the difference is not significant, and we describe it that way.
• Configurations are not identical. Same request limit, but a different batch (8192 versus 4096 tokens) and a different speculative decoding mechanism (LiLiCorr 7 versus MTP 4).
• One machine, one quantisation. Both models in NVFP4/FP4; BF16 would not fit in 128 GB of unified memory, so that comparison is missing.
• Invoices are synthetic. Six types with two samples each tests mechanics and one policy, not deployment effectiveness. We did not test multi-hour agent runs, real document workflows or the full one-million-token window. A long workflow on fictional documents is covered in the update at the end.

FAQ

Which AI model is the best?

None of them is best at everything. Qwen wins where response time and instruction-following matter: 37.02 tok/s on prose against 24.62 and 0.049 s to first token against 0.296 s. Nemotron did better on the chart (3/5 against 0/5, a small sample) and had faster prefill in this configuration. On JSON and tool calling both scored 10/10.

How do I choose an AI model for a specific task?

Start from the task, not the model: decide whether latency or accuracy matters, and build a checkable test set. Work out the hardware ceiling, equalise the server configuration, measure on your own data and language, and separate the agent's mechanics from the arithmetic that code should verify.

Is 500 tokens per second achievable on a DGX Spark?

Not in our measurements. The fastest single stream reached 67.86 tok/s (JSON on Qwen), prose 37.02 tok/s on Qwen and 24.62 tok/s on Nemotron, the aggregate with eight parallel requests 80.1 tok/s. The vendor's over-600 TPS figure refers to a cloud endpoint, with no hardware or concurrency stated.

Update, 2 November: a long document workflow

This post left NVIDIA's promise open: can an agent read a document, check it across several sources, apply company policy and call several tools? A day later we tested it directly.

The test: 30 cases on 82 fictional documents. They include text and scanned invoices, purchase orders, delivery notes, a contract, correspondence and a three-page company policy. 20 cases contain deliberate defects, 2 try to talk the agent into breaking the policy, and 21 require combining at least three documents. The agent has eight tools, and writing to the ledger really changes its state. We score the final state, not the text of the answer. Reasoning is on for both models. Each model did two runs: one long session up to the 65,536-token window limit, and every case on its own, in a clean context.

30 cases × 2 runs = 60 decisions per model, temperature 0.2, reasoning on. p (McNemar) = 1.0.

• The workflow mechanics work for both models. They read several documents, apply the policy, resist manipulation and retry after a transient tool error (7/7). Quality did not drop towards the end of the long session.
• Nemotron has no edge. 56 vs 58 is within chance.
• Speed depends on how you run it. In the long session Nemotron handled 1.6× more cases per hour. On separate cases it reasoned 2.5× longer and was 1.6× slower. Overall it is a tie, 101 vs 106 cases per hour. "Up to 5× higher throughput" does not show up in this setup.
• Code still has to guard what the model cannot see. Both models booked a duplicate when the ledger lookup returned an empty result, because to a model an empty ledger looks the same as no duplicate. Nemotron once approved a scan with a wrong line total. It is the same lesson as in the short test: validation belongs in code, not in the model.

Limits: 30 cases, one configuration and fictional data. The tools run locally, not in a real accounting system. We reduced the window to 65,536 tokens so that the long session would reach the limit. We discarded the first, overnight run after an audit: the test set had errors in its expected answers, and the separate-case run did not clear the context.

For model choice: in a document workflow both models are equally accurate. The way you run it decides: Nemotron for long sessions, Qwen for single documents.

Annex and sources

Sources

• Nemotron 3.5 Super VL: NVFP4 model card, NVIDIA launch blog (600+ TPS, 2.3× with LiLiCorr, Hermes Bench 63.44%, LangChain Evals 87.05%), LiLiCorr card, vLLM recipe.
• Qwen3.8-Flash-Next: FP4-Local card, base model card (thinking on by default with reasoning_effort=xhigh, SWE-bench Multilingual 81.0, Toolathlon 73.5), vLLM recipe.
• Hardware: DGX Spark specification, ASUS Ascent GX10.
• Measurements: our own, ailone, November 1, 2026; every number comes with its method and sample count under the table.

These results cover the configuration described and data from November 1, 2026. Before deploying, repeat the test on your own data — that is the whole point of this article.