GLM 5.3 Flash locally — NVFP4 on vLLM vs EXL3 on TensorFold, measured on 2× DGX Spark

What GLM 5.3 Flash is

GLM 5.3 Flash is a model from the Chinese company Zhipu AI (brand Z.ai). Before the weights came out it was tested anonymously on OpenRouter as “Ox Alpha”; the weights reached Hugging Face on August 26–27, 2026 per source. The key facts:

• 320 billion parameters in a mixture-of-experts design, 18 billion active per token: 288 experts, 8 of which work at a time.
• Hybrid attention: sparse (NoPE-MLA) interleaved with linear (KDA), plus mHC connections; 45 layers and one MTP layer for predicting the next tokens.
• Multimodal: text, image and video; a native window of up to 1 million tokens.
• MIT license.

According to its maker, it's the best open model on Terminal Bench 3.0 and scores 50% higher than GLM-5.2 on Z.ai Code Bench per source. The recommended sampling is temperature 1.0 and top_p 0.95.

GLM 5.3 Flash requirements: NVFP4 vs EXL3 vs GGUF

The original FP8 weights are about 306–328 GB and don't fit even on two DGX Spark units (2× 128 GB). 4-bit versions fit on two; only more heavily compressed ones fit on one.

Weight size vs DGX Spark memory

sizes as reported by the sources (GB or GiB) · │ 128 GB — one unit · │ 256 GB — two units · ■ one unit ■ two ■ doesn't fit

BF16

598 GiB

FP8 (original)

306–328

GGUF Q4_K_XL

~200 GB

NVFP4 · ours

184–190

EXL3 4 bpw · ours

164 GiB

GGUF Q2_K_XL

~109 GB

EXL3 2.05 bpw

~85 GB

GLM 5.3 Flash GGUF or NVFP4?

“GLM 5.3 Flash GGUF” is the most searched variant, so to be upfront: we haven't run the GGUF. Unsloth publishes GGUFs at several compression levels. Q2_K_XL (about 109 GB) fits on one DGX Spark and reportedly runs there at 17.7 tok/s, but it needed Unsloth's llama.cpp branch rather than stock Ollama (as of September 12 — check the current state before you start) per source. That's the one-unit route, at the cost of heavy compression.

The other one-unit route is EXL3 at 2.05 bpw (about 85 GB). According to a thread on the NVIDIA forum it gives about 64 tok/s on structured content and 25 tok/s on prose, but users reported loops and hangs with parallel requests per source.

We went with NVFP4 and EXL3 4 bpw on two units: less compression and two proven engines. The differences between them are below.

How we run GLM 5.3 Flash on DGX Spark

The cluster is the same as for DeepSeek V4 Flash: two DGX Spark / ASUS GX10 units linked with a QSFP56 200 Gb/s cable, the model split across both (tensor parallel = 2), with a chat app, an API gateway and agents running alongside. Same test method too: median of 6 runs after 3 warm-ups, temperature 0.6, prose, code and JSON separately. The hardware and link are described on our DGX Spark cluster page.

Setup A: vLLM + NVFP4 — all MIT

Checkpoint: RedHatAI/GLM-5.3-Flash-NVFP4 (184.29 GiB, 92.27 GiB per unit). We chose it because of accented characters — details in the quality section. NVIDIA's official checkpoint (from September 9) uses the same ModelOpt format as the LibertAIDAI one we rejected, and is 6 GiB bigger.

Engine: vLLM with community patches. Stock vLLM doesn't run GLM on GB10 — it fails with pe_dim must be 64, because one kernel was written for DeepSeek's model geometry, which GLM doesn't have (vLLM issue #57578 is still open). We used the community image tonyd2wild v11 plus two additions of our own:

• a top-k patch — without it, decoding fails beyond about 24K tokens of context, because the kernel wants 128 KB of shared memory and GB10 has about 99–101 KB;
• our own chat template with a thinking switch (why — see silent failures).

Speculation: the native MTP module, 3 tokens ahead — as fast as the non-commercial DFlash2 on this checkpoint, and all MIT. The full settings list is in the silent-failures section, because nearly every one of them breaks the model quietly.

Setup B: TensorFold + EXL3 — faster, but non-commercial

• TensorFold v0.6.0 engine (Apache-2.0) following MiaAI-Lab's recipe v1.10.
• Weights: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold (164 GiB, experts in 4-bit EXL3).
• Speculative module: incoai/GLM-5.3-Flash-DFlash2 — CC-BY-NC-ND-4.0.
• Dense layers in FP8 instead of the default 4-bit. In the recipe authors' test, the 4-bit version cut off 12 of 48 short code answers (it didn't finish the turn). With FP8, on the same 48 tasks in Polish: 0 of 48.

GLM 5.3 Flash benchmark on DGX Spark: vLLM vs TensorFold

vLLM + NVFP4

Measured by us September 23, 2026; median of 6, temperature 0.6, apps running alongside.

KV cache pool: 484,786 tokens (1.85× the 262K window); after the run about 10 GB stays free per unit — exactly our safety threshold. Startup takes about 8.5 minutes (about 10 the first time, while kernels compile). For comparison, himorishige's recipe — without apps alongside and with a higher 0.85 memory limit — reports 26.5 tok/s with MTP 3, 14.6 tok/s without speculation and about 76 tok/s in total with 8 conversations per source. Our 24.3 is the same ballpark under worse, because real, conditions.

Same model, different engine

Switching engines did more than any parameter tuning: prose from 24.3 to 36.0 tok/s, code from 32.3 to 67.0, and the first token arrives twice as fast.

vLLM + NVFP4 vs TensorFold + EXL3

tok/s · measured by us September 23 (vLLM) and October 10, 2026 (TensorFold) · 2× DGX Spark, apps alongside · ■ vLLM ■ TensorFold

prose

vLLM

24.3

TensorFold

36.0

code

vLLM

32.3

TensorFold

67.0

JSON

vLLM

32.0

TensorFold

71.2

4 conversations at once, total

vLLM

55.9

TensorFold

85.7

Long context: a needle found at 777K tokens

The “needle in a haystack” test: we hide one sentence in a long text and ask about it at the end. Both setups found the needle every time, but TensorFold with its 1M-token window reaches much further. Reading speed: 1,300–1,400 tok/s in vLLM (259K tokens → first token after 195 s) and 1,240–1,690 tok/s in TensorFold (777K → after 629 s, about 10.5 minutes).

Where it found the needle

position in the context, scale up to 1,048,576 tokens · ● needle found · shading = the setup's window · measured by us

vLLM + NVFP4 · 262K window · September 23, 2026

39K130K259K

TensorFold + EXL3 · 1M window · October 10, 2026

39K169K337K777K

Quality: accented characters, cut-off answers and images

Test accented characters before anything else

NVFP4 checkpoints in the ModelOpt format (LibertAIDAI, in our case) dropped a UTF-8 continuation byte — about 2 broken characters (U+FFFD, the “�”) per 10,000. In English text that's nearly invisible. But letters with diacritics take two bytes in UTF-8, so the error lands exactly on them. We work in Polish, which is full of them; the same applies to any text with non-ASCII characters.

Why “ą” is risky

illustration · result measured by us

a letter in UTF-8

ą

C4 85

two bytes: a lead byte and a continuation byte

ModelOpt checkpoint

ksi�żka

lost continuation byte → “�”; about 2 per 10,000 characters

RedHatAI NVFP4

książka

0 errors in 4,509 characters with 289 diacritics

Cut-off answers: 0 of 48

In the TensorFold setup, the recipe authors noticed that with dense layers in 4 bits the model cut off 12 of 48 short code answers — it simply didn't finish the turn. With dense layers in FP8, on the same 48 tasks in Polish, not a single one was cut off in our run.

Images

GLM 5.3 Flash understands images. In the vLLM setup (vision profile, September 24) measured by us:

The practical takeaway: about 40 large photos per request is safe. TensorFold caps it at 50 images but keeps ≥ 14.5 GB of memory free (50/50 in 16 s).

GLM 5.3 Flash was also the judge in our decision model benchmark — as a classifier it reached 86.5% on Polish emails, but needs about 4 s per document.

Silent failures: settings that break the model without an error

With GLM on vLLM, nearly every key setting has one thing in common: a wrong value gives no error message. The model starts and answers — just badly.

The GLM-5.3 chat template has no thinking switch. It only has reasoning_effort (low, high, max — max by default) and always ends the prompt with a <think> tag. Apps that turn thinking off — for example generating a 40-token conversation title — got empty answers, because thinking ate the whole budget. We added a branch to the template that switches thinking off.

On top of that come the pitfalls shared by all GB10 machines, described on the DeepSeek V4 Flash page: a unit that runs out of memory freezes instead of killing the process, and the wrong RDMA device silently costs about 40% of performance.

License: what a company may do

• The GLM 5.3 Flash model: MIT. Commercial use, modification and offering it to customers are allowed.
• The RedHatAI NVFP4 checkpoint: MIT. Setup A (vLLM + NVFP4 + MTP) is therefore MIT throughout.
• The DFlash2 module (incoai/GLM-5.3-Flash-DFlash2): CC-BY-NC-ND-4.0 — non-commercial only. Setup B with this module is for personal and non-commercial use.
• According to the recipe authors, TensorFold can use the built-in MTP head instead of DFlash2, which makes it license-clean — but it then handles one request at a time per source. We haven't measured that.

For companies: setup A. It's slower, but nothing in it carries a commercial restriction.

GLM 5.3 Flash compared with DeepSeek V4 Flash and Qwen3.8-Flash-Next

On the same cluster, GLM on vLLM is the slowest of the three (24.3 tok/s on prose), while with TensorFold it writes code and JSON fastest (67.0 and 71.2 tok/s) and is the only one we run with a 1M-token window. DeepSeek V4 Flash reads long documents faster (about 2,300 tok/s) and is more thoroughly tested on documents and agent work. Qwen3.8-Flash-Next starts answering fastest and leaves the most free memory, but its license requires a separate agreement before offering the model to third parties. For offering a model to customers: GLM on vLLM or DeepSeek — both MIT. The full side-by-side comparison, with charts and seven lessons from running them: Best LLM for 2× DGX Spark.

GLM 5.3 Flash FAQ

Does GLM 5.3 Flash fit on one DGX Spark?

Only heavily compressed. According to Unsloth the GGUF UD-Q2_K_XL is about 109 GB, and according to an NVIDIA forum user EXL3 at 2.05 bpw is about 85 GB — both fit on one unit, but we haven't tested either. The versions we run (NVFP4 at 184 GiB and EXL3 4 bpw at 164 GiB) need two units.

Which GLM 5.3 Flash NVFP4 checkpoint should I use?

We use RedHatAI/GLM-5.3-Flash-NVFP4. Checkpoints in the ModelOpt format dropped a UTF-8 continuation byte — about 2 broken characters per 10,000 — and letters with diacritics take two bytes. RedHatAI gave us 0 errors in 4,509 characters of Polish text with 289 diacritics.

GLM 5.3 Flash GGUF or NVFP4?

Unsloth's GGUF (UD-Q4_K_XL about 200 GB, UD-Q2_K_XL about 109 GB) runs in llama.cpp with Unsloth's branch and lets you go down to one unit. NVFP4 (184–190 GiB) runs in vLLM on two units, and that's the route we measured: 24.3 tok/s on prose and 70.6 tok/s in total with six conversations. We haven't tested GGUF.

How many tokens per second does GLM 5.3 Flash get on DGX Spark?

On our 2× DGX Spark cluster: with vLLM and NVFP4, 24.3 tok/s on prose and about 32 on code and JSON; with TensorFold and EXL3, 36.0 on prose, 67.0 on code and 71.2 on JSON. With several conversations at once: 70.6 tok/s in total (6 conversations, vLLM) and 85.7 tok/s (4 conversations, TensorFold).

Can GLM 5.3 Flash be used commercially?

The model itself is MIT, so yes. Watch the add-ons: the DFlash2 speculative decoding module used in the faster TensorFold setup is CC-BY-NC-ND-4.0, which means non-commercial use only. For companies, the right setup is vLLM with NVFP4 and the native MTP speculation — all MIT.

Why does vLLM need patches to run GLM 5.3 Flash on GB10?

Stock vLLM fails with “pe_dim must be 64”, because one kernel was written for DeepSeek's model geometry, which GLM doesn't have (vLLM issue #57578). A second patch fixes the top-k kernel: without it, decoding fails beyond about 24K tokens of context, because the kernel wants 128 KB of shared memory and GB10 has about 99–101 KB.

Sources

• Z.ai, GLM-5.3-Flash model card (320B / 18B active, MIT, architecture) and config.json (288 experts, 8 active, 45 layers + MTP, 1,048,576 window); Z.ai documentation — 1M window, video input, benchmark results, recommended sampling.
• TechNode, Zhipu identifies Ox Alpha as GLM-5.3-Flash and releases model weights (August 27, 2026).
• NVFP4 checkpoints: RedHatAI, NVIDIA, LibertAIDAI.
• Recipes for 2× DGX Spark: tonyd2wild, himorishige (MTP 3, no-speculation and 8-conversation results), barrydeen; Classmethod, GLM-5.3-Flash NVFP4 on two DGX Sparks with vLLM.
• vLLM: GLM-5.3-Flash recipe and issue #57578 (pe_dim must be 64).
• TensorFold: engine (Apache-2.0), MiaAI-Lab's 2× DGX Spark recipe, Mia-AiLab EXL3 4 bpw weights, incoai DFlash2 (CC-BY-NC-ND-4.0).
• Unsloth, GLM-5.3-Flash-GGUF — UD-Q4_K_XL and UD-Q2_K_XL sizes, Unsloth's llama.cpp branch.
• NVIDIA Developer Forums, 60 tok/s GLM 5.3 Flash on a single DGX Spark (September 3, 2026) — EXL3 2.05 bpw, about 85 GB, reported hangs with parallel requests.
• Measurements and tests: our own, ailone's 2× DGX Spark / ASUS GX10 cluster, September 23–24 (vLLM) and October 10, 2026 (TensorFold).

As of October 11, 2026. Community images, patches and engine versions change every week — the numbers apply to the setups described above. Measurements are medians from one series, with apps running alongside.