Measuring quantized LLMs: what is lost when an open model is compressed

Third instalment of Cluster X-ray. It is the best example of what the overview post said: the cluster today is above all a measurement laboratory. Not a single parameter is trained here.

The question #

Open models are distributed quantized: the same weights compressed to 8, 6, 5, 4, 3 or 2 bits so they fit on a small card or a laptop. The community publishes hundreds of variants, but rarely with a comparable measure of what is lost. The line asks how well each quantization level of each family really performs on tasks that matter for using the model as a tool, and at what speed and energy cost it does so on each card.

How it is approached #

Almost 300 different GGUF files from families such as Qwen (2.5 to 3.8), Gemma 3 and 4, GLM, DeepSeek, Mixtral, Llama 3, Mistral, gpt-oss, Nemotron, Granite, SmolLM or Phi-3, plus experimental ternary models, and the full ladder: BF16, Q8_0, Q6_K, Q5_K_M, Q4_K_M (the most common), Q3, Q2, Unsloth’s UD series with its IQ1 to IQ4, MXFP4 and NVFP4. For several families, curves of seven or eight levels of the same model are generated.

Three exams, chosen because they measure the model as a tool and not as a conversationalist:

• BFCL, function calling: whether the model produces the right call with the right arguments.
• BigCodeBench, 148 programming problems with real libraries.
• RULER, retrieval in contexts from 4K to 128K tokens.

On top of that, tokens per second per card, the real memory each file occupies, perplexity on wikitext-2 in the quantization sweeps, and energy: each node’s idle consumption is measured and subtracted from the consumption during generation.

The method has two rules that make it comparable. Each data point carries a manifest with the file, the quantization, the KV cache, the context, the binary with its checksum and the card. And before measuring anything, the published score of an anchor model is reproduced: if the deviation exceeds five points, the fault lies with the harness, not the model.

What is learned #

Several findings were written down in the scripts themselves, with dates:

• Quantizing the KV cache is not free and is not part of the protocol. With Gemma 4 it breaks: degenerate loops in 101 of 148 problems, against a clean result with the cache in f16.
• Changing cards changes the output. Only 21 of 148 responses were identical between an L4 and an H200 with the same configuration. Comparisons are always made on the same card.
• The model is measured, not the scaffolding. BFCL is attacked through the raw completion API: “via the native route we would be measuring the model plus the scaffolding”.
• An anchor that cannot be downloaded is not an anchor: when Llama 3.1 8B ended up behind an acceptance wall, the long-context anchor moved to Phi-3 mini.

And a security rule that is part of the method: the cluster generates text, it never executes the code the model writes. BigCodeBench grading, which does run that code, happens in an isolated container on a separate machine. BFCL’s multi-turn phase, which executes calls during generation, runs inside a sandbox with no network and with the correct answers hidden. A comment that corrects itself records that misreading the harness code “would have put the isolation in the wrong place”.

On the cluster #

Short jobs of one GPU, four cores and reduced priority, half of them on the modest 24 GB L4. The first 593 failed because of an SSH tunnel between the laptop and the node; since then client and server run on the same node. Five September jobs hung some 18 hours past their limit until Slurm closed them all at once; in the series’ figures they are counted only up to the requested limit.

In the fourth instalment we change subject completely: folding proteins.

Figures from Slurm accounting (reserved capacity, not measured usage) and the archived sbatch files. Anonymised post: no identifiable users, paths, emails or project names. Quotations are from the script comments.