Efficient ML

Overview

The cost of running a model is set by three numbers: the bytes it occupies, the bytes it moves per inference and the operations it performs. Efficient ML is the set of techniques that shrink one or more of those without shrinking the model’s usefulness by as much. The field matters because inference, not training, is where most of the money goes once a model is in use, and because a model that fits on the device in a user’s hand needs no network, no server bill and no data leaving the device.

This hub is for engineers who have a model that is too big, too slow or too expensive, and need to know which lever to pull first. The levers are quantisation, pruning and sparsity, distillation, architectural choices such as mixture of experts, and the decision to use a smaller model in the first place. The formats and file types are tabulated in /reference/quantisation-formats/, and the hardware limits that make these techniques pay are in /topics/hardware-and-acceleration/.

Key ideas

Quantisation: fewer bits per number

A trained weight is a 32-bit or 16-bit float. Quantisation maps it to an integer with fewer bits plus a scale factor, so that a block of weights becomes a block of int8 or int4 values and one float. The common scheme is symmetric absmax: find the largest magnitude in a block, divide by 127 to get the scale, round each weight divided by the scale to an integer. The error is at most half a scale step per weight, which is why the block size matters: a per-tensor scale lets one outlier coarsen every other weight, and per-channel or per-group scales (groups of 32 to 128 in GPTQ, AWQ and the GGUF k-quants) keep the step small where most weights are.

Post-training quantisation (PTQ) does this to a finished model, with a small calibration set to decide scales and, in GPTQ and AWQ, to compensate for rounding error layer by layer. Quantisation-aware training (QAT) simulates the rounding during fine-tuning so the model adapts to it, and is what makes int4 and below usable without the accuracy drop PTQ would leave. For large language models at the time of writing, weight-only int8 is close to free, 4-bit group quantisation costs a measurable but usually acceptable amount, and anything lower needs QAT or a very large model with headroom. LLM.int8() (Dettmers et al., 2022) showed why outlier features in large transformers make naive int8 activations fail, and QLoRA (Dettmers et al., 2023) showed that a model can be fine-tuned while its base weights sit in 4-bit, which is what /topics/fine-tuning-and-alignment/ relies on for single-GPU work.

Pruning, sparsity and conditional compute

Pruning removes weights. Unstructured pruning zeroes individual weights by magnitude or by a saliency score and can remove half of them from a large language model with a small loss (SparseGPT, Wanda), but a GPU does not run faster on scattered zeros without hardware support. Structured pruning removes whole heads, channels or layers and gives real speed-ups at a higher accuracy cost. NVIDIA’s 2:4 structured sparsity, where two of every four weights are zero, is the hardware middle ground. Mixture of experts is conditional compute rather than removal: the model holds many feed-forward experts and routes each token to a few, so a model with 109 B total parameters might compute with 17 B active per token. The memory cost is the total, the compute cost is the active count, which is why MoE models are cheap to serve at scale and awkward on a laptop.

Distillation and the small-model option

Distillation (Hinton, Vinyals and Dean, 2015) trains a small student to match a large teacher’s output distribution rather than hard labels, which carries more information per example. For language models the practical version is to generate a dataset with the teacher and fine-tune the student on it. SmolLM2-135M (135 M parameters, 8 k context, Apache-2.0, November 2024) runs on a laptop CPU in milliseconds per token. Gemma 3 1B (1 B parameters, 32 k context, Gemma Terms of Use, March 2025) and Llama 3.2 1B and 3B (128 k context, Llama 3.2 Community Licence, September 2024) are built for phones. The honest rule is that a small model fine-tuned on your task usually beats a large general model prompted for it, at a fraction of the cost, provided your task is narrow and you can build the training set.

Where to start

Read /learn/beginner/07-your-first-neural-network/ and /learn/beginner/08-embeddings-and-similarity/ so that weights and activations are concrete, then /learn/beginner/10-from-notebook-to-service/ for the deployment side. Then take a model you have, quantise it with torch.ao.quantization or export it to GGUF with llama.cpp’s conversion script, and measure accuracy on your own evaluation set before and after. The /projects/lora-fine-tune-apple-silicon/ project combines 4-bit base weights with a small adapter, which is the standard single-machine recipe.

Tools and platforms

Common mistakes

• Reporting perplexity alone after quantisation. Perplexity moves little while a downstream task collapses. Evaluate on the task.
• Quantising with a calibration set from the wrong distribution. Calibrate on data like the data you will serve.
• Expecting speed from unstructured sparsity on a GPU without 2:4 support or a sparse kernel.
• Choosing a 70 B model at 2-bit over a 7 B model at 8-bit without testing both. The smaller, less damaged model often wins.
• Forgetting the KV cache. Quantising weights to 4-bit does nothing for a cache that grows with context length.
• Distilling from a teacher without checking its licence allows training on its outputs. Several API terms and open-weight licences restrict this.

Going deeper

• Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022.
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023.
• Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, 2015.
• Frantar and Alistarh, SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, 2023.
• PyTorch quantisation documentation and the llama.cpp quantisation notes.