Fine-tuning and alignment

Overview

A pretrained model predicts plausible continuations of text. It does not answer questions, follow a format or refuse anything, because nothing in next-token prediction asks it to. Fine-tuning and alignment are the stages that turn the base model into the assistant you call through an API, and the same stages, at smaller scale, are how you adapt an open model to your own task, format or domain. This hub covers the methods a working engineer will meet: supervised fine-tuning, parameter-efficient adapters, preference optimisation, and the evaluation that tells you whether any of it worked.

The decision that matters most comes before any of this. Fine-tuning changes how a model behaves, and is poor at adding facts. If the problem is missing knowledge, retrieval is cheaper and easier to update. If the problem is format, tone, a narrow task or latency from a long prompt, fine-tuning a small open model often beats prompting a large one.

Key ideas

Supervised fine-tuning teaches the format

Supervised fine-tuning (SFT) continues training on examples of the behaviour you want: a prompt and the ideal response, rendered through the model’s chat template with role markers. The loss is the same next-token cross-entropy as pretraining, usually masked so that only the assistant’s tokens count. A few thousand carefully written examples beat a hundred thousand scraped ones, which was the finding of the LIMA paper in 2023, and the quality bar is whether you would be happy to see each response in production. Get the chat template exactly right. A model trained with one template and served with another degrades silently.

LoRA and QLoRA make it affordable

Full fine-tuning of a 7 B model in 16-bit with Adam needs about 16 bytes per parameter for weights, gradients and optimiser state, so more than 110 GB before activations. LoRA freezes the base weights and learns a low-rank update for selected matrices: instead of changing W it adds B times A, where A and B have a small inner dimension r, typically 8 to 64. The trainable parameters drop by a factor of a hundred or more, the adapter is a few tens of megabytes, and you can keep several adapters for one base model. QLoRA goes further by storing the frozen base in 4-bit NF4 and dequantising on the fly, which brings a 7 B fine-tune under 8 GB and a 70 B one onto a single 48 GB card.

# snippet
import torch

class LoRALinear(torch.nn.Module):
    def __init__(self, base: torch.nn.Linear, r: int = 16, alpha: int = 32):
        super().__init__()
        self.base = base                     # frozen pretrained weight
        self.base.weight.requires_grad_(False)
        self.a = torch.nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.b = torch.nn.Parameter(torch.zeros(base.out_features, r))  # zero: starts as identity
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + (x @ self.a.T @ self.b.T) * self.scale

After training the adapter can be merged into the base weights for serving at no extra cost, but merge into the 16-bit weights, not the 4-bit ones, or the rounding eats part of what you learned.

Preference optimisation: RLHF and DPO

SFT can only imitate. To make a model prefer the better of two answers, you need a signal about preference. In reinforcement learning from human feedback, people compare pairs of responses, a reward model is trained to predict those comparisons, and the policy is updated with PPO to raise reward while a KL penalty keeps it close to the SFT model. That is the InstructGPT recipe from 2022 and still the backbone of most assistant training.

Direct preference optimisation, from 2023, showed that the same objective can be written as a classification loss on the preference pairs themselves, with the reference model standing in for the reward model. It needs no sampling loop and no separate reward network, which is why it is the method most teams try first. For tasks with a checkable answer, mathematics and code above all, recent work replaces human preference with a verifier and a group-relative policy update (GRPO), which is how DeepSeek-R1 (671 B total, 37 B active, 128 k context, MIT licence, January 2025) was trained to produce long reasoning traces.

Evaluating whether it worked

Compare against the base model and the SFT model on the same held-out set, never against nothing. For open-ended outputs the usual tool is an LLM judge with a written rubric, which correlates well with human raters on average but has known biases towards longer answers and towards its own style. Check instruction following with IFEval, check that general capability has not regressed with a few standard benchmarks, and look for over-refusal and sycophancy by hand. Preference-trained models can learn to game the reward, so read a sample of outputs every time, not only the score. The evaluation hub covers harnesses and the safety hub covers red-teaming the result.

Where to start

Read Using an LLM well first, so that you fine-tune only when prompting has failed, and Your first neural network in PyTorch for the training loop underneath. Then run the LoRA fine-tune on Apple Silicon project, which adapts SmolLM2-135M-Instruct (135 M parameters, 8 k context, Apache-2.0, November 2024) in minutes on a laptop. The quantisation formats reference explains NF4 and the GGUF types you will meet when serving the result.

Tools and platforms

Common mistakes

• Fine-tuning to teach facts that change. The model will state last quarter’s figures with confidence. Use retrieval.
• Training with a chat template that differs from the one used at serving time, including whitespace and the system prompt slot.
• Leaving evaluation examples in the training set because both were sampled from the same export.
• Using a full-fine-tune learning rate for LoRA, or the reverse. LoRA typically wants a rate ten times higher.
• Ignoring the base model’s licence. A derivative of Llama carries the Llama licence, a derivative of an Apache-2.0 model does not.
• Reporting a judge score without reading outputs. Preference-trained models find the judge’s blind spots before you do.

Going deeper

• Ouyang et al., Training language models to follow instructions with human feedback, 2022, the InstructGPT paper.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021, and Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023.
• Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023.
• Zhou et al., LIMA: Less Is More for Alignment, 2023.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023.
• The TRL documentation and PEFT documentation.