Generative models

Overview

A discriminative model learns a boundary. A generative model learns the whole distribution of the data and can draw new samples from it. That one difference explains why generative models are harder to train, harder to evaluate and far more useful: the same machinery that produces an image from a caption can inpaint, upscale, denoise, generate audio, propose protein structures or fill in missing sensor readings.

This hub is for engineers who need to pick a family, not a paper. At the time of writing five families cover nearly everything in production: variational autoencoders, generative adversarial networks, diffusion models, flow-matching models and autoregressive models. Large language models are autoregressive generative models of text and have their own hub at /topics/transformers-and-llms/. This page is about the rest, and about the ideas the families share.

Key ideas

Every family is a different answer to one question

The question is how to turn noise into data. A VAE (Kingma and Welling, 2013) learns an encoder that maps data to a small latent vector and a decoder that maps it back, with a loss that keeps the latent space close to a Gaussian. Sampling is one decoder pass, so it is fast, but the reconstruction term averages over possibilities and the samples are blurry. A GAN (Goodfellow et al., 2014) trains a generator against a discriminator that tries to spot fakes. The samples are sharp and sampling is one pass, but training is a two-player game that can oscillate or collapse to a few modes, and there is no likelihood to monitor. An autoregressive model factorises the joint distribution into a product of conditionals and predicts one token, pixel or audio sample at a time. It gives exact likelihoods and scales well, but sampling costs one forward pass per element.

Diffusion learns to undo noise

A diffusion model (Ho et al., 2020) takes the opposite route. The forward process is fixed: add a small amount of Gaussian noise to the data over T steps until nothing but noise is left. The model learns the reverse process, a network that looks at a noisy input and the step index and predicts the noise that was added. Training is ordinary supervised regression on that noise prediction, which is why diffusion training is stable where GAN training is not. Sampling runs the learned denoiser backwards from pure noise, typically 20 to 50 steps with a modern solver. Latent diffusion (Rombach et al., 2022) runs the whole process in the latent space of a pretrained VAE rather than in pixel space, which is what made Stable Diffusion affordable on one GPU.

Flow matching straightens the path

Flow matching (Lipman et al., 2022) and rectified flow (Liu et al., 2022) keep the diffusion training recipe but change the target. Instead of a noise schedule that bends through many small steps, the model learns the velocity of a straight-line interpolation between a noise sample and a data sample. Straight paths can be integrated with fewer steps, and the objective is a plain regression on velocity. Stable Diffusion 3 (Esser et al., 2024) and the FLUX.1 family use this formulation. FLUX.1 [dev] (12 B parameters, text-to-image, FLUX.1 [dev] Non-Commercial Licence, August 2024) is the open-weights example most people will meet. Its training data is undisclosed.

Conditioning and guidance decide what you get

A generative model that only produces random samples is a curiosity. Conditioning on text, class labels, a sketch or a low-resolution input turns it into a tool. For diffusion and flow models the standard trick is classifier-free guidance (Ho and Salimans, 2022): train with the condition randomly dropped, then at sampling time push the prediction away from the unconditional one by a guidance scale. Higher scales follow the prompt more closely and lose diversity, and past a point they saturate colours and introduce artefacts. The text encoder matters as much as the denoiser, which is why recent models bolt on a large language model as the conditioner.

Where to start

Read /learn/beginner/04-training-loop-and-loss/ first, because every family here is a training loop with an unusual loss. Then /learn/beginner/07-your-first-neural-network/ for the network side and /learn/beginner/08-embeddings-and-similarity/ for the latent-space intuition that VAEs and latent diffusion rely on. The /projects/mnist-cnn-pytorch/ project is a good base to extend into a small VAE on the same data. The /topics/deep-learning/ hub covers the U-Net and transformer backbones, and /topics/computer-vision/ and /topics/speech-and-audio/ cover the domains where these models are used most.

Tools and platforms

Common mistakes

• Judging a model by a handful of cherry-picked samples. Use FID or CLIP score on a fixed prompt set, and read /topics/evaluation-and-benchmarks/ on why even those drift.
• Training a GAN as a first generative model. Start with a VAE or a small diffusion model, where a falling loss means something.
• Setting the guidance scale by habit. The right value depends on the model and the solver, and the default in a pipeline was tuned for someone else’s prompts.
• Running latent diffusion with the wrong VAE. The decoder and the denoiser are trained together, and swapping one silently degrades every sample.
• Ignoring the licence on the weights. Most open image models are open weights with use restrictions, not open source. See /topics/open-source-ml/.
• Forgetting that samplers are hyperparameters. Twenty steps of a modern ODE solver beat a thousand steps of the original ancestral sampler, and the choice changes the output.

Going deeper

• Kingma and Welling, Auto-Encoding Variational Bayes, 2013.
• Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models, 2020.
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, 2022.
• Lipman et al., Flow Matching for Generative Modeling, 2022.
• Ho and Salimans, Classifier-Free Diffusion Guidance, 2022.
• Hugging Face Diffusers documentation, including the training examples.