Diffusion models: forward noising and reverse denoising

What's actually being asked

Explain, conceptually and mechanistically, what a diffusion model does — in particular, the two named processes: forward diffusion and reverse denoising.

Answer

The basic idea

A diffusion model learns to generate data by first learning to systematically destroy it. Take a real sample x_0, and over many small steps, add a little Gaussian noise each time until nothing recognizable is left — pure noise x_T. That corruption process is fixed and requires no learning. The generative part is the reverse: train a neural network to undo one small noising step at a time. Once trained, start from pure random noise and repeatedly apply the learned "denoise a little" step T times, and what comes out the other end is a brand-new, realistic sample — nothing was ever literally "decoded," it was built up by progressively removing noise the network was trained to recognize at every scale. In prose, the chain looks like: noise (x_T) → U-Net predicts ε_θ → denoise step → repeat ×T → generated x_0.

Forward diffusion process (fixed, not learned)

Each step adds a small, controlled amount of Gaussian noise, defined by a schedule β_t: q(x_t|x_{t-1}) = N(x_t; √(1−β_t) x_{t-1}, β_t I). This is a Markov chain — each x_t depends only on x_{t-1} — and it has no trainable parameters; β_t is a fixed schedule, typically small and increasing with t. Because Gaussian noise composes cleanly, there's a closed-form shortcut straight from x_0 to any x_t without simulating every intermediate step: x_t = √(ᾱ_t) x_0 + √(1−ᾱ_t) ε, ε ~ N(0,I), where α_t = 1 − β_t and ᾱ_t = Π_{s=1}^{t} α_s is the cumulative product up to t. ᾱ_t shrinks as t grows (it's a product of numbers less than 1), so the original signal's weight √(ᾱ_t) shrinks while the noise's weight √(1−ᾱ_t) grows — by t = T, x_T is essentially indistinguishable from pure N(0,I) noise, which is exactly what makes it trivial to start sampling from.

Reverse denoising process (learned)

The true reverse step q(x_{t-1}|x_t, x_0) would require knowing x_0, which isn't available at generation time. So a neural network (typically a U-Net, conditioned on the timestep t) is trained to approximate it by predicting the noise that was added: ε_θ(x_t, t). Training objective: 𝔼[‖ε − ε_θ(x_t, t)‖²] — a simple regression: sample real data, pick a random t, add the corresponding known noise via the closed form above, and train the network to recover exactly that noise. At generation time, start from x_T ~ N(0,I), and repeatedly use ε_θ(x_t, t) to estimate and subtract the noise component, stepping back toward x_{t-1}, down to x_0.

Exam tip

A common mistake is saying the forward process is "learned too" — it isn't; it's a fixed Markov chain defined entirely by the noise schedule β_t. Only the reverse process has trainable parameters.