Deriving the ELBO: posterior, prior, likelihood and KL

What's actually being asked

A clean, from-scratch derivation of the ELBO naming every term — q(z|x), p(z), the reconstruction likelihood, and the KL term — and why maximizing it is the right training objective for a generative model.

Answer

Setting up the quantities

• p(z) — the prior, a fixed, simple distribution over latent space, N(0, I). It's what we assume z looks like before seeing any data.
• q(z|x) — the approximate posterior, produced by the encoder: q(z|x) = N(μ(x), σ(x)²). It approximates the true posterior p(z|x) (how likely each z is, having seen x), which is intractable because it needs p(x) = ∫p(x,z)dz.
• p(x|z) — the reconstruction likelihood, produced by the decoder: how likely x is, given a latent code z.
• p(x) — the marginal likelihood/evidence we actually want to maximize but can't compute directly.

The derivation

Start from the log marginal likelihood and introduce q(z|x) by multiplying and dividing by it inside the integral:

log p(x) = log ∫ p(x,z) dz = log ∫ q(z|x) · p(x,z)/q(z|x) dz = log 𝔼_{q(z|x)} [p(x,z)/q(z|x)]

Because log is concave, Jensen's inequality lets us push the log inside the expectation (which only ever decreases the value):

log 𝔼_{q(z|x)}[p(x,z)/q(z|x)] ≥ 𝔼_{q(z|x)}[log(p(x,z)/q(z|x))]

Factor the joint p(x,z) = p(x|z)·p(z) and split the log:

𝔼_{q(z|x)}[log p(x,z) − log q(z|x)] = 𝔼_{q(z|x)}[log p(x|z) + log p(z) − log q(z|x)]

Group the reconstruction term separately from the prior/posterior ratio:

= 𝔼_{q(z|x)}[log p(x|z)] + 𝔼_{q(z|x)}[log p(z) − log q(z|x)] = 𝔼_{q(z|x)}[log p(x|z)] − 𝔼_{q(z|x)}[log q(z|x) − log p(z)] = 𝔼_{q(z|x)}[log p(x|z)] − D_KL(q(z|x) ‖ p(z))

That gives the Evidence Lower BOund:

log p(x) ≥ 𝔼_{q(z|x)}[log p(x|z)] − D_KL(q(z|x) ‖ p(z)) = ELBO

• First term: expected reconstruction likelihood under the approximate posterior — if we sample a z the encoder thinks plausible for this x, how likely is the decoder to reproduce x?
• Second term: D_KL(q(z|x)‖p(z)) = 𝔼_{q(z|x)}[log q(z|x) − log p(z)], the general KL-divergence formula D_KL(P‖Q)=Σ_x P(x)log(P(x)/Q(x)) applied with P=q(z|x), Q=p(z) — penalizes the approximate posterior for straying from the prior.

Negating gives the loss actually minimized in code: L(θ,φ;x) = −𝔼_{q_φ(z|x)}[log p_θ(x|z)] + D_KL(q_φ(z|x) ‖ p(z))

Why maximizing ELBO supports generative modelling

Since ELBO ≤ log p(x) always holds, pushing ELBO up pushes a lower bound on the data's log-likelihood up — the model is directly incentivized to assign higher probability to real data under its generative process p(z) → p(x|z). At the same time (shown by the alternate derivation log p(x) = ELBO + D_KL(q(z|x)‖p(z|x))), maximizing ELBO narrows the gap between the tractable q(z|x) and the true, intractable posterior p(z|x) — so the encoder becomes a better and better stand-in for exact Bayesian inference. Together, this is what lets the trained decoder, fed fresh z~p(z) at generation time, produce samples that look like they came from the real data distribution — the entire point of a generative model.

Worked intuition

Illustrative 1-D case: μ=0.0, σ=1.0 (i.e., q(z|x) = p(z) exactly) gives D_KL = −½(1+0−0−1)=0 — zero KL penalty, but then z carries no x-specific information at all, so the reconstruction term 𝔼[log p(x|z)] would be poor. Conversely μ=2.5, σ=0.05 (very peaked, very x-specific) gives a large D_KL but a near-perfect reconstruction. Training the ELBO balances exactly between these two extremes.

Exam tip

If the question says "derive," write every algebraic step (especially the Jensen's inequality line) — don't skip from the integral straight to the final ELBO formula even though you "know" the answer; derivation marks are given for the steps, not just the destination.