Mathematical foundations of generative vs discriminative learning

What's actually being asked

State the mathematical objective each paradigm optimizes, and walk through one classification example showing both approaches side by side.

Answer

Mathematical foundation

A classification problem is, at its core, a question about the relationship between input features x and a label y. There are exactly two mathematically valid ways to get from x to a prediction of y:

• Generative learning models the joint distribution P(x,y). In practice this is done by learning the class prior P(y) and the class-conditional likelihood P(x|y) separately from data, since P(x,y) = P(x|y)·P(y). At prediction time, Bayes' theorem converts this into what you actually want: P(y|x) = P(x|y)P(y) / P(x), where P(x) = Σ_y P(x|y)P(y). The model never directly optimizes the quantity it's evaluated on (P(y|x)) — it gets there indirectly, through the joint.

• Discriminative learning models P(y|x) directly, skipping the joint entirely. There's no term for P(x) or P(x|y) anywhere in the model — it only ever learns the boundary/mapping that best predicts y from x, typically by directly minimizing a loss like cross-entropy over (x,y) training pairs.

Worked classification example: email spam detection

Let y ∈ {spam, not-spam} and x = presence of the word "free" in the email.

Generative approach (Naïve Bayes):

1. Estimate the prior from training data, e.g. P(spam) = 0.4, P(not-spam) = 0.6 (illustrative numbers — use your actual training split's counts).
2. Estimate the likelihood, e.g. P("free" | spam) = 0.7, P("free" | not-spam) = 0.1.
3. Apply Bayes' theorem to classify: P(spam | "free") = P("free"|spam)P(spam) / [P("free"|spam)P(spam) + P("free"|not-spam)P(not-spam)] = (0.7×0.4) / (0.7×0.4 + 0.1×0.6) = 0.28 / 0.34 ≈ 0.82. The model built the joint first, then derived this number.

Discriminative approach (Logistic Regression): The model directly learns weights w, b and outputs P(spam|x) = σ(w·x + b) (sigmoid), trained by minimizing cross-entropy loss between predicted and true labels over the training set — it never estimates P("free"|spam) or P(spam) as separate quantities at all; it fits the conditional boundary end-to-end.

Both can reach a similar final number for P(spam|x), but the generative route passes through an explicit model of how spam/non-spam emails generate the word "free," while the discriminative route jumps straight to the decision — which is also why only the generative model could, in reverse, be asked to generate a plausible new spam-like email.

Why this matters mathematically

The generative route's intermediate quantities (P(spam), P("free"|spam)) are each individually meaningful and reusable — you could use the same P("free"|spam) estimate to answer a completely different question, like "how unusual is this email overall?" The discriminative route's learned weights w, b have no such standalone interpretation — they exist only to produce a good P(y|x), nothing else. That's the real mathematical cost of skipping the joint: you gain a simpler, often more accurate training problem, but you lose every other probabilistic question the joint distribution could have answered.

Exam tip

The phrase examiners are listening for is "the generative model learns P(x,y) and derives P(y|x) via Bayes' theorem, while the discriminative model learns P(y|x) directly, with no model of P(x)." Say it in those exact terms, then back it with the worked example.