Bayes' theorem derivation and full generative-discriminative comparison

What's actually being asked

A full explanation of both paradigms, a step-by-step algebraic derivation of Bayes' theorem connecting P(x,y), P(x|y), P(y), P(y|x), and a detailed multi-axis comparison.

Answer

The two paradigms

Generative learning builds a model of how the data was generated: it estimates the class prior P(y) and the class-conditional likelihood P(x|y), which together define the joint P(x,y). Classification/prediction is then a derived step via Bayes' theorem, and generation of new x falls out for free since the model knows P(x|y).

Discriminative learning skips modeling how x arises altogether and learns P(y|x) (or a direct decision function) straight from data, by minimizing a loss that only cares about getting y right given x.

Deriving P(y|x) from P(x|y) and P(y) — step by step

Start from the definition of conditional probability, applied two ways to the same joint P(x,y):

Step 1. By definition, the conditional probability of y given x is: P(y|x) = P(x,y) / P(x) ... (i)

Step 2. By the same definition, applied the other way round, the conditional probability of x given y is: P(x|y) = P(x,y) / P(y) ... (ii)

Step 3. Rearrange (ii) to isolate the joint probability: P(x,y) = P(x|y) · P(y) ... (iii)

Step 4. Substitute (iii) into (i), replacing P(x,y): P(y|x) = [P(x|y) · P(y)] / P(x) ... (iv)

Step 5. Expand P(x) using the law of total probability — since y can take any of its possible values y', sum the joint over all of them: P(x) = Σ_y' P(x, y') = Σ_y' P(x|y') P(y')

Step 6. Substitute this into (iv) to get the final, fully expanded form of Bayes' theorem: P(y|x) = P(x|y) P(y) / Σ_y' P(x|y') P(y')

This is exactly the canonical form P(y|x) = P(x|y) P(y) / P(x), with P(x) written out as a normalizing constant that sums the numerator's shape over every possible class y' — it's what makes the posterior probabilities over all classes add up to 1.

Detailed comparison

Axis	Generative	Discriminative
Mathematical formulation	Models P(x,y) via P(x|y), P(y); obtains P(y|x) through the derivation above	Models P(y|x) directly as a parametric function of x (e.g. σ(w·x+b))
Model training	Estimate priors and likelihoods from data (often closed-form, e.g. counting/Gaussian fitting)	Iterative optimization (gradient descent) of a conditional loss (cross-entropy, hinge loss)
Decision boundary	Implicit, indirect — emerges from comparing posteriors across classes; shape constrained by the assumed likelihood form	Explicit, learned directly to separate classes as accurately as possible on the training distribution
Computational complexity	Can be cheaper to train (closed-form estimates for simple likelihoods like Naïve Bayes) but modeling the full P(x) can be expensive for high-dimensional x	Training is typically iterative/costlier per step, but avoids ever modeling the expensive high-dimensional P(x)
Advantages	Can generate new data; works reasonably with less labeled data; naturally handles missing features via marginalization	Typically higher classification accuracy with enough labeled data; fewer distributional assumptions to get wrong
Limitations	Accuracy suffers if the assumed likelihood form (e.g., Gaussian, feature independence) doesn't match reality	Cannot generate new samples; usually needs more labeled data; offers no insight into how the data is structured
Real-world applications	Image/text generation, data augmentation, anomaly detection, spam filtering via Naïve Bayes	Logistic regression credit scoring, SVM/NN image classifiers, discriminative sequence tagging

Exam tip

This is the "show full working" question — examiners specifically want to see the six-step algebraic derivation written out, not just the final Bayes' formula quoted. Losing the intermediate P(x,y) = P(x|y)P(y) step and the total-probability expansion of P(x) is the single most common way marks get dropped here.