Comparing GAN loss variants and gradient quality

What's actually being asked

Lay the three objectives side by side and analyze the gradient signal each one gives, at different stages of training, and what that implies for how stable training is.

Answer

The three objectives

1. Discriminator loss (same regardless of which G variant is used): D maximizes log D(x) + log(1 − D(G(z))).
2. Saturating Generator loss (original minimax): G minimizes log(1 − D(G(z))).
3. Non-saturating Generator loss (practical default): G maximizes log D(G(z)).

Gradient analysis

Let d = D(G(z)) and look at the slope of each G objective with respect to d.

• Saturating: loss = log(1 − d), derivative = −1/(1 − d). Early in training d ≈ 0 (D easily spots fakes). The curve log(1−d) is flat near d=0 and only gets steep as d → 1. So the gradient that actually reaches G's parameters (through the chain rule) is informative late — once G is already winning — and nearly vanishing exactly when G most needs guidance. This is the textbook vanishing-gradient failure for G.
• Non-saturating: loss = −log d, derivative = −1/d. Near d ≈ 0 (G currently bad) this magnitude is large — a strong gradient exactly when G needs it. As d → 1 the gradient naturally shrinks — steep when needed, gentle near convergence. A much better-behaved training signal.

Discriminator's loss and stability

D's own loss doesn't change between the two G variants — it's always plain binary cross-entropy. D's gradient quality depends instead on the capacity balance between D and G: an over-powerful D confidently saturates (D(x)→1, D(G(z))→0) regardless of which G loss is in use, starving G of useful gradient under the saturating loss, and still demanding careful balancing even under the non-saturating loss.

Stability comparison

Objective	Early gradient for G	Main risk
Saturating G loss	weak / vanishing	G stalls, D "wins" trivially, training gets stuck
Non-saturating G loss	strong	avoids the stall, but is no longer a clean zero-sum minimax game — no strict theoretical convergence guarantee, can still oscillate
D loss (either case)	depends on D/G capacity balance	D too strong starves both G variants; D too weak gives noisy, uninformative gradient to G

Bottom line

Non-saturating loss is the practical default precisely because it fixes the saturating objective's early-training vanishing-gradient problem, at the cost of giving up the clean game-theoretic symmetry of the original minimax formulation — it becomes a heuristic, non-zero-sum game. That's part of why GAN training, while far more usable with the non-saturating loss, still has no guaranteed convergence and needs care (learning-rate balance between G and D, label smoothing, etc.) on top of the loss choice.

Exam tip

When asked to "compare," show the derivative/slope argument explicitly (why log(1−d) is flat near d=0 but −log d is steep there) — that concrete mechanism is what separates a full-marks answer from "non-saturating is better because it's used in practice."