Comparing greedy, temperature, and top-k/top-p decoding

What's actually being asked

Run the same pretrained Transformer on three prompts, generate with three different decoding strategies (greedy, temperature sampling, top-k/top-p sampling), and compare what changes in diversity vs coherence.

Answer

The three decoding strategies, mechanically

All three start from the same thing — the model's next-token probability distribution P(v) = softmax(logits) — and differ only in how a token is picked from it:

• Greedy decoding: token = argmax_v P(v) at every step. Fully deterministic — same prompt always gives the same output.
• Temperature sampling: rescale the logits before softmax, P(v) = softmax(logits / T), then sample from the resulting distribution. T < 1 sharpens the distribution (closer to greedy, less diverse); T > 1 flattens it (more randomness, more diverse, but more risk of incoherence).
• Top-k / top-p (nucleus) sampling: restrict the sampling pool before sampling. Top-k keeps only the k highest-probability tokens (renormalized); top-p keeps the smallest set of tokens whose cumulative probability ≥ p (renormalized). Both cut off the long, unlikely tail that raw temperature sampling would otherwise occasionally sample from — giving diversity without picking obviously-wrong tokens.

Code

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
model.eval()
 
prompts = [
    "The future of artificial intelligence is",
    "In a quiet village, the old clock tower",
    "The key difference between machine learning and deep learning is",
]
 
def generate(prompt, **gen_kwargs):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        output_ids = model.generate(
            input_ids=inputs["input_ids"],
            attention_mask=inputs["attention_mask"],
            max_new_tokens=40,
            pad_token_id=tokenizer.eos_token_id,
            **gen_kwargs,
        )
    return tokenizer.decode(output_ids[0], skip_special_tokens=True)
 
for prompt in prompts:
    greedy_out   = generate(prompt, do_sample=False)
    temp_out     = generate(prompt, do_sample=True, temperature=1.2, top_k=0, top_p=1.0)
    nucleus_out  = generate(prompt, do_sample=True, top_k=50, top_p=0.9, temperature=0.8)
 
    print("PROMPT:", prompt)
    print("Greedy      :", greedy_out)
    print("Temperature :", temp_out)
    print("Top-k/Top-p :", nucleus_out)
    print("-" * 80)

top_k=0, top_p=1.0 on the temperature run disables the top-k/top-p filters so it's pure temperature sampling, isolating that strategy for a fair comparison.

Comparing diversity and coherence (qualitative)

• Greedy: highest coherence (every token is the single most likely continuation, so sentences stay grammatically and topically "safe"), but lowest diversity — running it twice on the same prompt gives the identical output, and it's known to fall into repetitive loops on longer generations ("...and the and the and the...").
• Temperature-only sampling (especially T > 1): noticeably higher diversity — different runs diverge quickly — but at the cost of coherence, since high temperature flattens the distribution enough that genuinely unlikely (sometimes off-topic or ungrammatical) tokens get picked occasionally.
• Top-k/top-p sampling: generally the best middle ground — it keeps the diversity benefit of sampling (different runs give different, natural-sounding continuations) while the cutoff prevents the model from ever sampling a token so unlikely it breaks coherence. This is why most production text-generation systems default to some form of top-k/top-p rather than raw temperature sampling or pure greedy.

No single strategy is "best" in an absolute sense — greedy suits deterministic/factual tasks (code completion, translation) where you want the single most probable answer; sampling-based methods suit creative/open-ended generation (stories, brainstorming) where variety is the point.

Exam tip

If asked to justify why top-k/top-p beats raw temperature sampling at the same diversity level, the answer is: temperature alone changes how flat the whole distribution is but still allows sampling from the entire (long) tail; top-k/top-p first truncates that tail to only plausible candidates, then samples — diversity from the sampling, safety from the truncation.