Text generation with a pretrained Transformer

What's actually being asked

Show real, runnable code that loads a pretrained Transformer (GPT-2, via Hugging Face transformers), tokenizes a prompt, runs autoregressive inference, decodes the resulting token ids back to text, and prints the result.

Answer

Pipeline

prompt string → tokenizer → input_ids + attention_mask → model.generate() (autoregressive loop) → output token ids → tokenizer.decode() → generated text

1. Tokenization

The tokenizer converts the raw prompt string into subword token ids using GPT-2's byte-pair encoding (BPE) vocabulary, and also returns an attention_mask (all 1s here since there's no padding in a single-sequence batch).

2. Model inference + decoding (code)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.eval()
 
prompt = "Generative AI is transforming the way we"
inputs = tokenizer(prompt, return_tensors="pt")
 
with torch.no_grad():
    output_ids = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=40,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.8,
        pad_token_id=tokenizer.eos_token_id,
    )
 
generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(generated_text)

3. What each stage is actually doing

• Tokenization: tokenizer(prompt, return_tensors="pt") turns the string into a [1, seq_len] tensor of integer ids plus the attention mask, both of which model.generate needs.
• Model inference: internally, generate() runs the decoder forward once to get logits for the next token, picks/samples one token (here, sampling with top_k/top_p/temperature rather than plain greedy), appends it to the sequence, and repeats — one forward pass per new token, up to max_new_tokens. pad_token_id=tokenizer.eos_token_id is set because GPT-2 has no dedicated pad token.
• Decoding: tokenizer.decode(...) maps the output integer ids back to text, merging subword pieces and dropping special tokens.

4. Final generated text

Example output (illustrative — GPT-2's output is stochastic with do_sample=True, so the exact wording will differ run to run; this shows the shape of a correct result, not a guaranteed literal output):

Generative AI is transforming the way we build software, create art, and
interact with machines — from writing assistants to image generators that
can turn a short text prompt into a finished picture in seconds.

5. Deterministic variant, for contrast

Swapping do_sample=True + top_k/top_p/temperature for simply do_sample=False switches to greedy decoding — at every step the single highest-probability next token is picked (argmax), so the same prompt always produces the exact same output. That's useful to know as the baseline this sampling-based run is being compared against: greedy is deterministic and safe but can repeat itself on longer generations, which is exactly why sampling with top_k/top_p is used here instead.

Exam tip

Be explicit that generate() is not one forward pass — it's a loop of max_new_tokens forward passes, each one conditioned on everything generated so far (that's the autoregressive property). Examiners sometimes ask what happens if you forget attention_mask or pad_token_id — GPT-2 will still run but may warn/misbehave on batched inputs with padding, since it has no native pad token.