LLM Generation Parameters: Temperature, Top-p, Top-k Guide

Picture the evening request show on an Indore FM station. Listeners have sent in 200 song requests, and the RJ has to pick the next track.

A cautious RJ plays the most requested song every time. A bold one sometimes plays a rare request.

The station manager adds rules: "only pick from the top five", "only pick from songs that make up 90% of requests", "do not play the same song twice in an hour", and "the show ends at the closing jingle or at 9 pm, whichever comes first".

Every rule changes what listeners hear, even though the requests stayed the same. Generation parameters do the same job for an LLM.

What are LLM generation parameters?

Mapping the radio show: the request counts are the model's next-token probabilities. Boldness is temperature. "Top five only" is top-k. "Songs covering 90% of requests" is top-p. "No repeats" is a frequency or presence penalty. The closing jingle is a stop sequence, and 9 pm is max tokens.

LLM generation parameters are settings passed with each request that control how the next token is chosen from the model's probability distribution, trading predictability against variety and limiting length and repetition.

None of these settings change the model's weights. They only change how its output scores are turned into one chosen token at each step. The basics of temperature were introduced in what is an LLM; this article covers every other knob and how they combine.

Top-k, top-p and min-p: cutting off unlikely tokens

Sampling straight from the full distribution has a hidden problem. A vocabulary has tens of thousands of tokens, and even if each odd one has a tiny probability, together they add up. Over a 500-token answer, the model will eventually pick something absurd. Truncation methods remove that long tail before sampling.

• Top-k keeps a fixed number of candidates. Simple, but blind: k = 40 is too many when the model is sure and too few when many words fit.
• Top-p, also called nucleus sampling, keeps the smallest group of tokens whose probabilities add up to p. When the model is confident, the group is tiny; when it is unsure, the group widens.
• Min-p keeps every token whose probability is at least a fraction of the top token's. With min-p = 0.1 and a top token at 0.38, the bar is 0.038.

Same distribution, three cut-offs. Top-k counts tokens, top-p adds up probability, min-p compares with the leader.

Here are all three on the same distribution, the candidates after "Sarafa bazaar is famous for its":

import numpy as np

# next-token candidates after "Sarafa bazaar is famous for its"
tokens = np.array(["food", "jewellery", "garadu", "crowds", "sweets", "lights", "parking", "rain"])
probs = np.array([0.38, 0.22, 0.14, 0.09, 0.08, 0.04, 0.03, 0.02])

def show(name, keep):
    p = np.where(keep, probs, 0)
    p = p / p.sum()                                  # renormalise what survives
    kept = ", ".join(f"{t} {q:.2f}" for t, q, k in zip(tokens, p, keep) if k)
    print(f"{name:<11} {keep.sum()} kept: {kept}")

order = np.argsort(probs)[::-1]

k = 3                                                # top-k: a fixed count
keep = np.zeros(len(probs), bool); keep[order[:k]] = True
show("top-k=3", keep)

p_cut = 0.9                                          # top-p: smallest set reaching 90%
cum = np.cumsum(probs[order])
keep = np.zeros(len(probs), bool); keep[order[:np.searchsorted(cum, p_cut) + 1]] = True
show("top-p=0.9", keep)

min_p = 0.1                                          # min-p: at least 10% of the top prob
show("min-p=0.1", probs >= min_p * probs.max())

After filtering, the surviving probabilities are renormalised so they add up to 1 again, and the model samples from them. This is why "food" goes from 0.38 to 0.51 under top-k: it now competes with only two rivals.

How the parameters work together

Libraries apply these steps in a fixed order for every token. A common order is shown below; exact details differ slightly between libraries and servers.

Penalties and temperature reshape the scores; truncation removes the tail; only then is a token drawn.

The following function chains temperature and top-p, then draws 200 tokens at four settings so you can see the spread:

import numpy as np

tokens = ["food", "jewellery", "garadu", "crowds", "sweets", "lights", "parking", "rain"]
logits = np.array([2.6, 2.1, 1.6, 1.2, 1.0, 0.6, 0.1, -0.3])

def sample(logits, temperature=1.0, top_p=1.0, rng=None):
    if temperature == 0:
        return int(np.argmax(logits))                # greedy: no randomness at all
    z = logits / temperature
    p = np.exp(z - z.max()); p /= p.sum()
    order = np.argsort(p)[::-1]
    cutoff = np.searchsorted(np.cumsum(p[order]), top_p) + 1
    p_kept = np.zeros_like(p); p_kept[order[:cutoff]] = p[order[:cutoff]]
    return int(rng.choice(len(p), p=p_kept / p_kept.sum()))

for t, top_p in [(0, 1.0), (0.3, 1.0), (0.8, 0.9), (1.5, 1.0)]:
    rng = np.random.default_rng(seed=11)             # fixed seed: repeatable demo
    picks = [tokens[sample(logits, t, top_p, rng)] for _ in range(200)]
    counts = {w: picks.count(w) for w in tokens if w in picks}
    print(f"T={t:<4} top_p={top_p:<4} distinct={len(counts)}  {counts}")

At temperature 0 every draw is "food": no randomness at all. At 0.3 the model mostly agrees with itself. At 0.8 with top-p 0.9, you get variety but the tail ("parking", "rain") is cut off completely. At 1.5 with no top-p, everything is fair game, including the odd choices.

Should you change temperature and top-p together?

Usually change one and leave the other at its default. Both control variety, and moving both makes results hard to reason about. A common pairing is a moderate temperature (0.7 to 1.0) with top-p 0.9 or min-p 0.05, which lets the model be creative while blocking the weirdest tail tokens.

Frequency and presence penalties: stopping repetition

Models sometimes loop: "tasty, tasty and tasty". Penalties push already-used tokens down before sampling. In the OpenAI-style formula, each token's logit is reduced by count × frequency_penalty, plus presence_penalty once if it has appeared at all.

import numpy as np
from collections import Counter

tokens = ["tasty", "crispy", "spicy", "hot", "fresh"]
logits = np.array([3.0, 2.4, 2.2, 1.8, 1.5])
generated = "the poha was tasty tasty and tasty and crispy".split()

counts = Counter(generated)
freq_penalty, presence_penalty = 0.5, 0.4

adjusted = logits.copy()
for i, tok in enumerate(tokens):
    c = counts[tok]
    # frequency grows with every repeat; presence is a one-time charge
    adjusted[i] -= c * freq_penalty + (c > 0) * presence_penalty

def softmax(z):
    e = np.exp(z - z.max()); return e / e.sum()

for tok, c, before, after in zip(tokens, [counts[t] for t in tokens], softmax(logits), softmax(adjusted)):
    print(f"{tok:<7} used {c}x   prob before {before:.2f}   after {after:.2f}")

"tasty" was used three times, so its probability falls from 0.40 to 0.11, while unused words gain. Use penalties gently. A large value forces the model to avoid words it genuinely needs, like a product name or "the", and quality drops. Hugging Face uses a different, multiplicative repetition_penalty where 1.0 means off and values around 1.1 to 1.2 are typical.

Max tokens, stop sequences and seed

• Max tokens caps the output, which caps cost and latency. Set it with room to spare; if it is too low, answers and JSON get cut off mid-way. Check the finish reason in the response: a value like "length" means the cap was hit.
• Stop sequences end generation the moment a given string appears, and the string itself is not returned. They are useful for "one line only" tasks and for clean structured JSON output.
• Seed makes sampling repeatable where the provider supports it. Even then, servers may give slightly different results after updates, so treat it as "mostly repeatable".

Recommended settings by task

These are starting points. Always test on 20 or 30 of your own prompts and compare outputs side by side.

Setting generation parameters in code

With an OpenAI-compatible chat API, the parameters go straight into the request. This needs an API key and network access, so it will not run in the browser.

from openai import OpenAI

client = OpenAI()   # reads OPENAI_API_KEY; pass base_url= for any compatible server
resp = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Suggest 3 names for a poha stall."}],
    temperature=0.9,
    top_p=0.9,
    max_tokens=120,
    frequency_penalty=0.3,
    stop=["\n\n\n"],
    seed=7,
    extra_body={"min_p": 0.05},   # sent as-is; servers like vLLM and llama.cpp accept min_p
)
print(resp.choices[0].message.content, resp.choices[0].finish_reason)

With an open model in Hugging Face transformers, use generate(). Sampling is off unless you set do_sample=True.

outputs = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,          # without this, temperature and top_p are ignored
    temperature=0.7,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Some reasoning-focused models fix their own sampling settings and ignore or reject temperature, so read the model's documentation before tuning.

Common mistakes with generation parameters

• Raising temperature to fix wrong answers. It adds randomness, not knowledge. Improve the prompt with the prompt engineering guide instead.
• Forgetting do_sample=True in Hugging Face, then wondering why temperature does nothing.
• Setting max tokens too low for JSON, which produces truncated, unparseable output.
• Heavy penalties on technical text, which forces the model to avoid terms it must repeat.
• Assuming temperature 0 is perfectly deterministic across runs and server updates.

Key takeaways

• Generation parameters change how tokens are picked, never the model's weights.
• Temperature reshapes probabilities; top-k, top-p and min-p cut off the unlikely tail before sampling.
• Penalties fight repetition but hurt quality when set too high.
• Max tokens and stop sequences control length and clean endings; check the finish reason.
• Start from task-based defaults, change one setting at a time, and test on your own prompts.

The previous article explained mixture of experts. Next, LLM decoding strategies compares greedy decoding, beam search and sampling as whole approaches. To test what you learned, take a short MCQ round on Debugshala Test.