What is an LLM? How Large Language Models Actually Work

Picture an aunty who has run a tiffin service in Vijay Nagar for twenty years. You call and say "Two thalis with dal, rice and..." and she says "...extra papad?" before you finish. She has not memorised your sentence. She has heard thousands of orders and knows what usually comes next.

Now imagine a system that has read a huge slice of public books, websites and code, and plays exactly that game: given some text, guess what comes next. It plays it so well that the guesses add up to essays, emails, summaries and working programs.

That system is a large language model, and nearly everything in generative AI is built on this one skill.

What is an LLM? The analogy mapped to real terms

Here is the tiffin aunty mapped to the real vocabulary.

A large language model (LLM) is a neural network with billions of learned parameters, trained on very large amounts of text to predict the next token, and used to generate text by repeating that prediction one token at a time.

"Language model" is an old idea: a system that assigns probabilities to word sequences. What changed is scale, plus an architecture called the transformer, which lets the model look at every earlier token when predicting the next one.

Tokens: how an LLM reads text

An LLM never sees letters or words directly. It sees tokens: chunks of text that are often a whole common word, part of a rarer word, a number or a punctuation mark. A word like "unbelievable" might become "un", "believ" and "able".

Each token is mapped to an integer ID from a fixed vocabulary, which for modern models usually holds tens of thousands to a few hundred thousand entries. The model only ever does maths on those IDs.

Why should you care? Because usage limits, pricing and the context window are all counted in tokens, not words. A handy rule for English is that 100 tokens are roughly 75 words. Hindi and other non-Latin scripts often need more tokens per word.

Here is a simple tokenizer that splits words, numbers and punctuation. Real tokenizers learn their pieces from data (a method called byte-pair encoding is common), but the counting idea is the same.

import re

def tokenize(text):
    # words, numbers and single punctuation marks become tokens
    return re.findall(r"[A-Za-z]+|\d+|[^\sA-Za-z\d]", text)

text = "Poha costs 30 rupees at Sarafa, isn't that cheap?"
tokens = tokenize(text)

print(tokens)
print("Words :", len(text.split()))
print("Tokens:", len(tokens))

"isn't" became three tokens and the punctuation got its own, which is why token counts beat word counts. Next, map tokens to IDs the way a vocabulary does.

sentence = "the chai is hot and the poha is fresh"

vocab, ids = {}, []
for word in sentence.split():
    if word not in vocab:
        vocab[word] = len(vocab)   # give each new token the next free ID
    ids.append(vocab[word])

print("Vocabulary:", vocab)
print("Token IDs :", ids)

If loops are new to you, the Python for loop tutorial covers everything this code uses.

Next-token prediction: the one trick behind everything

Give the model "Indore is famous for" and it does not return a word. It returns a probability for every token in its vocabulary: maybe 0.41 for "poha", 0.22 for "Sarafa", 0.09 for "its" and tiny values for the rest. A decoding step picks one token, appends it to the text, and the whole thing runs again.

An LLM generates text by predicting one token, appending it, and feeding the longer text back in.

This is called autoregressive generation. A 300-token answer means roughly 300 passes through the network. That is why long answers take longer and why you see replies "typing" on screen.

The surprising part is how much ability falls out of this simple goal. To predict the next token well across billions of sentences, the model has to pick up grammar, facts, writing styles and common patterns of reasoning.

The flip side is that it is trained to produce plausible text, not verified text, which is why LLMs sometimes state wrong things confidently. This is called hallucination.

Build a toy next-word model in pure Python

The simplest language model is a bigram model: it counts which word follows which, then samples the next word using those counts. It is nowhere near an LLM, but the generation loop is the same.

import random
from collections import defaultdict, Counter

corpus = """indore is famous for poha . indore is famous for sarafa bazaar .
people in indore love poha and jalebi . people love sarafa bazaar at night .
poha and jalebi is a classic breakfast . people in indore love chai ."""
words = corpus.split()

bigrams = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    bigrams[current][nxt] += 1

print("After 'famous':", dict(bigrams["famous"]))
print("After 'love'  :", dict(bigrams["love"]))

random.seed(7)
word, output = "people", ["people"]
for _ in range(12):
    options = bigrams[word]
    word = random.choices(list(options), weights=list(options.values()))[0]
    output.append(word)
print("Generated:", " ".join(output))

Try other seeds. The sentences sound locally sensible but wander, because the model sees only one previous word. A real LLM looks at thousands of previous tokens at once and uses learned weights instead of raw counts, which is what lets it stay on topic for pages.

Training vs inference: two very different jobs

People mix these up constantly.

Training itself usually has stages. Pretraining on raw text produces a model that can continue any text. Instruction tuning on examples of questions and good answers turns it into a helpful assistant. Preference tuning, where humans or other models rate answers, nudges it toward replies people prefer.

Two consequences matter in practice. First, a model knows nothing after its training cutoff date unless you give it fresh text in the prompt. Second, the model does not learn from your chat during inference. If it seems to remember you, that is because earlier messages are being sent again inside the context window.

The context window: an LLM's working memory

The context window is the maximum number of tokens the model can consider in one go. It includes everything: system instructions, the chat history, any pasted documents, and the answer being written. Current models range from a few thousand tokens to hundreds of thousands.

When a conversation outgrows the window, the application must drop or summarise older parts. The simplest strategy keeps only the most recent tokens:

history = ("user asked about poha prices then asked about sarafa timings "
           "then asked which bus goes to rajwada and finally asked about chai").split()

window = 8   # pretend the model can only see 8 tokens
visible = history[-window:]
dropped = history[:-window]

print("Total tokens :", len(history))
print("Model sees   :", " ".join(visible))
print("Forgotten    :", " ".join(dropped))

A bigger window is not a free pass. Models can pay less attention to facts buried in the middle of a very long prompt, and every extra token costs time and money. That is why most real systems fetch only the relevant passages, an approach explained in What is RAG?

Temperature: how adventurous should the model be?

Before picking a token, the model produces raw scores called logits. A function called softmax turns them into probabilities. Temperature divides the logits before softmax: a low temperature sharpens the distribution so the top token almost always wins, and a high temperature flattens it so less likely tokens get a real chance.

import numpy as np

words = ["poha", "jalebi", "samosa", "pizza"]
logits = np.array([3.0, 2.0, 1.0, 0.2])   # raw scores from the model

def softmax(z, temperature):
    z = z / temperature
    e = np.exp(z - z.max())               # subtract max for numerical safety
    return e / e.sum()

for t in [0.2, 1.0, 2.0]:
    probs = softmax(logits, t)
    row = ", ".join(f"{w} {p:.2f}" for w, p in zip(words, probs))
    print(f"T={t}: {row}")

Same logits, three temperatures. Lower values make the model predictable; higher values make it varied.

Use a low temperature (0 to 0.3) for extraction, classification and code, where you want the same reliable answer. Use a moderate one (around 0.7 to 1.0) for brainstorming or creative writing. Many APIs also offer top-p, which samples only from the smallest set of tokens whose probabilities add up to p.

Common mistakes beginners make with LLMs

• Treating output as fact. The model optimises for plausible text. Check names, numbers and citations.
• Assuming it learns from your chat. It does not update its weights; it only rereads the conversation each turn.
• Counting words instead of tokens. Limits and bills are in tokens, and non-English text uses more of them.
• Raising temperature to fix a weak answer. Randomness rarely fixes a vague request. A clearer prompt does; see the prompt engineering guide.
• Pasting everything into the prompt. Long, unfocused context is slower and often less accurate than a few relevant passages.

Key takeaways

• An LLM predicts the next token and generates text by repeating that step.
• Tokens are sub-word pieces; limits and costs are counted in tokens.
• Training changes the weights once; inference uses frozen weights for every request.
• The context window is the model's only memory within a conversation.
• Temperature trades predictability for variety; keep it low for factual tasks.

To go further, read how text meaning becomes numbers in Embeddings Explained, then test your basics with practice MCQs on Debugshala Test.