Mixture of Experts (MoE) Explained: How Sparse LLMs Work

Picture a large multi-speciality hospital on AB Road. It employs forty specialists: cardiologists, orthopaedic surgeons, dermatologists, ENT doctors. A patient walks in with a twisted ankle.

The reception desk does not send her to all forty doctors. It looks at her complaint and sends her to the two most relevant ones, an orthopaedic surgeon and a physiotherapist. She gets expert care in an hour, not a week.

The hospital still pays all forty salaries and needs rooms for all of them. But each patient only takes up the time of two. That is exactly the deal a mixture-of-experts LLM makes.

What is mixture of experts (MoE)?

Mapping the hospital to the model: each specialist is an expert, a small feed-forward network. The reception desk is the router (also called the gating network). The patient is one token. Seeing two doctors is top-2 routing. Salaries and rooms are memory; each patient's consultation time is compute.

Mixture of experts (MoE) is a neural network design in which a layer contains several parallel sub-networks called experts and a learned router that activates only a few experts for each input, so the model can hold many parameters while using only a fraction of them per token.

In a standard transformer, every token passes through the same feed-forward network (FFN) in every block, as described in transformers and attention explained. Models built that way are called dense.

In an MoE transformer, attention stays the same, but the FFN is swapped for a set of experts. Such models are called sparse, because most experts sit idle for any given token.

Dense FFN vs MoE layer: what changes in the transformer

A dense layer runs one big FFN for every token; an MoE layer runs only the experts the router picks.

The idea is older than LLMs. Google's GShard (2020) and Switch Transformer (2021) scaled MoE to very large language models, and Mistral AI's Mixtral 8x7B, released in December 2023, made it popular for open-weight models. Several of today's largest open models are MoE designs.

Two details matter:

• Routing happens per token and per layer. In one sentence, "refund" might go to experts 2 and 5 in layer 10 and to experts 1 and 7 in layer 11. There is no single "billing expert" for a whole request.
• Experts are not hand-assigned topics. Nobody tells expert 3 to handle cricket. Specialisation emerges in training and is often about token patterns, punctuation or syntax rather than human-friendly subjects.

How the MoE router picks experts

The router is surprisingly simple: a single learned weight matrix that gives each expert a score for the current token. The steps are:

1. Multiply the token's vector by the router matrix to get one score per expert.
2. Keep the top k scores, usually 1 or 2 in older models and up to 8 in newer fine-grained designs.
3. Apply softmax over just those k scores to get gate weights that add up to 1.
4. Run the token through the chosen experts and add their outputs, each multiplied by its gate weight.

import numpy as np

rng = np.random.default_rng(42)
d, n_experts, top_k = 6, 8, 2
tokens = ["refund", "my", "order"]
X = rng.normal(size=(len(tokens), d))              # token vectors entering the layer

W_router = rng.normal(size=(d, n_experts))         # learned with the rest of the model
experts = [rng.normal(0, 0.3, (d, d)) for _ in range(n_experts)]  # tiny "FFNs"

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

for tok, x in zip(tokens, X):
    logits = x @ W_router                          # one score per expert
    chosen = np.argsort(logits)[-top_k:][::-1]     # indices of the top-2 experts
    gates = softmax(logits[chosen])                # renormalise over the chosen two
    y = sum(g * (x @ experts[e]) for g, e in zip(gates, chosen))
    picks = ", ".join(f"expert {e} ({g:.2f})" for e, g in zip(chosen, gates))
    print(f"{tok:<7} -> {picks}   output norm {np.linalg.norm(y):.2f}")

print(f"\nEach token used {top_k} of {n_experts} experts = {top_k / n_experts:.0%} of the FFN weights")

Each token took a different route, and the gate weights show how much each chosen expert contributed. The router's weights are trained jointly with everything else, so it gradually learns which experts handle which kinds of token well.

Total vs active parameters

This is the number you will see on every MoE model card. Total parameters is everything stored, all experts included. Active parameters is what one token actually uses: attention, embeddings and only the k chosen experts.

Here is the calculation for a configuration shaped like Mixtral 8x7B, using the layer sizes from its public configuration file:

# a configuration shaped like Mixtral 8x7B (sizes from its public config)
layers, d_model, d_ff, vocab = 32, 4096, 14336, 32000
n_experts, top_k = 8, 2

expert = 3 * d_model * d_ff                        # gate, up and down matrices
attention = 2 * d_model * d_model + 2 * d_model * 1024   # Q, O full; K, V grouped
embeddings = 2 * vocab * d_model                   # input table + output layer

total = layers * (n_experts * expert + attention) + embeddings
active = layers * (top_k * expert + attention) + embeddings

print(f"total parameters : {total / 1e9:5.1f} B   (must all sit in memory)")
print(f"active per token : {active / 1e9:5.1f} B   (decides compute per token)")
print(f"active share     : {active / total:.0%}")
for name, bytes_per in [("fp16", 2), ("4-bit", 0.5)]:
    print(f"weights at {name:<5}: about {total * bytes_per / 1e9:.0f} GB")

The result matches the published figures of about 46.7 billion total and 12.9 billion active parameters. Notice the name is misleading: "8x7B" is not 56B, because attention and embeddings are shared rather than copied eight times.

An MoE model needs the memory of its total size but the per-token compute of its active size.

So an MoE model generates text at roughly the speed of a 13B dense model while carrying the knowledge capacity of a much larger one. Larger designs push this further. DeepSeek-V3, for example, reports 671 billion total parameters with about 37 billion active per token, using many small experts plus a "shared" expert that every token always visits.

Load balancing: keeping every expert busy

Left alone, a router tends to collapse. Early in training a couple of experts get slightly better, so the router sends them more tokens, so they improve faster, and the rest starve. You end up paying memory for eight experts and using two.

The standard fix is an auxiliary load-balancing loss added to the training loss. In the Switch Transformer version, it multiplies each expert's share of tokens by its average router probability and sums the products. It is lowest when tokens are spread evenly.

import numpy as np

rng = np.random.default_rng(1)
n_tokens, E = 2000, 8

def aux_loss(router_logits):
    p = np.exp(router_logits)
    p /= p.sum(axis=1, keepdims=True)              # router probabilities per token
    top1 = p.argmax(axis=1)
    f = np.bincount(top1, minlength=E) / n_tokens  # share of tokens each expert got
    P = p.mean(axis=0)                             # average probability per expert
    return E * np.sum(f * P), np.bincount(top1, minlength=E)

balanced = rng.normal(size=(n_tokens, E))
skewed = balanced + np.array([3, 2, 0, 0, 0, 0, 0, 0])   # router favours experts 0 and 1

for name, logits in [("balanced", balanced), ("collapsed", skewed)]:
    loss, counts = aux_loss(logits)
    print(f"{name:<10} tokens per expert {counts}  aux loss {loss:.2f}")
print("The loss is smallest (1.0) when every expert gets an equal share.")

The collapsed router sends three quarters of all tokens to expert 0, and the auxiliary loss jumps from 1.0 to 3.5. Training pushes that number back down.

Systems also set a capacity factor, a cap on how many tokens one expert may take per batch, so a popular expert cannot become a bottleneck. Some newer models balance load with an adjustable bias on router scores instead of an extra loss term.

MoE vs dense models: the trade-offs

For someone running models on one machine, the memory row is the one that bites. A 47B-total MoE needs roughly the memory of a 47B dense model, even though it runs like a 13B one.

The memory maths from open-source LLMs explained applies to total parameters, not active ones. Some local runtimes can keep the experts in system RAM and the shared layers on the GPU, which is a practical compromise covered in how to run LLMs locally.

Common mistakes and myths about MoE

• "Active parameters decide the hardware." They decide speed. Memory is set by the total.
• "Each expert is a topic specialist." Studies of real routers find specialisation is mostly about token types and syntax, not subjects like law or medicine.
• "MoE is a separate ensemble of models." It is one model; the experts live inside each layer and are trained together.
• "Top-1 is always cheaper and just as good." Fewer active experts saves compute but can hurt quality and stability; designs trade these off carefully.

Key takeaways

• MoE replaces each FFN with several experts plus a router that activates only the top k per token.
• Total parameters decide memory; active parameters decide compute and speed.
• Routing is learned and happens per token, per layer; experts do not map neatly to human topics.
• A load-balancing loss or bias keeps tokens spread across experts and prevents router collapse.
• MoE gives more capacity per unit of compute, at the price of memory and serving complexity.

This article follows how LLMs are trained. Next, LLM generation parameters moves from architecture to the settings you control every time you call a model. For a quick self-test, attempt the MCQ practice sets on Debugshala Test.