A "dense" language model (the standard architecture for years) uses every one of its parameters on every single query it processes. A mixture-of-experts (MoE) model doesn't: it's built from many specialized sub-networks ("experts"), and for any given input, a routing mechanism activates only a subset of them. The model might have hundreds of billions of total parameters while only using a fraction of that for any single query. This architectural choice has become common across several major frontier and open-source model families, and it's worth understanding why.

The core trade-off it addresses

Model quality has historically scaled with parameter count. Bigger models, trained on enough data, tend to perform better. But a dense model's inference cost scales directly with its total size, because every parameter is used on every query, which means a bigger dense model is proportionally more expensive to run, every time, for every user. Mixture-of-experts breaks that direct link: you can have a very large total parameter count (and the capability that comes with it) while keeping the *active* computation for any single query much smaller, because routing sends each query to only a handful of relevant experts rather than the whole network.

How the routing actually works

A learned routing mechanism, trained alongside the rest of the model, decides which experts should handle each token as it's processed. This isn't a hand-designed rule ("route math questions to the math expert"), the specialization that emerges is learned during training and often isn't cleanly interpretable along human-legible categories; experts don't reliably correspond to "the coding expert" or "the history expert" in a way you could label and explain, even though the routing does produce genuine, measurable specialization patterns.

Why this matters for cost, not just capability

The practical payoff shows up directly in inference economics: a mixture-of-experts model can offer capability closer to a much larger dense model's, at an inference cost closer to a much smaller one's. Because you're paying, computationally, for the experts actually activated on a given query, not the full parameter count. This is a large part of why MoE architectures have become common in models designed to be served at scale to many users simultaneously, where inference cost, not just training cost, is a first-order business constraint.

The trade-offs that come with it

Sparse activation isn't a free lunch. MoE models require holding all the experts in memory even though only a few are active per query, which shifts the cost bottleneck from computation to memory rather than eliminating it. Training MoE models also introduces its own challenges. Keeping routing balanced across experts so some don't sit chronically underused while others become overloaded is a genuinely nontrivial optimization problem layered on top of the usual challenges of training a large model.

Where this shows up in the current model landscape

Several prominent model families, including some of the more capable open-weight releases covered in our piece on the open-weight surge, have adopted mixture-of-experts architectures specifically to offer strong capability at a more serveable cost, a strategic choice that's increasingly common as more of the field's competition shifts from "who has the most capable model" toward "who has the most capable model at a cost that's actually viable to serve at scale," a shift also visible in the pricing dynamics we cover in our Claude 3.5 Sonnet retrospective.

The takeaway

Mixture-of-experts is best understood as an efficiency architecture, not a capability architecture in its own right. It's a way to get more usable capability per dollar of inference cost, which has made it an increasingly standard tool in frontier model design rather than a niche technique.