Nearly every large language model in production today (GPT, Claude, Gemini, Llama, Mistral, all of them) is a descendant of a single 2017 paper out of Google: "Attention Is All You Need." It's one of the most cited papers in computer science, which also means it's one of the most cited-without-being-read. Here's what it actually argued, in plain terms.

The problem it was solving

Before 2017, the dominant approach to processing sequences of text (for translation, summarization, and similar tasks) relied on recurrent neural networks (RNNs) and their more capable variant, LSTMs. These models processed text one word at a time, in order, carrying forward a running "memory" of everything they'd seen so far. That sequential processing was a fundamental bottleneck: it made these models slow to train, because you couldn't easily parallelize the computation, and it made them prone to "forgetting" information from early in a long passage by the time they reached the end.

The idea: attention, without the recurrence

The paper's core proposal was to drop the recurrent, one-word-at-a-time processing entirely and replace it with a mechanism called self-attention. Instead of processing a sentence in sequence, a transformer looks at all the words in a passage simultaneously and computes, for every word, how much it should "attend to" every other word when building its representation. A pronoun can directly attend to the noun it refers to, regardless of how many words sit between them, instead of relying on that connection surviving a long chain of sequential updates.

This had two enormous practical effects. First, because there's no sequential dependency, the computation parallelizes efficiently on GPUs. You can process an entire passage at once instead of word by word, which made it dramatically cheaper to train on more data. Second, it turned out to scale unusually well: bigger transformers, trained on more data, kept getting reliably better in ways that weren't holding for the architectures that came before them.

"Multi-head" attention and why it's not just one mechanism

The paper's models don't compute a single attention pattern. They compute several in parallel ("multi-head attention"), each one potentially learning to track a different kind of relationship: one head might specialize in tracking grammatical structure, another in tracking topical relevance across a longer span. Those parallel views get combined into a single representation that's richer than any one attention pattern alone.

Why it mattered beyond translation

The original paper was framed around machine translation, which was the benchmark task at the time. But the architecture generalized far beyond it. Because self-attention doesn't care what the sequence "means," the same basic mechanism turned out to work for modeling code, protein sequences, and (most consequentially) for the next-word prediction task that underlies modern language models. The scaling behavior discovered in the years after this paper (bigger transformer, more data, more compute, reliably better results) is the direct basis for the current wave of large language models; our explainer on mixture-of-experts models covers one of the more significant architectural refinements built on top of that base.

What it didn't predict

It's worth being honest about what the paper didn't anticipate: nothing in it forecasts chatbots, few-shot prompting, or the emergent capabilities that showed up once transformers were scaled to billions and then hundreds of billions of parameters. That connection (between "efficient sequence architecture" and "system you can have a conversation with") took several more years of scaling research to establish. If you want the fuller picture of how a raw transformer becomes something like GPT-4o or Claude, our explainer on what a large language model actually is picks up where this paper leaves off.

The legacy

"Attention Is All You Need" is a systems and efficiency paper wearing the clothes of an architecture paper. Its real contribution wasn't a clever new way to model language. It was removing the sequential bottleneck that had capped how much data and compute you could usefully throw at the problem. Once that cap came off, scale did the rest.