Notes: Transformers Explained - The Discovery That Changed AI Forever

This post summarizes Y Combinator’s explainer video on the history and development of the Transformer architecture that powers nearly every state-of-the-art AI system today.

Introduction

Nearly every state-of-the-art AI system - whether it’s ChatGPT, Claude, Gemini, or Grok - is built on the same underlying model architecture: the Transformer. But where did the Transformer architecture come from? And what can its development teach us about the way breakthroughs in AI happen?

A Transformer is a neural network that uses self-attention to take input data (like text or images), model the relationships between that data, and generate outputs like meaningful text responses, translations, or classifications.

The original Transformer architecture was introduced in a now-famous 2017 paper from Google called “Attention Is All You Need.” But there were three key developments that made this breakthrough possible:

1. Long Short-Term Memory (LSTM)
2. Seq2seq with Attention
3. Transformers

The Challenge: Understanding Sequences

One of the core challenges motivating early AI research was getting neural networks to understand sequences. Natural language is inherently sequential - the meaning of a word depends on what comes before or after it, and understanding an entire sentence requires maintaining context across many words.

Feedforward Neural Networks: The Limitation

Early architectures like feedforward neural networks process each input in isolation. They weren’t capable of understanding context, or they required looking at inputs of a fixed length.

Recurrent Neural Networks (RNNs): A Solution

Researchers developed RNNs as a solution. In simple terms, an RNN iterates over inputs in order, one at a time, and consumes the previous outputs as additional input at each step. If an input is of length N, there are N feedforward pass steps.

The Vanishing Gradient Problem: During the backwards pass, the gradient with respect to early inputs is the result of N matrix multiplications. In practice, this meant we often faced vanishing gradients - early inputs in a sequence had less and less influence on the network’s output as the sequence grew longer.

Long Short-Term Memory (LSTM)

In the 1990s, Hochreiter and Schmidhuber proposed a solution: Long Short-Term Memory networks (LSTMs).

Key Innovation: LSTMs were a type of RNN that attempted to fix the vanishing gradient problem by introducing gates, which could learn what information to keep, update, or forget. This made it possible to learn long-range dependencies, something vanilla RNNs struggled with.

The Compute Problem: LSTMs were too expensive to train at scale in the ’90s, and progress stalled.

The Revival: Fast forward to the early 2010s - GPU acceleration, better optimization techniques, and new large-scale datasets brought LSTMs back into the spotlight. Suddenly, this relatively old architecture was viable again and began to dominate natural language processing.

LSTMs were quickly adopted for everything from speech recognition to language modeling. In these years, NLP and computer vision were somewhat separate worlds - RNNs and LSTMs were preeminent in language tasks, while CNNs were winning in vision.

The Fixed-Length Bottleneck

Despite their success, LSTMs still had limitations. The most fundamental was the fixed-length bottleneck.

For sequence-to-sequence tasks like translation:

1. Take the input sentence
2. Feed it into an encoder LSTM
3. Boil the input down to a single fixed-size vector
4. A decoder LSTM takes that vector and constructs the target sentence word by word

The Problem: That single vector was still unable to accurately capture the meaning of long or complex sentences. There wasn’t a great way to encode the concept of order into a fixed-size vector.

Example: In English, we put adjectives before nouns. In Spanish, adjectives often come after nouns. These models worked okay on short inputs but quickly fell apart as sequences got longer.

This pointed to a deeper architectural problem: allowing the decoder to only see one static summary of the input was a fundamental limitation. Why not give it access to all the intermediate information that the encoder saw?

Seq2seq with Attention

In 2014, a paper introduced what would become the new standard for sequence translation: sequence-to-sequence (seq2seq) models with attention.

The Core Idea

Like before, the core idea was to train two neural networks jointly:

• Encoder: Reads the input sequence and builds a representation of it
• Decoder: Generates the output sequence one step at a time

Both models were LSTMs, trained together end to end.

The Key Insight: Attention

Even though seq2seq used a fixed-length vector, researchers realized that if you could let the decoder look back (or attend to) the encoder’s hidden states, you could let the model learn how to align parts of the input to parts of the output.

Bahdanau, Cho, and Bengio showed that these models could significantly outperform traditional rule-based systems and existing seq2seq models on tasks like machine translation.

Real-World Impact

• These models showed near state-of-the-art performance, beating even the best statistical systems at the time
• It was a sign that neural models could compete head-to-head with mature production-grade systems
• Google Translate adopted a neural seq2seq architecture around this time - you may remember this as the era when Google Translate started to finally work well

Beyond NLP

This insight - learning to align and translate at the same time - was transformative, and it wouldn’t just stay in NLP. Yoshua Bengio soon applied similar alignment-based architectures to computer vision. This was the first sign that these sequence models might be useful beyond language.

Remaining Limitations

Even when augmented with attention, RNNs were still constrained by their sequential architecture:

• Processing tokens one at a time made it challenging to run computations in parallel
• Runtimes scaled linearly with sequence length
• Training models on large datasets became intractably slow

Researchers developed techniques like factorizing LSTM matrices or conditionally activating only parts of a network, but the fundamental linear runtime constraint remained.

The Transformer Breakthrough (2017)

Then came the big breakthrough in 2017, when a team of researchers at Google published “Attention Is All You Need,” proposing a new machine translation architecture called the Transformer.

The Revolutionary Change

Transformers scrapped recurrence entirely, instead relying solely on an attention mechanism to generate outputs.

How Transformers Work

At a high level, Transformers use a modified version of the encoder-decoder architecture originally proposed in seq2seq:

• Instead of compressing inputs into a single vector embedding, Transformers kept separate embeddings for each input token
• These embeddings are updated through self-attention - a mechanism that updates token representations based on a learned weighted dot product over the embeddings of all other tokens in the sequence

The Key Advantage: Parallelization

Because each token in this architecture could attend to all others simultaneously, Transformers could process an entire sequence in parallel, making them dramatically faster than RNNs.

Remarkably, they were also much more accurate on machine translation benchmarks.

Evolution of Transformer Architectures

Over the next few years, researchers experimented with different variations:

The Original Architecture

The architecture described in the original Google paper featured an encoder and decoder that each had self-attention and cross-attention between the two - resembling the original seq2seq architectures, but without the recurrence.

BERT: Encoder-Only

A series of models called BERT focused on using only the encoder to do masked language modeling.

GPT: Decoder-Only

Efforts to use only the decoder for autoregressive modeling gave rise to OpenAI’s GPT series of models.

Both model series can be described as subsets of the original “Attention Is All You Need” Transformer model.

Scaling Up

It quickly became clear that these models could scale to large numbers of parameters. Ultimately, one model type - the Generative Pre-trained Transformer (GPT) - would be scaled up to create the LLMs that we regularly use today in products like ChatGPT or Claude.

From Single-Task to General Intelligence

Not long ago, it wasn’t obvious that there might be one model to rule them all. People were training variants of model architectures for every task:

• One for machine translation
• Another for named entity recognition
• And so on

Each had a shared backbone but slight differences in the final model layers. These models were intelligent (high accuracy), but they were largely single-task models.

At this point, there wasn’t really a concept of prompting the models because there was no chat interface. Instead, people interacted with models through domain-specific inputs.

It was only as labs started to experiment with training autoregressive models on much larger datasets that they began to look and feel more like generally intelligent systems.

Conclusion

This history helps contextualize what it took to get these models to a place where we could scale them:

• From the vanishing gradient problem to LSTMs
• From the fixed-length bottleneck to attention mechanisms
• From sequential processing to parallel Transformers
• From single-task models to general-purpose LLMs

The next frontier involves the architectural and engineering innovations needed to push these models to even higher performance levels.