Hung-yi Lee ML 2025 Lecture 3: The Neuroscience of AI - Inside Language Model Mechanisms

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

The Neuroscience of AI

This lecture explores how transformers actually work internally - not how to train them, but what happens inside an already-trained model. Think of it as neuroscience for AI.

Important caveat: Most interpretability research is done on smaller, older models (like GPT-2) because analyzing massive models requires enormous compute. Results may not fully generalize to frontier models.

What Does a Single Neuron Do?

In a transformer’s feed-forward layer, each output dimension is computed by taking a weighted sum of inputs, passing through an activation function (ReLU). Each such computation is called a “neuron.”

How to analyze a neuron’s function:

1. Observe: When does this neuron activate? What outputs follow?
2. Intervene: Remove the neuron - does the behavior disappear?
3. Modulate: Does stronger activation produce stronger effects?

The Trump Neuron: In 2021, OpenAI researchers found a neuron in CLIP (an image model) that activates specifically for Trump-related content - his face, cartoons of him, even text containing his name. It’s highly selective: activates for Trump and his associates, but not for Obama or other politicians.

The Grandmother Neuron Problem: In neuroscience, the “grandmother neuron” hypothesis (one neuron = one memory) was proposed as a strawman to contrast with the more accepted view that memories involve many neurons working together. Similarly in AI, single neurons rarely have clean, interpretable functions.

Most neurons are polysemantic - they respond to multiple unrelated concepts. One neuron might activate for physics terms, medical terminology, AND counterfeit-related words. When asked to explain what a random neuron does, even GPT-4.5 gives vague, unhelpful answers.

From Single Neurons to Functional Vectors

If single neurons don’t have clear functions, maybe combinations of neurons do. The hypothesis: specific patterns of neuron activations (vectors) correspond to specific functions.

Finding functional vectors:

1. Collect many inputs that trigger a behavior (e.g., refusal)
2. Extract the representation at a specific layer for each
3. Average them to get “refusal + other stuff”
4. Do the same for inputs that don’t trigger the behavior
5. Subtract to isolate the functional vector

The Refusal Vector: Researchers found a vector that, when added to representations, makes models refuse even harmless requests (“Yoga is dangerous, I can’t help”). When subtracted, models comply with harmful requests they’d normally refuse.

Other discovered vectors:

• Sycophancy vector: Add it, and the model agrees with everything (“Your idea is the greatest in human history!”)
• Truthfulness vector: Add it, and the model gives literal answers instead of playing along with idioms
• In-context learning vector: Extracted from demonstrations, it can make models follow patterns without seeing examples

These vectors can even be combined arithmetically: (first-word + capital) - (first-word + copy) + (last-word + copy) = (last-word + capital).

Sparse Autoencoders: Better Feature Discovery

The problem with raw neurons: they’re polysemantic. Solution: train a sparse autoencoder to decompose representations into interpretable features.

A sparse autoencoder learns to:

1. Expand the representation into a much higher-dimensional space
2. Force most dimensions to be zero (sparsity)
3. Reconstruct the original representation

The resulting features are often monosemantic - each corresponds to a single concept. Anthropic found features in Claude for:

• The Golden Gate Bridge
• Sycophantic praise
• Code errors
• Specific languages

Steering with features: Amplify the “Golden Gate Bridge” feature, and Claude becomes obsessed with it, mentioning it in every response. Amplify the sycophancy feature, and it calls your made-up proverb “the greatest sentence since humanity began.”

Logit Lens: Reading the Model’s Mind

Transformers have residual connections - each layer adds information to a “residual stream” that flows from input to output. The final layer converts this stream to token probabilities via “unembedding.”

Key insight: You can apply unembedding to intermediate layers too! This reveals what the model is “thinking” at each stage.

Example - Capital cities: Ask “What is the capital of Poland?” and trace the residual stream:

• Early layers: Confused, no clear token
• Layer 15: Suddenly “Poland” appears
• Layer 19: Switches to “Warsaw”

The model first identifies the subject, then retrieves the answer.

Language of thought: When LLaMA-2 translates French→Chinese, logit lens reveals it goes French→English→Chinese internally. English is its “language of thought.”

PatchScope: Richer Interpretations

Logit lens only outputs single tokens. PatchScope extracts richer descriptions by:

1. Creating a template: “Leonardo: American actor. TSMC: Taiwanese company. X:”
2. Replacing X’s representation with the one you want to analyze
3. The model completes with a description

This reveals how understanding evolves across layers. For “Diana, Princess of Wales”:

• Layers 1-2: “A country in the United Kingdom” (just sees “Wales”)
• Layer 4: “A title for royal women”
• Layer 5: “Wife of the Prince of Wales”
• Layer 6: Full information about Diana

Practical Application: Fixing Multi-Hop Reasoning

For questions like “The spouse of the performer of Imagine is ___”:

• E1 (explicit): Imagine (the album)
• E2 (implicit): John Lennon (performer)
• E3 (answer): Yoko Ono (spouse)

Analysis shows E2 must be resolved by ~layer 15 for E3 to be computed by ~layer 25. When E2 resolves too late, the model fails.

Solution (Backpatching): Take later-layer representations and inject them into earlier layers, giving the model “more time” to reason. This fixes 40-60% of previously wrong answers - similar in spirit to chain-of-thought reasoning (“when depth isn’t enough, use length”).

Key Takeaways

1. Single neurons rarely have clean, interpretable functions
2. Functional vectors (neuron combinations) can control specific behaviors
3. Sparse autoencoders find monosemantic features for precise steering
4. Logit lens and PatchScope reveal the model’s “thoughts” at each layer
5. Understanding internals enables targeted interventions and fixes