What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

Paper: What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal Authors: Stephen Cheng, Sarah Wiegreffe, Dinesh Manocha Categories: cs.LG

The Gap

Steering vectors have become a popular lightweight technique for aligning LLMs—you add a vector to the model’s hidden states at inference time to encourage or discourage specific behaviors like refusal, hallucination, or persona. They work remarkably well in practice, achieving competitive performance against fine-tuning and prompting. But we have no mechanistic understanding of how they work. Which internal circuits do they activate? How do they interact with attention and MLPs? Without this understanding, we can’t assess robustness, diagnose failures, or design better steering interventions.

This paper conducts the first comprehensive mechanistic analysis of steering vectors, focusing on refusal steering—a critical safety capability where steering has proven highly effective at both inducing and bypassing refusal responses.

[ Steering Vectors ] -> [ Add to Hidden States ] -> [ Behavior Changes ]
         |                        |                          |
         v                        v                          v
[ Problem: Black Box ] -> [ Unknown: Which    ] -> [ Method: Activation ]
[  Mechanism Unknown  ]    [ Circuits Matter?  ]    [ Patching Framework ]
         |                        |                          |
         v                        v                          v
[ Evidence: Refusal  ] <--- [ ~10% of Edges    ] <--- [ Result: OV Circuit ]
[  Steering Works    ]      [ Recover Behavior ]      [ Dominates Effect   ]

The Increment

One sentence: Before this paper, steering vectors were effective but opaque interventions; after this paper, we know they operate through highly localized circuits (~10% of model edges), interact primarily with attention’s OV circuit while largely ignoring the QK circuit, and can be sparsified by 90-99% while retaining performance.

Core Mechanism

The authors extend traditional mechanistic interpretability techniques (typically applied only to standard inference) to steered inference runs. They propose a multi-token activation patching framework that identifies which model components are causally responsible for propagating steering effects across generated sequences.

The key findings:

1. Circuit Localization: Only ~10% of model edges are needed to recover 85% of steering behavior. Different steering methods (Difference-in-Means, Next Token Prediction, Preference Optimization) leverage functionally interchangeable circuits with >90% overlap.

2. OV Circuit Dominance: Steering vectors primarily interact with attention through the OV (output-value) circuit. Freezing all attention scores (the QK circuit) drops performance by only ~8.75%, while freezing the OV circuit drops it by ~72%.

3. Steering Value Vector (svv) Decomposition: The authors mathematically decompose the direct effect of steering on attention heads, revealing a “steering value vector” for each head. Using logit lens on these svvs surfaces semantically interpretable tokens related to refusal and harmfulness—even when the raw steering vector itself is not interpretable.

4. Extreme Sparsity: Steering vectors can be sparsified to 90-99% zeros while mostly retaining performance. Different steering methods converge on a shared low-dimensional subspace of important dimensions.

Steering Vector s
       |
       v
[ Layer ℓ ] --> [ Attention ] --> [ OV Circuit ] --> svv_h = (s ⊙ γ)W_OV^h
       |              |                                      |
       |              v                                      v
       |        [ QK Circuit ]                    [ Logit Lens Reveals ]
       |        (Frozen: -8.75%)                  [ "forbidden", "illegal" ]
       |                                          [ "dangerous", "unsafe" ]
       v
    [ MLP ]
(Direct Effect: -44.5%)

Think of steering like adjusting a sound equalizer. The steering vector is like boosting certain frequency bands. The OV circuit acts as the main amplifier that propagates these adjustments through the audio chain, while the QK circuit (which determines attention patterns) is like the speaker placement—it matters, but the core signal modification happens elsewhere. The svv decomposition is like analyzing which specific frequencies got boosted, revealing interpretable concepts even when the raw equalizer settings look like random numbers.

Key Concepts

• Activation Patching: A causal intervention technique that replaces activations from a “corrupted” run with those from a “clean” run to measure which components matter. The authors adapt this to compare steered vs. unsteered generations.

• EAP-IG (Edge Attribution Patching with Integrated Gradients): An efficient approximation method that computes importance scores for edges in the model’s computational graph by taking gradients with linearly increasing steering coefficients.

• Circuit Interchangeability: High overlap doesn’t just mean circuits share edges—it means a circuit discovered for one steering method faithfully recovers behavior when using a different steering method’s vector.

• Gradient-Based Sparsification: Zeroing out dimensions where the ratio of importance score to steering coefficient is below a threshold, effectively keeping only dimensions with high normalized contributions.

Framework Shift

Before (Steering as Black Box):        After (Mechanistic Understanding):
[ Steering Vector ]                    [ Steering Vector ]
        |                                      |
   [Add to h_ℓ]                          [Decompose Effect]
        |                                      |
   [??? Magic ???]                        [~10% Circuit]
        |                                 /          \
   [Behavior Change]              [OV: 72%]    [QK: 8.75%]
                                       |              |
                                   [svv_h]      [Frozen Scores]
                                       |
                                [Interpretable]
                                ["forbidden"]

One sentence: From treating steering as an empirical trick to understanding it as a targeted intervention on specific attention circuits, with the ability to decompose, interpret, and sparsify the mechanism.

Expert Assessment

Problem choice: Excellent. Steering vectors are increasingly used in production systems, but their opacity is a genuine safety concern. Focusing on refusal steering is smart—it’s both safety-critical and has clear success metrics (Attack Success Rate).

Method maturity: The multi-token activation patching framework is a solid extension of existing techniques. The svv decomposition is mathematically clean and provides a reusable tool for future work. The experimental design is thorough—testing multiple steering methods, two model families (Gemma 2 2B, Llama 3.2 3B), and multiple datasets.

Experimental integrity: Strong. The circuit faithfulness metrics are well-defined, the interchangeability tests are convincing (random circuits fail as expected), and the sparsification results are validated across unseen benchmarks. The statistical significance testing on IoU (p<1e-10) is rigorous.

Writing quality: Clear and well-structured. The paper builds from circuit discovery to attention analysis to sparsification in a logical progression. The visualizations (especially Figure 1 showing frozen QK generations) effectively communicate the findings.

Limitations acknowledged: The authors are honest about not deeply analyzing MLPs (which do appear in circuits), only evaluating one steering layer per model, and focusing solely on refusal (though they justify this choice well).

Verdict: Strong Accept — This is foundational work for mechanistic interpretability of steering vectors. The tools (multi-token patching, svv decomposition, gradient-based sparsification) will be widely reused, and the findings have immediate practical implications for making steering more robust and efficient.

Takeaways

Practitioners can immediately apply gradient-based sparsification to compress steering vectors by 90%+ without significant performance loss. This reduces memory overhead and potentially improves robustness by removing noisy dimensions.

The OV circuit dominance finding suggests that future steering methods should focus on how they interact with value projections rather than trying to reshape attention patterns. This could inform more targeted interventions.

The svv decomposition provides a debugging tool: if your steering vector isn’t working, project its effect through individual attention heads to see which concepts are being expressed (or suppressed). This can reveal inefficiencies where some heads work against others.

More broadly, the finding that different steering methods converge to similar circuits and sparse subspaces suggests there may be a “natural” low-dimensional representation of concepts like refusal in LLMs—a promising direction for more principled alignment techniques.