CORAL: Correctness-Optimized Residual Activation Lens for Inference-Time Steering

Paper: 2602.06022 Title: Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steering Categories: cs.CL, cs.LG

Abstract

Inference-time steering has emerged as a promising approach for controlling Large Language Model (LLM) behavior without expensive fine-tuning. CORAL (Correctness-Optimized Residual Activation Lens) introduces a novel framework that learns transferrable steering vectors optimized for correctness rather than mere activation matching, while maintaining calibration awareness to prevent overconfident or miscalibrated outputs.

Key Contributions

• Correctness-Optimized Training: Unlike traditional activation steering that matches target activations, CORAL directly optimizes for downstream task correctness
• Residual Activation Lens: A learned projection that identifies the most influential directions in the residual stream for steering
• Transferability: Steering vectors learned on one task transfer effectively to related tasks and even different model scales
• Calibration Awareness: Built-in mechanisms to maintain well-calibrated confidence estimates during steering

Background: Activation Steering

Traditional activation steering methods work by:

1. Collecting activations from contrastive prompts (e.g., helpful vs. harmful)
2. Computing a steering vector as the difference in mean activations
3. Adding this vector to the residual stream during inference

The steering intervention at layer $l$ can be written as:

$h_l' = h_l + \alpha \cdot v$

where $h_l$ is the original hidden state, $v$ is the steering vector, and $\alpha$ controls the steering strength.

The Problem with Existing Methods

Activation Matching vs. Correctness: Traditional methods optimize for activation similarity, not task performance. A steering vector that perfectly matches target activations may not improve actual model behavior.

Calibration Degradation: Aggressive steering can push the model into overconfident predictions, degrading calibration even when accuracy improves.

Limited Transferability: Steering vectors are often task-specific and fail to generalize across different prompts or model configurations.

CORAL Framework

Residual Activation Lens

CORAL learns a projection matrix $W$ that identifies the most influential subspace for steering:

$v_{coral} = W \cdot (a_{target} - a_{base})$

where $a_{target}$ and $a_{base}$ are activations from target and base behaviors. The lens $W$ is learned to maximize downstream correctness.

Correctness-Optimized Training

The training objective combines task performance with calibration:

$\mathcal{L} = \mathcal{L}_{task} + \lambda \cdot \mathcal{L}_{calibration}$

The task loss $\mathcal{L}_{task}$ measures performance on the target behavior, while $\mathcal{L}_{calibration}$ penalizes miscalibrated confidence estimates using Expected Calibration Error (ECE).

Calibration-Aware Steering

CORAL introduces adaptive steering strength based on model uncertainty:

$\alpha_{adaptive} = \alpha_0 \cdot \sigma(u)$

where $u$ is a learned uncertainty estimate and $\sigma$ is a sigmoid function. This prevents oversteering on inputs where the model is already confident.

Methodology

Training Pipeline

1. Data Collection: Gather contrastive examples representing target and base behaviors
2. Activation Extraction: Extract residual stream activations at multiple layers
3. Lens Learning: Train the projection matrix $W$ to maximize correctness
4. Calibration Tuning: Fine-tune steering strength for calibration preservation

Inference-Time Application

At inference, CORAL applies the learned steering:

$h_l' = h_l + \alpha_{adaptive} \cdot W \cdot \Delta a$

The intervention is applied at selected layers identified during training as most effective for the target behavior.

Results

CORAL demonstrates improvements across multiple steering tasks:

Task	Baseline	Activation Steering	CORAL
Truthfulness	62.3%	71.5%	78.2%
Helpfulness	74.1%	79.8%	84.6%
Safety	81.2%	86.4%	91.3%

Calibration Preservation

Method	Accuracy	ECE (lower is better)
Base Model	65.4%	0.082
Activation Steering	73.2%	0.156
CORAL	78.2%	0.091

CORAL maintains calibration while improving accuracy, unlike traditional steering which degrades calibration.

Transferability

Steering vectors learned on one task show strong transfer:

• Cross-prompt: 89% effectiveness on unseen prompt formats
• Cross-task: 72% effectiveness on related but distinct tasks
• Cross-scale: 81% effectiveness when transferred to larger models

Why This Matters

Efficient Alignment: CORAL enables behavior modification without fine-tuning, reducing computational costs by orders of magnitude.

Interpretable Control: The learned lens provides insights into which activation directions matter for specific behaviors.

Safe Deployment: Calibration awareness ensures that steered models maintain reliable uncertainty estimates, crucial for high-stakes applications.

Takeaways

1. Optimize for correctness, not activation matching: Direct optimization for task performance yields better steering vectors than activation similarity
2. Calibration matters: Steering methods must preserve calibration to be useful in practice
3. Transferability is achievable: Well-designed steering vectors can generalize across tasks and model scales
4. Residual stream is key: The residual stream provides a natural intervention point for behavior modification
5. Adaptive steering: Uncertainty-aware steering strength prevents overconfident predictions