CoPE: Clipped RoPE as a Free Lunch for Long Context LLMs

Paper: 2602.05258 Authors: Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang Categories: cs.CL, cs.AI, cs.LG

Abstract

CoPE applies soft clipping to low-frequency components of Rotary Positional Embedding (RoPE), addressing out-of-distribution issues and improving semantic signal quality. The method prevents spectral leakage caused by hard clipping and demonstrates performance improvements scaling to 256k context length, establishing a new state-of-the-art for length generalization.

Key Contributions

• Soft clipping for RoPE: A simple modification to low-frequency components that eliminates outliers
• Spectral leakage prevention: Avoids artifacts from hard clipping approaches
• 256k context generalization: New state-of-the-art for length generalization
• Free lunch: No additional training cost or architectural changes required

Background: RoPE and Long Context

Rotary Positional Embedding (RoPE) encodes position information by rotating query and key vectors at frequencies determined by their position. Different frequency components capture different scales of positional information:

• High-frequency components: Capture local, fine-grained positional relationships
• Low-frequency components: Capture long-range, global positional relationships

When extending to longer contexts than seen during training, the low-frequency components encounter out-of-distribution values, causing attention degradation.

The CoPE Approach

CoPE applies soft clipping to the low-frequency components of RoPE. This achieves two goals:

1. Mitigates OOD issues: Prevents low-frequency components from taking extreme values at unseen positions
2. Improves semantic signals: Eliminates outliers that corrupt attention patterns

Why Soft Clipping?

Previous approaches used hard clipping (e.g., NTK-aware scaling, YaRN), but hard clipping introduces spectral leakage — discontinuities that create artifacts in the frequency domain. Soft clipping provides a smooth transition that preserves signal quality.

Results

• State-of-the-art length generalization up to 256k context length
• No additional training cost — works as a drop-in modification
• Compatible with existing RoPE-based models
• Code, data, and models publicly available

Why This Matters

1. Free lunch: Performance improvement with zero additional compute cost
2. Principled approach: Grounded in signal processing theory (spectral leakage prevention)
3. Practical deployment: Drop-in replacement for existing RoPE implementations
4. Scaling enabler: Unlocks longer context windows without retraining