Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-Tuning

Paper: 2602.04998 Authors: Yu-Ang Lee, Ching-Yun Ko, Pin-Yu Chen, Mi-Yen Yeh Categories: cs.LG, cs.AI, cs.CL

Abstract

A systematic re-evaluation of LoRA variants through comprehensive hyperparameter optimization reveals that once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%) across mathematical and code generation tasks at various model scales. Reported improvements in LoRA variants may reflect inadequate hyperparameter tuning rather than genuine methodological advantages.

Key Contributions

• Debunking LoRA variants: Most reported improvements disappear with proper learning rate tuning
• Second-order analysis: Links performance differences to variations in the largest Hessian eigenvalue
• Practical guidance: Proper hyperparameter search matters more than adapter architecture

The Uncomfortable Truth

The LoRA ecosystem has produced many variants — DoRA, rsLoRA, AdaLoRA, PiSSA, and others — each claiming improvements over vanilla LoRA. This paper challenges those claims by showing that the improvements largely disappear when you properly tune the learning rate.

The key insight: different LoRA variants have different optimal learning rate ranges. When papers compare methods using the same learning rate, they’re not comparing the methods fairly — they’re comparing how well each method happens to work at that particular learning rate.

Methodology

The authors conducted comprehensive hyperparameter sweeps across:

• Multiple model scales
• Mathematical reasoning and code generation tasks
• All major LoRA variants

Second-Order Analysis

The paper connects the learning rate sensitivity to the largest Hessian eigenvalue — a measure of the loss landscape’s curvature. Different LoRA variants create different loss landscapes, requiring different learning rates to navigate effectively.

Practical Implications

1. Don’t chase variants: Invest time in learning rate tuning instead of switching LoRA architectures
2. Fair comparisons require sweeps: Any LoRA comparison without proper hyperparameter optimization is suspect
3. Vanilla LoRA is a strong baseline: With proper tuning, it matches or comes within 1-2% of all variants

Takeaways

1. Hyperparameters > architecture: For LoRA fine-tuning, learning rate selection matters more than adapter design
2. Occam’s razor applies: The simplest approach (vanilla LoRA) works when properly configured
3. Beware of confounded comparisons: Many reported improvements in the literature may be artifacts of suboptimal hyperparameter tuning