Paper: arXiv

.09685 Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, et al. (Microsoft)

Overview

LoRA (Low-Rank Adaptation) revolutionized how we fine-tune large language models. Instead of updating all model parameters, LoRA freezes the pretrained weights and injects trainable low-rank decomposition matrices into each layer, reducing trainable parameters by 10,000x while maintaining model quality.

Key Contributions

  1. Parameter Efficiency: Reduces trainable parameters dramatically (e.g., from 175B to 17M for GPT-3)
  2. No Inference Latency: Merged weights can be used without additional computation
  3. Task Switching: Multiple LoRA adapters can be swapped efficiently
  4. Memory Efficiency: Significantly reduces GPU memory requirements for fine-tuning

How It Works

For a pretrained weight matrix W₀:

  • Instead of updating W₀ directly, add a low-rank decomposition: W = W₀ + BA
  • B and A are much smaller matrices (rank r < d)
  • Only B and A are trained; W₀ remains frozen

Impact

LoRA has become the standard for efficient fine-tuning:

  • Enables fine-tuning on consumer GPUs
  • Powers most open-source model adaptations
  • Spawned variants like QLoRA, DoRA, and more

论文: arXiv

.09685 作者: Edward J. Hu, Yelong Shen, Phillip Wallis 等 (Microsoft)

概述

LoRA（低秩适应）彻底改变了大型语言模型的微调方式。LoRA不更新所有模型参数，而是冻结预训练权重，在每层注入可训练的低秩分解矩阵，将可训练参数减少10000倍，同时保持模型质量。

主要贡献

  1. 参数效率：大幅减少可训练参数（如GPT-3从1750亿减少到1700万）
  2. 无推理延迟：合并后的权重可直接使用，无需额外计算
  3. 任务切换：多个LoRA适配器可高效切换
  4. 内存效率：显著降低微调所需的GPU内存

工作原理

对于预训练权重矩阵W₀：

  • 不直接更新W₀，而是添加低秩分解：W = W₀ + BA
  • B和A是更小的矩阵（秩 r < d）
  • 只训练B和A；W₀保持冻结

影响

LoRA已成为高效微调的标准：

  • 使消费级GPU上的微调成为可能
  • 支撑大多数开源模型适配
  • 衍生出QLoRA、DoRA等变体