Benign Overfitting in Adversarial Training for Vision Transformers

Paper: 2604.19724 Authors: Jiaming Zhang, Meng Ding, Shaopeng Fu, Jingfeng Zhang, Di Wang Categories: cs.LG, cs.CV, cs.AI

Problem

Despite Vision Transformers’ success, they remain vulnerable to adversarial examples like CNNs. Theoretical underpinnings of adversarial training’s robustness in ViTs remain largely unexplored.

Key Contribution

First theoretical analysis of adversarial training under simplified ViT architectures, showing that:

• When trained under a signal-to-noise ratio satisfying certain conditions
• Within a moderate perturbation budget
• ViTs can achieve nearly zero robust training loss and robust generalization error under certain regimes

Key Result: Benign Overfitting in ViTs

This leads to strong generalization even in the presence of overfitting - a phenomenon known as benign overfitting, previously only observed in CNNs with adversarial training.

Conditions for Benign Overfitting

1. Signal-to-noise ratio condition: Specific threshold for stability
2. Perturbation budget: Moderate epsilon within which theory holds
3. Architecture constraints: Simplified ViT structure assumptions

Experiments

Experiments on both synthetic and real-world datasets validate theoretical findings.

Takeaways

• ViTs share the benign overfitting property with CNNs under adversarial training
• Theoretical understanding enables principled design of adversarial training for ViTs
• Overfitting ≠ poor generalization in adversarial settings