Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

Paper: 2602.17645 Authors: Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen Categories: cs.LG, cs.AI, cs.CL, cs.CV

Abstract

This paper addresses the challenge of black-box adversarial attacks on Large Vision-Language Models (LVLMs), where gradient information is unavailable and multimodal decision boundaries are complex. The authors identify critical limitations in existing transfer-based approaches like M-Attack: local crop-level matching produces high-variance, nearly orthogonal gradients across iterations due to Vision Transformer (ViT) translation sensitivity and structural asymmetry between source and target crops. To address this, they propose M-Attack-V2, which reformulates local matching as an asymmetric expectation framework and introduces gradient-denoising mechanisms. The approach achieves substantial improvements in attack success rates across frontier LVLMs, including Claude-4.0 (8% → 30%), Gemini-2.5-Pro (83% → 97%), and GPT-5 (98% → 100%).

Key Contributions

• Problem Identification: Discovered that existing local crop-level matching in M-Attack generates high-variance, nearly orthogonal gradients that destabilize optimization, attributed to ViT translation sensitivity and structural asymmetry
• Multi-Crop Alignment (MCA): Reduces gradient variance on the source side by averaging gradients from multiple independently sampled local views per iteration
• Auxiliary Target Alignment (ATA): Replaces aggressive target augmentation with a semantically correlated auxiliary set, producing a smoother, lower-variance target manifold
• Patch Momentum: Reinterprets momentum as a mechanism for replaying historical crop gradients, combined with refined patch-size ensemble (PE+) to strengthen transferable attack directions
• State-of-the-Art Performance: Achieves significant improvements over prior black-box LVLM attacks across multiple frontier models

Technical Methodology

The core innovation of M-Attack-V2 lies in its gradient-denoising framework that addresses the instability of local crop-level matching. The authors reformulate the optimization problem as:

$\mathbb{E}_{T_s \sim \mathcal{T}_s, T_t \sim \mathcal{T}_t}[\mathcal{L}(f(T_s(x + \delta)), f(T_t(x_{target})))]$

where $T_s$ and $T_t$ represent source and target transformations respectively.

Multi-Crop Alignment (MCA) samples $K$ independent local crops from the source image per iteration and averages their gradients:

$g_{MCA} = \frac{1}{K}\sum_{i=1}^{K} \nabla_\delta \mathcal{L}(f(crop_i(x + \delta)), f(x_{target}))$

This variance reduction technique ensures more coherent gradient directions across iterations, addressing the spike-like gradients caused by ViT’s translation sensitivity.

Auxiliary Target Alignment (ATA) constructs a small auxiliary target set $\mathcal{X}_{aux}$ from a semantically correlated distribution rather than aggressive augmentations of a single target. This produces a smoother target manifold:

$g_{ATA} = \mathbb{E}_{x_{aux} \sim \mathcal{X}_{aux}}[\nabla_\delta \mathcal{L}(f(x + \delta), f(x_{aux}))]$

Patch Momentum extends traditional momentum by maintaining a history of crop-specific gradients, enabling the replay of successful local attack patterns. Combined with PE+, which ensembles multiple patch sizes, this mechanism identifies and amplifies transferable adversarial directions that generalize across different LVLM architectures.

Experimental Results and Analysis

M-Attack-V2 demonstrates substantial improvements across multiple frontier LVLMs in black-box settings. The results reveal several important patterns:

Performance on Frontier Models: The attack achieves dramatic improvements on Claude-4.0 (from 8% to 30% success rate), representing a 3.75× improvement. On Gemini-2.5-Pro, success rates increase from 83% to 97%, while GPT-5 reaches near-perfect 100% from 98%. These results indicate that the gradient-denoising approach effectively addresses the transferability gap in black-box scenarios.

Ablation Studies: Individual component analysis shows that MCA contributes the most significant variance reduction, while ATA provides complementary benefits by smoothing the target manifold. Patch Momentum and PE+ together enhance the stability of transferable directions. The modular design allows practitioners to selectively apply components based on computational constraints.

Computational Efficiency: Despite sampling multiple crops per iteration, M-Attack-V2 maintains reasonable computational overhead through efficient batching and parallel processing. The auxiliary target set in ATA is kept small (typically 5-10 images) to balance effectiveness and efficiency.

Cross-Model Transferability: The improved gradient coherence leads to better transferability across different LVLM architectures, suggesting that the approach captures more fundamental vulnerabilities in multimodal alignment rather than model-specific artifacts.

Security Implications and Future Directions

The success of M-Attack-V2 highlights critical security vulnerabilities in current LVLM deployments. The ability to achieve high attack success rates in black-box settings—where attackers have no access to model internals—demonstrates that even frontier commercial models remain susceptible to carefully crafted adversarial perturbations.

Defense Considerations: The gradient-denoising perspective suggests potential defense mechanisms. Randomized cropping and multi-scale processing during inference could disrupt the coherent local alignment that M-Attack-V2 exploits. Adversarial training with diverse crop-level perturbations may improve robustness.

Broader Impact: This work underscores the need for robust evaluation of LVLMs before deployment in security-critical applications. The modular framework provides a standardized benchmark for measuring LVLM robustness and tracking progress in adversarial defenses.

Future Research: Promising directions include extending the gradient-denoising framework to other modalities (audio, video), investigating adaptive defenses that detect multi-crop attack patterns, and developing certified robustness guarantees for multimodal models.

Takeaways

1. Local crop-level matching in transfer-based attacks suffers from high-variance, orthogonal gradients due to ViT translation sensitivity and structural asymmetry
2. Multi-Crop Alignment (MCA) and Auxiliary Target Alignment (ATA) provide complementary gradient-denoising mechanisms that stabilize optimization
3. Patch Momentum combined with refined patch-size ensemble (PE+) strengthens transferable adversarial directions across LVLM architectures
4. M-Attack-V2 achieves state-of-the-art black-box attack performance on frontier LVLMs, with success rates reaching 30% on Claude-4.0, 97% on Gemini-2.5-Pro, and 100% on GPT-5
5. The modular design enables flexible deployment based on computational constraints and target model characteristics
6. Results highlight critical security vulnerabilities in current LVLM deployments and emphasize the need for robust evaluation before production use