CORAL: Correspondence Alignment for Improved Virtual Try-On

Paper: 2602.17636 Authors: Jiyoung Kim, Youngjin Shin, Siyoon Jin, Dahyun Chung, Jisu Nam, Tongmin Kim, Jongjae Park, Hyeonwoo Kang, Seungryong Kim Categories: cs.CV

Abstract

Virtual Try-On (VTON) systems face significant challenges in preserving fine garment details, particularly in unpaired settings where establishing accurate person-garment correspondence is critical. This paper introduces CORAL (CORrespondence ALignment), a novel DiT-based framework that addresses these limitations through explicit correspondence alignment. The authors first conduct a systematic analysis of full 3D attention mechanisms in Diffusion Transformers, revealing that person-garment correspondence fundamentally depends on precise query-key matching within the attention layers. CORAL leverages this insight by integrating two complementary loss functions: a correspondence distillation loss that aligns external robust matches with attention patterns, and an entropy minimization loss that sharpens attention distributions. The framework also introduces a VLM-based evaluation protocol to better capture human preferences in try-on quality assessment.

Key Contributions

• Systematic analysis of full 3D attention in DiT architectures, revealing the critical role of query-key matching in establishing person-garment correspondence
• CORAL framework that explicitly enforces correspondence alignment through two complementary loss components
• Correspondence distillation loss that transfers knowledge from robust external correspondence models to the attention mechanism
• Entropy minimization loss that sharpens attention distributions for more precise alignment
• VLM-based evaluation protocol that better reflects human preference compared to traditional metrics
• Comprehensive ablation studies validating each design choice and demonstrating consistent improvements in both global shape transfer and local detail preservation

Methodology and Technical Approach

The CORAL framework builds upon Diffusion Transformer architectures, specifically targeting the full 3D attention mechanism that processes concatenated person and garment features. The key insight is that effective virtual try-on requires the attention mechanism to implicitly learn correct correspondences between person body parts and garment regions.

The correspondence distillation loss $\mathcal{L}_{distill}$ operates by first obtaining robust correspondences from external models (such as optical flow or feature matching networks), then encouraging the attention weights to align with these correspondences. Formally, given attention weights $A \in \mathbb{R}^{N_p \times N_g}$ between person tokens and garment tokens, and external correspondence map $C$, the loss minimizes the divergence between $A$ and $C$.

The entropy minimization loss $\mathcal{L}_{entropy}$ complements this by encouraging sharper, more confident attention distributions. This prevents the attention from being diffuse across multiple garment regions for a single person location, which would result in blurred or mixed details. The entropy term is computed as:

$\mathcal{L}_{entropy} = -\sum_{i,j} A_{ij} \log A_{ij}$

where lower entropy indicates more concentrated attention patterns.

The total training objective combines these losses with the standard diffusion loss:

$\mathcal{L}_{total} = \mathcal{L}_{diffusion} + \lambda_1 \mathcal{L}_{distill} + \lambda_2 \mathcal{L}_{entropy}$

Experimental Results and Analysis

CORAL demonstrates consistent improvements over baseline DiT-based VTON methods across multiple benchmarks. The experiments evaluate both paired and unpaired settings, with unpaired scenarios being particularly challenging due to the lack of ground-truth person-garment alignment.

Quantitative results show improvements in standard metrics including FID, LPIPS, and SSIM. More importantly, the proposed VLM-based evaluation reveals that CORAL better aligns with human preferences, particularly in preserving intricate garment patterns, textures, and structural details like collars, sleeves, and hemlines.

Ablation studies systematically validate each component. Removing the correspondence distillation loss results in misaligned garment placement and distorted patterns. Without entropy minimization, the model produces blurrier results with mixed details from multiple garment regions. The combination of both losses yields the best performance, demonstrating their complementary nature.

Qualitative comparisons highlight CORAL’s superiority in challenging scenarios: complex patterns (stripes, florals, text), loose-fitting garments, and cases with significant pose differences between person and garment images. The attention visualization confirms that CORAL learns more precise correspondences, with attention weights concentrated on semantically correct regions.

Implications and Future Directions

CORAL’s explicit correspondence alignment approach represents a significant advancement in understanding and improving DiT-based generative models for structured tasks. The framework’s success suggests several broader implications:

First, the analysis of attention mechanisms in DiTs provides valuable insights for other correspondence-dependent tasks beyond virtual try-on, such as image editing, style transfer, and multi-modal generation. The finding that query-key matching drives correspondence learning could inform architecture design for these applications.

Second, the correspondence distillation paradigm offers a general strategy for injecting structural priors into diffusion models without requiring architectural modifications. This approach could be extended to incorporate other types of geometric or semantic constraints.

Third, the VLM-based evaluation protocol addresses a critical gap in generative model assessment. Traditional metrics often fail to capture perceptual quality and alignment with human preferences, particularly for detail-oriented tasks. This evaluation approach could be adopted more broadly in the generative modeling community.

Future work could explore adaptive weighting schemes for the loss components, investigate alternative correspondence sources (such as 3D body models or semantic segmentation), and extend the framework to video-based virtual try-on where temporal consistency becomes crucial.

Takeaways

1. Person-garment correspondence in DiT-based VTON critically depends on precise query-key matching within full 3D attention mechanisms
2. Explicit correspondence alignment through distillation and entropy minimization significantly improves detail preservation compared to implicit learning
3. The correspondence distillation loss effectively transfers knowledge from robust external correspondence models to the attention mechanism
4. Entropy minimization sharpens attention distributions, preventing detail mixing and improving local fidelity
5. VLM-based evaluation protocols better capture human preferences than traditional metrics for perceptual quality assessment
6. The framework achieves consistent improvements in both global shape transfer and fine-grained detail preservation across diverse garment types and poses
7. Ablation studies confirm the complementary nature of the two loss components, with both necessary for optimal performance