SOTAlign: Semi-Supervised Alignment of Vision and Language Models via Optimal Transport

Paper: 2602.23353 Authors: Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot, Zeynep Akata Categories: cs.LG, cs.AI

Abstract

This paper introduces SOTAlign, a semi-supervised framework for aligning pretrained vision and language models with minimal supervision. Building on the Platonic Representation Hypothesis—which suggests neural networks trained on different modalities converge toward shared representations—the authors propose a two-stage approach that uses small amounts of paired image-text data alongside large quantities of unpaired samples. The method first establishes a coarse alignment using a linear teacher on limited paired data, then refines this alignment through an optimal-transport-based divergence that preserves relational structure in the unpaired data. SOTAlign demonstrates substantial improvements over supervised and semi-supervised baselines across multiple datasets and encoder combinations.

Key Contributions

• Introduction of a semi-supervised alignment setting that dramatically reduces the need for paired image-text data while maintaining or improving alignment quality
• A two-stage framework combining linear teacher initialization with optimal transport refinement for robust cross-modal alignment
• Novel application of optimal transport divergence to transfer relational structure from unpaired data without overconstraining the embedding space
• Comprehensive experimental validation showing significant performance gains over existing supervised and semi-supervised methods across diverse encoder pairs and datasets

Methodology: Two-Stage Alignment Framework

SOTAlign operates through two distinct phases designed to maximize the utility of both paired and unpaired data.

Stage 1: Linear Teacher Initialization

The first stage establishes a coarse alignment using the limited paired samples available. Given pretrained vision encoder $f_v$ and language encoder $f_l$, the method learns a lightweight linear projection layer that maps between the two embedding spaces. This linear teacher provides an initial geometric correspondence that serves as the foundation for subsequent refinement. The simplicity of the linear mapping ensures that the method can learn meaningful structure even from very small paired datasets (e.g., 1-10% of typical training data).

Stage 2: Optimal Transport Refinement

The second stage leverages unpaired images and text to refine the alignment. Rather than treating unpaired samples as independent points, SOTAlign uses optimal transport to capture and transfer the relational structure within each modality. The key insight is that while individual samples may not have cross-modal correspondences, the geometric relationships between samples within a modality (e.g., semantic similarity between images) should be preserved in the aligned space.

The optimal transport formulation minimizes a divergence measure:

$\mathcal{L}_{OT} = W_2(\mu_v, \mu_l)$

where $W_2$ denotes the 2-Wasserstein distance between the distributions of vision embeddings $\mu_v$ and language embeddings $\mu_l$ after projection. This approach allows the model to learn from the rich structural information in unpaired data without requiring explicit correspondences, effectively transferring intra-modal relationships to the joint embedding space.

Experimental Results and Analysis

The authors conduct extensive experiments across multiple benchmark datasets and encoder combinations, demonstrating SOTAlign’s effectiveness in low-supervision regimes.

Performance with Limited Paired Data

When trained with only 1-10% of paired samples, SOTAlign consistently outperforms fully supervised baselines trained on 100% of paired data. This remarkable result suggests that the method successfully exploits the abundant unpaired data to compensate for limited supervision. The performance gap widens as the amount of paired data decreases, highlighting the framework’s robustness in extreme low-resource scenarios.

Comparison with Semi-Supervised Baselines

SOTAlign significantly outperforms existing semi-supervised alignment methods that rely on pseudo-labeling or consistency regularization. Unlike these approaches, which often struggle to effectively utilize unpaired data from both modalities simultaneously, SOTAlign’s optimal transport formulation naturally handles asymmetric unpaired data and preserves the distinct geometric properties of each modality.

Cross-Dataset and Cross-Encoder Generalization

The learned alignments demonstrate strong generalization across different datasets and encoder architectures. When aligned encoders are evaluated on downstream tasks like zero-shot classification and cross-modal retrieval, SOTAlign maintains competitive performance even when the test distribution differs from training. This robustness suggests that the method learns fundamental cross-modal correspondences rather than dataset-specific artifacts.

Implications and Future Directions

SOTAlign addresses a critical bottleneck in multimodal learning: the requirement for massive paired datasets. By demonstrating that effective alignment can be achieved with minimal supervision, this work opens several promising research directions.

Practical Impact: The reduced data requirements make cross-modal alignment accessible for domains where paired data is expensive or difficult to obtain, such as medical imaging with clinical reports, scientific figures with captions, or specialized technical domains.

Theoretical Insights: The success of optimal transport in this context provides empirical support for the Platonic Representation Hypothesis, suggesting that pretrained models do indeed converge toward compatible geometric structures that can be aligned with minimal supervision.

Scalability: The two-stage design allows for efficient training, as the computationally intensive optimal transport calculations operate on batches of unpaired data rather than requiring exhaustive pairwise comparisons. This makes the approach practical for large-scale applications.

Future work could explore extending SOTAlign to more than two modalities, investigating different optimal transport formulations, or combining the approach with self-supervised pretraining objectives to further reduce supervision requirements.

Takeaways

1. Meaningful vision-language alignment can be achieved with dramatically less paired data than previously thought necessary—as little as 1-10% of typical training sets
2. Optimal transport provides an effective mechanism for transferring relational structure from unpaired data, preserving intra-modal geometry while learning cross-modal correspondences
3. The two-stage framework (linear teacher + OT refinement) balances computational efficiency with alignment quality, making semi-supervised alignment practical at scale
4. SOTAlign’s strong generalization across datasets and encoder architectures suggests it learns fundamental cross-modal relationships rather than superficial correlations
5. This work validates the Platonic Representation Hypothesis in practice, demonstrating that pretrained models from different modalities possess compatible geometric structures that can be aligned with minimal supervision