UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

Paper: 2606.20559 Authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das Categories: cs.CV, cs.LG

The Gap

Egocentric video understanding is stuck in a narrow tunnel. Most prior work trains separate models for each data type: one model for RGB from the head-mounted camera, another for depth, another for skeleton poses. Some methods try to incorporate exocentric (third-person) videos as extra supervision, but they simply concatenate features or average losses. The result: a fragmented set of encoders, each blind to the complementary signals the others carry. Worse, when researchers attempt multi-teacher distillation—taking knowledge from several pre-trained “teacher” models and squeezing it into one “student”—they hit a wall: teachers disagree on both what to represent and how to represent it. Incompatible architectures and feature geometries produce conflicting gradients during distillation, degrading the student instead of enriching it.

This paper asks: can we build a single egocentric encoder that absorbs knowledge across viewpoints, modalities, and foundation models without the destructive interference? The answer is a two-stage mediation strategy.

[ASCII logic topology: Problem -> Assumption -> Method -> Evidence -> Conclusion]

                              +---

![Concept animation](/arxiv-visuals/uniego-proxies-as-mediators-for-unified/ConceptScene.gif)

--------------+
                              | Problem:        |
                              | Fragmented ego  |
                              | representations |
                              +--------+--------+
                                       |
                                       v
+--------------------------------------+--------------------------------------+
| Assumption:                          |                                      |
| Conflicting gradients from           |                                      |
| heterogeneous teachers can be        |                                      |
| resolved by inserting per-           |                                      |
| representation "proxies" that        |                                      |
| translate knowledge into a common    |                                      |
| egocentric space, then adaptively    |                                      |
| select reliable outputs.             |                                      |
+--------------------------------------+--------------------------------------+
                                       |
                                       v
+----------------------+       +----------------------+       +-------------------+
| Method:              | ----> | Stage 1: Train       | ----> | Stage 2: Select   |
| Hierarchical multi-  |       | Proxy models for each|       | and distill only  |
| teacher distillation |       | teacher (ego-exo,    |       | confident+correct |
| with proxy mediators |       | RGB, depth, skeleton,|       | proxy outputs,    |
| and selective        |       | foundation models)   |       | with convex init  |
| distillation (SPD)   |       | to homogeneous space |       | for student model |
+----------------------+       +----------------------+       +-------------------+
                                       |
                                       v
+----------------------+       +----------------------+       +-------------------+
| Evidence:            | ----> | SOTA on three tasks: | ----> | Conclusion:       |
| Experiments on three |       | action recognition,  |       | Proxy-mediated    |
| ego-exo benchmarks  |       | video retrieval,     |       | distillation      |
| (EPIC-Kitchens, EGTEA,|      | action segmentation  |       | produces richer,  |
| Assembly101)         |       | on all three         |       | unified ego reps  |
+----------------------+       +----------------------+       +-------------------+

The Increment

One sentence: Before this paper, multi-teacher distillation for egocentric video was a mess of conflicting gradients; after this paper, we have a principled two-stage framework—proxy translation followed by selective distillation—that yields state-of-the-art performance on three core tasks, using a single unified encoder.

Core Mechanism

The method has two main stages. First, train a set of Proxy models. Each proxy is a small network that takes the output of one specific teacher (e.g., a skeleton-based teacher or a foundation model like CLIP) and translates it into a fixed-dimensional representation in a shared “egocentric space.” The proxy learns to align the teacher’s heterogeneous feature space with a common target: the output of a strong egocentric RGB encoder (used as a pivot). This is done via a contrastive or regression loss—no gradients flow back to the teacher, only to the proxy. After training, we have nine proxies, each capable of outputting a representation in the shared space.

Second, Selective Proxy Distillation (SPD). Now we train the final student model (UNIEGO) to mimic the aggregated knowledge from these proxies. But not all proxies are equally reliable for every training sample. A proxy might be confident but wrong (e.g., for an ambiguous action), or correct but uncertain. SPD introduces a weighting mechanism: for each sample, we compute a “correctness” score (does the proxy agree with a small labeled validation set?) and a “confidence” score (entropy of the proxy’s output). Only proxies that are both correct and confident are used as teachers for that sample. The distillation loss is a weighted KL divergence, where weights are zero for unreliable proxies.

To stabilize the initial phase of SPD, the authors initialize UNIEGO’s parameters as a learned convex combination of the proxy parameters. This places the student in a well-conditioned region of the loss landscape, avoiding bad local minima.

[ASCII diagram of method internals: components, data flow, operations]

                  Teacher 1 (Ego-RGB)    Teacher 2 (Exo-Depth)  ... Teacher 9 (Foundation Model)
                        |                       |                        |
                        v                       v                        v
                 +-----------+          +-----------+            +-----------+
                 | Proxy 1   |          | Proxy 2   |            | Proxy 9   |
                 | (small MLP|          | (small MLP|            | (small MLP|
                 | + contrast|          | + contrast|            | + contrast|
                 +-----------+          +-----------+            +-----------+
                        |                       |                        |
                        +----------+------------+------------+----------+
                                   |            |            |
                                   v            v            v
                          +----------------------------------------+
                          |    Shared Egocentric Representation    |
                          |    (fixed dim, e.g., 512)              |
                          +----------------------------------------+
                                   |
                                   v
                          +----------------------------------------+
                          |       SPD Module (per sample)          |
                          |  Compute correctness + confidence      |
                          |  Select trusted proxies                |
                          |  Weighted distillation loss            |
                          +----------------------------------------+
                                   |
                                   v
                          +----------------------------------------+
                          |   Student UNIEGO (trained from scratch)|
                          |   Initialized as convex combo of      |
                          |   proxy parameters                    |
                          +----------------------------------------+
                                   |
                                   v
                          Output: unified ego features

Now, a structural metaphor: Think of the research group as a translation agency. The teachers are experts who speak different languages: one speaks “skeleton grammar,” another “depth topography,” yet another “foundation model dialect.” You want to hire a single interpreter (UNIEGO) who can answer any question posed in the “egocentric language.” But the experts give advice in their own languages, and if you try to translate them directly into egocentric, the mixed-up instructions confuse the interpreter.

So you first hire proxy translators: one per expert. Each proxy translator is a bilingual specialist who learns to convert their expert’s jargon into clean, standard egocentric speech. They practice by listening to a reliable bilingual guide (the pivot encoder) that already speaks egocentric well. After training, each translator can produce a coherent statement in the common language.

Now you want to train the new interpreter. But some translators occasionally give bad advice—they’re confident but wrong (imagine a translator who confidently says “turn left” when the correct answer is “turn right”). So you implement Selective Proxy Distillation: for each question you ask, you check each translator’s answer against a small answer key (labeled validation data) and also measure how sure they sound (low entropy). Only translators who are both correct and confident get to speak. Their answers are averaged with weights, and the interpreter learns from that filtered consensus.

Finally, to give the interpreter a good starting point, you don’t initialize them with random mental wiring. Instead, you set their initial “brain state” as a weighted average of all the translators’ own neural patterns (a convex combination of proxy parameters). This way, they start in a region of possibility space that’s already close to good answers.

The interpreter ends up being able to answer any egocentric question—recognize actions, retrieve videos, segment actions—better than any single expert could, because they’ve absorbed the complementary knowledge without the noise.

Key Concepts

• Proxy models: A small neural network (e.g., 2-layer MLP with layer norm) that maps the output of a teacher into a fixed-dimension shared space. The proxy is trained to align the teacher’s representation with a pivot encoder’s representation (e.g., a strong egocentric RGB encoder) using a contrastive loss (InfoNCE) plus a regression loss (L2). Critically, the teacher’s weights are frozen—only the proxy learns. This makes the proxy a “translator” that doesn’t alter the source expertise.

• Selective Proxy Distillation (SPD): A per-sample weighting scheme for multi-teacher distillation. For each training sample, SPD computes two scores for each proxy: (1) *correctness—measured by whether the proxy’s top-1 class prediction matches the ground truth on a held-out validation set (or using a small labeled subset); (2) confidence—the inverse entropy of the proxy’s output distribution. Only proxies exceeding thresholds on both scores are selected as teachers. The distillation loss weights are set to zero for unselected proxies. This prevents noisy or adversarial misguidance.

• Convex combination initialization: Before any distillation, the student UNIEGO’s parameters are set to a weighted sum of the proxy parameters: Theta_student = sum_i alpha_i ** Theta_proxy_i, where alpha_i are learned parameters initialized uniformly and then fine-tuned with a small amount of supervised data. This places the student’s initial position in parameter space inside the convex hull of the proxy parameter sets, which empirically helps avoid bad local minima during SPD.

Framework Shift

Before (mainstream approach):        After (this paper):

Teacher 1 ---+                       Teacher 1 --> Proxy 1 --+
Teacher 2 ---+-> Student (naive      Teacher 2 --> Proxy 2 --+-> SPD --> Student
Teacher 3 ---+  weighted avg loss)   Teacher 3 --> Proxy 3 --+   (selective)
    |                               More teachers: ... --> Proxies ...--+
    |                                                                   v
    Conflict!                                                       Clean, unified
    Student confused                                                 representation

One sentence: From direct, potentially conflicting multi-teacher distillation to a mediated channel with per-teacher translators and per-sample selective listening, the core shift is inserting a homogeneous translation layer and a reliability filter between heterogeneous teachers and the student.

Expert Assessment

Problem choice: This is a real gap. Egocentric video is inherently impoverished (single viewpoint, single modality). The field desperately needs ways to fuse complementary knowledge from third-person cameras, depth sensors, and large foundation models. The paper correctly identifies that naive multi-teacher distillation fails because of incompatible feature spaces. This isn’t a manufactured problem—it’s the exact obstacle that any practitioner hitting “concatenate all features” would encounter.

Method maturity: Clever, not brute force. The two-stage decomposition (proxy training then selective distillation) is elegant and principled. The convex combination initialization is a nice touch, though it adds engineering complexity. A simpler approach might be to just use knowledge distillation with a shared head, but the authors show that fails. The SPD weighting mechanism is reasonably light (single forward pass through each proxy + entropy computation). No obvious simpler baseline overlooked.

Experimental integrity: Baselines are fair: they compare against single-modality encoders, naive multi-teacher distillation (with and without weighting), and prior SOTA models. The gains are consistent across three benchmarks (EPIC-Kitchens, EGTEA, Assembly101) and three tasks. One red flag: the paper uses a large number of teachers (9), which might make the approach computationally expensive at training time. But the authors report inference time is comparable to a standard encoder. Also, the proxy training uses a pivot encoder—what if that pivot encoder is itself biased? They would need to show robustness to pivot choice; the paper doesn’t explore this.

Writing quality: Well-structured, clear motivation, good diagrams. The paper cuts corners in the ablation study: they don’t show the effect of removing each individual teacher, only aggregate. The section on convex combination initialization is under-explained—figuring out how alpha_i are learned and why it helps is left to the reader’s inference. If the authors rewrote Section 3.3 with a small toy example, the whole paper would be easier to grok.

Verdict: weak accept — the core idea is sound and the empirical results are convincing, but the ablation gaps and lack of pivot encoder sensitivity analysis prevent it from being a blockbuster.

Takeaways

1. Use proxy models as “impedance matchers” whenever you want to distill knowledge from heterogeneous pre-trained models into a single student. The proxy doesn’t need to be large—a small MLP with contrastive loss is sufficient to align feature spaces. This is a concrete technique you can steal for any multi-teacher distillation problem (text, audio, multimodal).

2. Selective distillation based on per-sample correctness+confidence is a general trick. You don’t have to trust every teacher equally. The paper’s method of using a small validation set to gauge correctness is practical. You could adapt this to online settings by using a held-out clean set.

3. Initialization matters more than most people think. The convex combination of teacher parameters is a cheap way to get a student into a good loss landscape region. This is especially useful when student and teacher architectures are similar—but the paper shows it works even with dissimilar architectures (proxies are different from student). Worth trying in any multi-teacher scenario.

If you work on egocentric video, this paper gives you a recipe to build a unified model that outperforms siloed specialists. If you work on representation learning generally, steal the two-stage mediation idea for your own heterogeneous data sources.