Bi-CMPStereo: Bridging Event and Frame Cameras Through Bidirectional Cross-Modal Prompting

Paper: 2604.15312 Authors: Ninghui Xu, Fabio Tosi, Lihui Wang, Jiawei Han, Luca Bartolomei, Zhiting Yao, Matteo Poggi, Stefano Mattoccia Categories: cs.CV

The Gap

Event cameras and frame cameras capture the world differently: frames give you rich context but blur under motion, events give you microsecond precision but sparse data. Combining them for stereo depth estimation should be a slam dunk — complementary strengths, right? But existing methods hit a wall: the modality gap is so wide that cross-modal matching marginalizes domain-specific cues. Prior work (like DDES, E2VID-based approaches) either converts events to pseudo-frames (losing temporal precision) or uses asymmetric architectures that favor one modality over the other. The result: you get neither the frame’s semantic richness nor the event’s temporal sharpness when you need both.

Problem: Event-frame stereo matching fails under motion/illumination
    |
    v
Assumption: Modality gap causes loss of domain-specific cues
    |
    v
Method: Bidirectional prompting projects each modality into both domains
    |
    v
Evidence: Outperforms SOTA on MVSEC, DSEC benchmarks (accuracy + generalization)
    |
    v
Conclusion: Aligned canonical space + bidirectional projection = robust matching

The Increment

One sentence: Before this paper, event-frame stereo chose one modality’s representation space and forced the other to adapt; after, both modalities project into each other’s space bidirectionally, preserving domain-specific strengths.

Core Mechanism

Bi-CMPStereo has three stages. First, separate encoders extract features from event and frame inputs — no forced conversion yet. Second, a cross-modal prompting module projects event features into frame space AND frame features into event space simultaneously. This isn’t just feature fusion; it’s creating four representations: original event, event-prompted-by-frame, original frame, frame-prompted-by-event. Third, a stereo matching network operates in a canonical space where all four representations are aligned, computing cost volumes and refining disparity maps.

The data flow: left/right event streams and frames enter parallel encoders → cross-modal prompting generates bidirectional projections → all representations feed into a unified stereo matcher → disparity output. The key operation is the prompting mechanism itself: learnable queries extract domain-specific patterns from one modality and inject them into the other’s feature space, guided by attention mechanisms that preserve structural coherence.

Event_L ----[Encoder]----> E_L ----\
                                     \
Frame_L ----[Encoder]----> F_L ------+--> [Cross-Modal Prompting] --> [Canonical Space]
                                     /                                        |
Event_R ----[Encoder]----> E_R ----/                                         v
                                                                      [Stereo Matcher]
Frame_R ----[Encoder]----> F_R ----\                                         |
                                     \                                        v
                                      +--> [Cross-Modal Prompting]      Disparity Map
                                     /
                                    /

Think of it like a bilingual negotiation. You have an English speaker (frame camera) and a Mandarin speaker (event camera) trying to agree on a 3D map. Old methods forced one to speak the other’s language exclusively — the Mandarin speaker learns English but forgets nuanced Mandarin idioms, or vice versa. Bi-CMPStereo gives each speaker a real-time interpreter that works both ways: the English speaker gets Mandarin context injected into their thoughts, the Mandarin speaker gets English structure woven into theirs. They both think in their native language AND the other’s simultaneously. The “canonical space” is the shared conference room where all four versions of understanding (original English, English-with-Mandarin-context, original Mandarin, Mandarin-with-English-structure) converge to hammer out the final agreement. No one loses their native fluency, but everyone gains cross-cultural insight.

Key Concepts

• Cross-Modal Prompting: Imagine you’re a detective with two witnesses — one saw the crime in daylight (frame), one heard it in darkness (event). Traditional approaches make you pick one testimony as “ground truth” and try to translate the other into that format. Cross-modal prompting instead asks: “What would the daylight witness have seen if they had the night listener’s temporal hearing?” and “What would the night listener have heard if they had the daylight witness’s spatial vision?” You generate hypothetical testimonies that blend both perspectives. Technically, this uses learnable prompt tokens (queries) that attend to one modality’s features and generate augmented features for the other modality. The prompts are trained end-to-end to extract complementary information — not just copy features, but synthesize what’s missing.

• Canonical Space Alignment: When you match stereo pairs, you need corresponding points to “look similar” in feature space. But event features and frame features live in different universes — like comparing temperature in Celsius to distance in miles. A canonical space is a shared coordinate system where both get mapped. The trick: instead of forcing one modality to adopt the other’s coordinates (which loses information), you learn a third space where event-ness and frame-ness are both expressible. It’s like inventing a new unit that can represent both temperature and distance meaningfully. The alignment happens through contrastive learning: corresponding points from left/right images should cluster together in this space, regardless of whether they came from events or frames.

• Bidirectional Projection: Most fusion methods are one-way streets: convert events to frames, then do frame-based stereo. Bidirectional means traffic flows both ways simultaneously. You project events into frame domain (gaining semantic context) AND frames into event domain (gaining temporal precision). Why does this matter? Because each modality has blind spots. Frames can’t see fast motion; events can’t see static texture. By projecting bidirectionally, you fill each modality’s blind spots with the other’s strengths. Concretely: frame features get augmented with event-derived motion cues, event features get augmented with frame-derived texture cues. Both augmented versions then participate in stereo matching, so the network can choose the best cue for each pixel.

Framework Shift

Before (mainstream approach):          After (this paper):

Event --> [Convert to Frame] ----\     Event ----[Encoder]----> E ----\
                                  \                                    \
Frame ----[Encoder]-------------> F    Frame ----[Encoder]----> F -----+--[Prompt]--\
                                  |                                    /              \
                                  v                                   /                v
                          [Frame-based Stereo]                  E<--[Prompt]    [Canonical Space]
                                  |                                                    |
                                  v                                                    v
                             Disparity                                            Disparity

(Single representation space,          (Four representations: E, F, E', F'
 one modality dominates)                all contribute equally)

From asymmetric adaptation to symmetric collaboration, the core shift is treating both modalities as first-class citizens in their own domains while enabling mutual enrichment.

Expert Assessment

Problem choice: Real gap. Event-frame stereo is genuinely hard because the modalities are fundamentally different (asynchronous sparse events vs synchronous dense frames), and prior conversion-based methods demonstrably lose information. This sits at the intersection of neuromorphic vision and 3D perception — a growing area as event cameras become cheaper. Not manufactured.

Method maturity: Clever insight with some brute force. The bidirectional prompting idea is elegant — it respects modality differences instead of erasing them. But the architecture is heavy: dual encoders, dual prompting modules, canonical space learning. A simpler baseline would be: what if you just concatenated event/frame features and let a big network figure it out? The paper doesn’t ablate against this “dumb fusion” thoroughly enough. The prompting mechanism uses standard cross-attention; the novelty is in the bidirectional application, not the operation itself.

Experimental integrity: Baselines are fair (DDES, E2VID-based methods, recent event-frame works). Numbers look solid on MVSEC and DSEC benchmarks. One red flag: generalization claims are based on cross-dataset evaluation, but both datasets are driving scenes — would this hold on indoor scenes or non-rigid objects? The ablation studies are thorough (removing bidirectionality hurts, removing canonical space hurts), which is good. No obvious cherry-picking in qualitative results.

Writing quality: The method section is dense — too much notation, not enough intuition upfront. Figure 2 (architecture diagram) tries to show everything at once and ends up cluttered. If they rewrote Section 3.2 (Cross-Modal Prompting) to lead with the bilingual negotiation metaphor, then formalize, the paper would be 30% more accessible. The related work section is thorough but reads like a list; grouping by “conversion-based vs architecture-based vs fusion-based” would clarify positioning.

Verdict: weak accept — Solid contribution to a real problem, method is sound and results are convincing, but the architecture complexity and limited scope of generalization experiments keep it from strong accept territory.

Takeaways

Steal the bidirectional prompting pattern for any multi-modal fusion problem where modalities have complementary strengths. The key transferable idea: instead of projecting modality A into B’s space (or vice versa), project both into each other’s spaces AND into a shared space. This preserves domain-specific features while enabling cross-modal reasoning. Concretely: if you’re fusing LiDAR and camera, don’t just colorize point clouds or project images to 3D — do both, then match in a learned canonical space. The canonical space alignment via contrastive learning is also portable: train corresponding points to cluster regardless of source modality. Finally, the paper’s ablation methodology is worth copying: systematically remove each component (bidirectionality, canonical space, prompting) and measure impact on both accuracy and generalization.