Sensor2Sensor: Converting Dashcam Videos into Multi-Modal AV Sensor Data

Paper: 2605.22809 Authors: Jiahao Wang, Bo Sun, Yijing Bai, Vincent Casser, Songyou Peng, Zehao Zhu, Meng-Li Shih, Xander Masotto, Shih-Yang Su, Kanaad V Parvate Categories: cs.CV

The Gap

Autonomous vehicle companies have high-quality sensor data (multi-camera rigs, LiDAR) but limited scale and diversity. The internet has billions of dashcam videos covering rare scenarios and diverse geographies, but AV systems can’t consume single-camera footage—they expect synchronized multi-modal inputs. Prior work either simulates synthetic data (lacks realism) or manually collects paired data (doesn’t scale). No one has cracked cross-embodiment sensor conversion: taking real-world monocular video and generating the multi-sensor suite an AV would have captured.

Problem: AV data scarcity + Internet video incompatibility
    |
    v
Assumption: Monocular video contains implicit 3D/multi-view info
    |
    v
Method: 4DGS reconstruction -> Paired data synthesis -> Diffusion-based conversion
    |
    v
Evidence: Quantitative metrics on generated cameras/LiDAR + Qualitative internet video demos
    |
    v
Conclusion: In-the-wild video becomes usable AV training data

The Increment

One sentence: Before this paper, internet dashcam videos were unusable for AV training; after, they can be converted into multi-camera + LiDAR sensor suites that AV systems expect.

Core Mechanism

Sensor2Sensor has two stages. First, create training pairs: take real AV logs (which have multi-camera + LiDAR), reconstruct the 4D scene using Gaussian Splatting, then render a synthetic dashcam view from the driver’s perspective. Now you have paired data: dashcam-style input → real multi-sensor output.

Second, train a diffusion model on these pairs. The model learns to invert the rendering process: given a monocular dashcam frame, generate what the surrounding cameras would see and what LiDAR points would exist in 3D space. The diffusion architecture handles the one-to-many ambiguity (many 3D scenes could produce the same 2D view) by learning a distribution over plausible sensor configurations.

At inference, feed in wild dashcam footage. The model hallucinates the missing sensor modalities—side cameras, rear camera, LiDAR—conditioned on the single front view. The output is a synthetic AV log that downstream perception systems can consume.

Training:
  Real AV log -> 4DGS reconstruction -> Render dashcam view
       |                                        |
       +----------------------------------------+
       |                                        |
       v                                        v
  [Multi-sensor ground truth]  <-paired->  [Dashcam input]
                                   |
                                   v
                            Diffusion model training

Inference:
  Wild dashcam video -> Diffusion model -> Generated multi-camera + LiDAR

Think of it like a forensic sketch artist working in reverse. A witness (dashcam) describes what they saw from one angle. The artist (diffusion model) has studied thousands of crime scenes (AV logs) and learned the typical spatial relationships—where objects usually are, how scenes are structured. From the single witness account, the artist reconstructs the full 360° scene, filling in what the other cameras would have seen and where solid objects must exist in 3D. The artist doesn’t know the exact truth, but produces a plausible reconstruction consistent with the witness testimony and learned scene priors.

Key Concepts

• 4D Gaussian Splatting (4DGS): Imagine representing a video scene as a cloud of 3D blobs (Gaussians), each with a position, color, opacity, and motion over time. To render any camera view, you project these blobs onto the image plane and blend their colors. 4DGS optimizes the blob parameters to match observed video frames. Once optimized, you can render novel views (like a dashcam perspective) from the same scene. It’s a differentiable renderer that turns multi-view video into a manipulable 3D representation. Concrete example: Given 6 camera views of a car passing by, 4DGS places Gaussians where the car’s surface is, then you can render what a 7th camera (the dashcam) would have seen.

• Cross-embodiment conversion: Different robots/vehicles have different sensor configurations (embodiments). A Tesla has 8 cameras in specific positions; a Waymo has different cameras plus LiDAR. Cross-embodiment means translating data from one sensor setup to another. Here, it’s extreme: from 1 camera (dashcam) to 5+ cameras + LiDAR (AV suite). The challenge is that sensor geometry, fields of view, and modalities differ—you’re not just filling in missing pixels, you’re inventing entire new viewpoints and 3D structure.

• Diffusion for sensor generation: Diffusion models learn to denoise: start with random noise, iteratively refine it into a sample from the training distribution. For sensor data, the model learns P(multi-sensor | dashcam). During training, it sees paired examples and learns what multi-camera + LiDAR configurations are consistent with a given dashcam view. At inference, it samples from this learned distribution, producing plausible sensor outputs. The stochasticity handles ambiguity—if the dashcam shows a car ahead but not its color, the model might generate red or blue, both valid.

Framework Shift

Before (mainstream approach):          After (this paper):
                                       
  Collect more AV data                   Leverage internet video
        |                                      |
        v                                      v
  [Expensive fleet ops]                  [Dashcam footage]
        |                                      |
        v                                      v
  Limited scale/diversity                4DGS reconstruction
        |                                      |
        v                                      v
  Train AV models                        Generate paired data
                                               |
                                               v
                                         Diffusion conversion
                                               |
                                               v
                                         Synthetic AV logs
                                               |
                                               v
                                         Train AV models
                                         (massive scale)

From closed-loop data collection to open-loop data synthesis, the core shift is treating the internet as a latent AV dataset that needs format conversion, not a separate unusable resource.

Expert Assessment

Problem choice: Real and urgent. AV companies are data-starved for long-tail scenarios (construction zones, rare weather, aggressive drivers). Internet video has this coverage but is locked behind an embodiment mismatch. The gap is genuine, not manufactured.

Method maturity: The 4DGS → diffusion pipeline is clever but feels like two heavy hammers. 4DGS reconstruction is expensive and assumes multi-view input exists for training data synthesis. The diffusion model is a black box—no explicit 3D reasoning, just learned correlations. A hybrid approach (explicit 3D lifting + learned refinement) might be more interpretable and data-efficient. The paper doesn’t explore lighter alternatives like NeRF-based synthesis or deterministic depth-based warping.

Experimental integrity: Baselines are thin. The paper compares against “no conversion” but not against simpler methods (monocular depth + geometric warping, or GAN-based approaches). Quantitative metrics (PSNR, LPIPS for images; Chamfer distance for LiDAR) are standard but don’t capture whether the generated data actually helps downstream AV tasks—no perception model evaluation. The qualitative internet video results look plausible but cherry-picked. Red flag: no failure case analysis or discussion of when the method breaks.

Writing quality: The abstract and intro are strong. The method section is dense—4DGS reconstruction is glossed over in half a page when it’s load-bearing for the whole pipeline. The experiments section front-loads metrics but buries the most interesting result (converting real internet footage) in supplementary material. Rewriting Section 4 to lead with qualitative internet examples, then justify with metrics, would make the contribution land harder.

Verdict: Weak accept — Addresses a real problem with a working solution, but the method feels over-engineered and the evaluation doesn’t prove the generated data is useful for actual AV training. The idea is more valuable than the execution.

Takeaways

Paired data synthesis via rendering: When you lack paired training data, reconstruct one modality into a manipulable 3D representation, then render the other modality. This trick generalizes beyond AVs—medical imaging (CT → MRI), robotics (RGB → depth), satellite imagery (optical → SAR).

Diffusion for multi-modal generation: Diffusion models handle one-to-many mappings gracefully. If your task has inherent ambiguity (one input, many valid outputs), diffusion lets you sample the distribution instead of forcing a single deterministic prediction. Useful for data augmentation, simulation, and any generative task where diversity matters.

Cross-embodiment as a framing: Reframing “data scarcity” as “embodiment mismatch” opens new solution paths. Instead of collecting more data in your target format, convert abundant data from other formats. This applies to any domain with heterogeneous sensors or data sources—medical devices, scientific instruments, IoT sensors.