OmniLiDAR: Unified Multi-Domain LiDAR Generation via Text-Conditioned Diffusion

Paper: 2605.13815 Authors: Youquan Liu, Weidong Yang, Ao Liang, Xiang Xu, Lingdong Kong, Yang Wu, Dekai Zhu, Xin Li, Runnan Chen, Ben Fei Categories: cs.CV, cs.RO

The Gap

Existing LiDAR generation methods train separate diffusion models for each domain—one for rainy weather, another for 32-beam sensors, yet another for drone-mounted LiDAR. This domain-specific approach creates three problems: (1) you need N models for N domains, (2) you can’t smoothly interpolate between conditions (e.g., “light rain with 64 beams”), and (3) each model sees only a narrow slice of the data distribution, limiting generalization.

Prior work like LiDARGen and LiDAR-Diffusion focused on single-domain generation with high fidelity but no cross-domain capability. The fundamental barrier: heterogeneous distribution shifts (weather corruption, beam count changes, platform differences) break the shared latent space assumption that standard diffusion models rely on.

Problem: Domain-specific models
    |
    v
Assumption: Text conditioning + mixed-batch training
            can unify heterogeneous domains
    |
    v
Method: CDTS (cross-domain batches) +
        CDFM (directional dependencies) +
        DAFS (domain-adaptive scaling)
    |
    v
Evidence: Single model matches/exceeds
          domain-specific baselines across 8 domains
    |
    v
Conclusion: Unified generation is feasible
            with proper conditioning + architecture

The Increment

One sentence: Before—one model per domain, no cross-domain control; after—one model for all domains with text-driven interpolation.

Core Mechanism

OmniLiDAR represents LiDAR scans as range images (2D grids where each pixel encodes distance, intensity, and other attributes). The model is a latent diffusion architecture: an autoencoder compresses range images into a latent space, then a denoising U-Net trained with DDPM learns to reverse the noise process conditioned on text prompts like “rainy weather, 32 beams, vehicle platform.”

Three components handle domain heterogeneity. Cross-Domain Training Strategy (CDTS) mixes samples from different domains within each mini-batch and uses text embeddings (via CLIP) to tell the model which domain it’s generating. Cross-Domain Feature Modeling (CDFM) adds directional attention layers that scan along azimuth (horizontal) and elevation (vertical) axes separately, capturing the anisotropic structure of rotating LiDAR sensors. Domain-Adaptive Feature Scaling (DAFS) modulates intermediate features with learned domain-specific scale and shift parameters, allowing the model to adjust its internal representations per domain without full parameter isolation.

Input: Noise + Text ("rainy, 32-beam, vehicle")
    |
    v
[Latent Encoder] --> Compressed representation
    |
    v
[U-Net Denoiser]
    |-- CDFM: Azimuth attention --+-- Elevation attention
    |                              |
    |-- DAFS: Domain-specific scaling per layer
    |
    v
[Latent Decoder] --> Range image (H x W x C)
    |
    v
Output: LiDAR scan

Think of OmniLiDAR as a multilingual translator with dialect adapters. The base U-Net is the core translation engine that understands the universal grammar of 3D geometry. CDTS is the training curriculum—instead of learning French in isolation, you learn French, Spanish, and Italian in the same classroom, forcing the model to extract shared structure. CDFM is the syntax parser that knows LiDAR “sentences” have directional grammar (azimuth sweeps left-to-right, elevation stacks top-to-bottom). DAFS is the dialect adapter—when translating to Québécois French vs Parisian French, you apply lightweight pronunciation shifts without retraining the whole engine. The text prompt is the instruction: “translate this noise into a rainy 32-beam vehicle scan.”

Key Concepts

• Range Image Representation: LiDAR sensors emit laser pulses in a rotating pattern, sweeping horizontally (azimuth) and vertically (elevation). Instead of storing raw 3D points, you can organize them into a 2D grid where rows correspond to elevation angles and columns to azimuth angles. Each grid cell stores the measured distance (range), intensity, and other attributes. This representation is natural for diffusion models because it’s a regular 2D image, but it preserves the anisotropic scanning structure—nearby cells in the azimuth direction are temporally adjacent in the sensor’s rotation, while nearby cells in elevation are spatially adjacent on the sensor’s vertical array. CDFM exploits this by applying separate attention along each axis.

• Cross-Domain Training Strategy (CDTS): Standard practice trains one model per domain to avoid conflicting gradients. CDTS does the opposite—it deliberately mixes domains in each mini-batch and relies on text conditioning to disambiguate. The key insight: if the model sees “rainy” and “sunny” scans in the same batch, it’s forced to learn a shared latent space where weather is a controllable axis, not a baked-in assumption. This is similar to multilingual NLP models that train on mixed-language batches—the model learns language-agnostic representations because it can’t afford to specialize.

• Domain-Adaptive Feature Scaling (DAFS): Different domains have different feature statistics—rainy scans have lower intensity values, 16-beam scans have sparser patterns. DAFS learns a small set of scale and shift parameters for each domain and applies them to intermediate U-Net features during denoising. It’s like batch normalization, but domain-specific. The parameters are predicted from the text embedding, so the model can smoothly interpolate between domains (e.g., “light rain” gets a blend of “sunny” and “heavy rain” scaling factors). This is cheaper than training separate models because you’re only modulating existing features, not learning new weights.

Framework Shift

Before (domain-specific models):        After (OmniLiDAR):

Model_rain --> Rain scans               Text: "rain, 32-beam, vehicle"
Model_fog  --> Fog scans                    |
Model_32   --> 32-beam scans                v
Model_64   --> 64-beam scans           [Single Model]
Model_car  --> Vehicle scans                |
Model_drone--> Drone scans                  +-- CDTS: Mixed batches
...                                         +-- CDFM: Directional attention
                                            +-- DAFS: Domain scaling
N models, no interpolation                  |
                                            v
                                       Any domain + interpolation

From isolated specialists to a unified generalist with controllable axes—the core shift is treating domains as conditioning variables rather than model boundaries.

Expert Assessment

Problem choice: Real gap. Autonomous driving and robotics need synthetic LiDAR data under diverse conditions (weather, sensor degradation, platform changes), but collecting real data at scale is prohibitively expensive. Existing single-domain generators don’t address the practical need for controllable, multi-condition synthesis. The problem sits at the intersection of generative modeling and sim-to-real transfer, both active areas.

Method maturity: Solid engineering with one clever insight (CDTS) and two incremental additions (CDFM, DAFS). CDTS is the load-bearing idea—mixed-batch training with text conditioning is simple but non-obvious, and it’s what enables the unified model. CDFM is a reasonable architectural choice given the anisotropic structure of range images, though it’s not groundbreaking (directional attention has been explored in other domains). DAFS feels like a hedge—the authors weren’t sure if CDTS alone would handle domain shifts, so they added lightweight domain-specific parameters. It works, but I suspect a simpler baseline (CDTS + standard U-Net) would get 80% of the gains.

Experimental integrity: The authors constructed their own 8-domain dataset by combining real scans (KITTI, nuScenes, etc.) with simulated weather and beam reduction. This is both a strength and a weakness—strength because no public benchmark exists, weakness because we can’t verify the simulation quality or compare to other methods on the same data. The baselines are fair (domain-specific diffusion models), and the downstream tasks (segmentation, detection) show consistent gains, which is more convincing than generation metrics alone. One red flag: no ablation isolating CDTS vs CDFM vs DAFS contributions. The paper shows “full model vs no DAFS,” but I want to see “CDTS only” vs “CDTS + CDFM” vs “full model.”

Writing quality: The abstract and introduction are clear, but the method section is dense and assumes familiarity with diffusion models. The CDFM explanation is buried in notation—Figure 3 helps, but a standalone paragraph walking through one forward pass would clarify. The experimental section is thorough but repetitive (eight domains × three tasks = a lot of tables). A summary figure showing relative gains across domains would be more digestible. The related work section is perfunctory—it lists prior LiDAR generation papers but doesn’t explain why they can’t handle multi-domain settings.

Verdict: Weak accept—the unified model is a practical contribution with solid experimental validation, but the method is more engineering than insight, and the lack of public benchmarks limits reproducibility.

Takeaways

Steal the training strategy: CDTS (mixed-batch training with text conditioning) is a general recipe for unifying heterogeneous domains. If you’re training models on data with natural partitions (languages, modalities, sensor types), try mixing them in mini-batches and using conditioning to disambiguate. This forces the model to learn shared structure instead of overfitting to domain-specific quirks.

Directional attention for structured data: If your data has anisotropic structure (e.g., time series with different temporal vs feature-wise dependencies, or images with known directional biases), consider separate attention mechanisms along each axis. CDFM’s azimuth/elevation split is a concrete example.

Lightweight domain adaptation: DAFS shows you don’t need separate models for domain shifts—learned scale/shift parameters can handle a lot of variation. This is useful for deployment: train one model, then fine-tune only the DAFS parameters for new domains.

Generative augmentation for 3D perception: The downstream experiments (segmentation, detection) show that synthetic LiDAR scans improve performance in limited-label regimes. If you’re working on 3D perception with scarce data, generative augmentation is worth trying—but validate carefully, because bad synthetic data can hurt more than help.