SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation

Paper: 2602.23359 Authors: Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat, Ravi Kiran Sarvadevabhatla, R. Venkatesh Babu Categories: cs.CV, cs.AI

Abstract

SeeThrough3D addresses a critical gap in 3D layout-conditioned image generation: the ability to reason about object occlusions. While existing text-to-image models can generate scenes following spatial layouts, they struggle with modeling precise inter-object occlusions, leading to depth inconsistencies and unrealistic spatial relationships. This work introduces an occlusion-aware 3D scene representation (OSCR) that depicts objects as translucent 3D bounding boxes in a virtual environment. The transparency encoding allows the model to understand which parts of objects should be hidden, while the rendered viewpoint provides explicit camera control. By conditioning a pretrained flow-based diffusion model with visual tokens derived from this representation and applying masked self-attention for object-text binding, SeeThrough3D achieves accurate multi-object generation without attribute mixing, generalizing effectively to unseen categories.

Key Contributions

• Introduction of occlusion-aware 3D scene representation (OSCR) using translucent 3D bounding boxes to explicitly encode object visibility and occlusion relationships
• A conditioning mechanism that integrates OSCR-derived visual tokens into pretrained flow-based text-to-image models for occlusion-aware generation
• Masked self-attention architecture that binds each object’s bounding box to its textual description, preventing attribute leakage across objects
• Construction of a synthetic training dataset featuring diverse multi-object scenes with strong inter-object occlusions
• Demonstrated generalization to unseen object categories with precise 3D layout control and realistic depth-consistent occlusions

Technical Methodology

The core innovation of SeeThrough3D lies in its occlusion-aware scene representation. Traditional 3D layout conditioning methods use opaque bounding boxes or depth maps, which fail to convey information about occluded regions. OSCR renders objects as semi-transparent boxes from a specified camera viewpoint, where the alpha channel encodes visibility—fully transparent regions indicate completely occluded areas, while opaque regions represent visible surfaces.

The architecture builds upon a pretrained flow-based diffusion model (likely Stable Diffusion 3 or similar). The OSCR representation is processed through a visual encoder to extract spatial tokens that capture both geometric layout and occlusion patterns. These tokens are injected into the model’s cross-attention layers alongside text embeddings.

A critical component is the masked self-attention mechanism. For each object $i$ with bounding box $B_i$ and text description $T_i$, the attention is masked such that spatial regions within $B_i$ can only attend to tokens from $T_i$. This prevents the common failure mode where attributes from multiple objects blend together (e.g., “red car” and “blue truck” producing a purple vehicle). Mathematically, the attention mask $M_{ij}$ is defined such that:

$M_{ij} = \begin{cases} 0 & \text{if pixel } j \in B_i \text{ and token from } T_i \\ -\infty & \text{otherwise} \end{cases}$

The training dataset is synthetically generated using 3D rendering engines, creating scenes with controlled occlusion patterns. This ensures the model learns robust occlusion reasoning across various spatial configurations, object scales, and camera angles.

Results and Capabilities

SeeThrough3D demonstrates significant improvements over baseline methods in several key areas:

Occlusion Accuracy: The model correctly renders partially occluded objects with appropriate depth ordering. Objects behind others are properly hidden in occluded regions while maintaining geometric consistency in visible areas.

Camera Control: The explicit viewpoint rendering in OSCR provides precise camera control. Users can specify exact camera positions and orientations, and the generated images accurately reflect these parameters with correct perspective and foreshortening.

Multi-Object Generation: The masked self-attention mechanism successfully prevents attribute mixing. When generating scenes with multiple objects having distinct properties (colors, textures, shapes), each object maintains its specified attributes without cross-contamination.

Generalization: Despite training on synthetic data, the model generalizes to real-world object categories not seen during training. This suggests the occlusion reasoning learned from geometric relationships transfers across semantic categories.

Scale and Depth Consistency: Objects maintain physically plausible sizes relative to their depth in the scene. Distant objects appear appropriately smaller, and the depth relationships remain consistent across the generated image.

Implications and Future Directions

SeeThrough3D represents a significant step toward controllable 3D-aware image synthesis. The explicit modeling of occlusions addresses a fundamental limitation in spatial conditioning for generative models. This has practical applications in:

• Virtual scene design: Designers can precisely control object placement and camera angles for concept art and visualization
• Data augmentation: Generating training data with controlled occlusions for object detection and segmentation tasks
• AR/VR content creation: Producing realistic scenes with proper depth relationships for immersive experiences

The approach opens several research directions. First, extending beyond bounding boxes to more detailed 3D representations (meshes, point clouds) could enable finer geometric control. Second, incorporating physical constraints (gravity, support relationships) could improve scene plausibility. Third, interactive editing capabilities where users can manipulate the 3D layout and see real-time updates would enhance usability.

The reliance on synthetic training data, while enabling controlled learning, may limit photorealism compared to models trained on real images. Hybrid approaches combining synthetic occlusion data with real image datasets could balance geometric accuracy with visual fidelity.

Takeaways

1. Occlusion reasoning is essential for depth-consistent 3D layout-conditioned generation, yet has been largely overlooked in existing methods
2. Translucent 3D box representations effectively encode visibility information, enabling models to learn which object regions should be hidden
3. Masked self-attention prevents attribute mixing in multi-object scenes by binding spatial regions to their corresponding text descriptions
4. Explicit camera viewpoint rendering in the conditioning signal provides precise control over perspective and viewing angles
5. Synthetic training data with controlled occlusions enables robust learning of geometric relationships that generalize to unseen categories
6. The approach bridges the gap between 2D image generation and 3D scene understanding, advancing controllable synthesis capabilities