BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering

Paper: 2606.17049 Authors: Yi-Ruei Liu, Jie-Ying Lee, Zheng-Hui Huang, Yu-Lun Liu, Chih-Hao Lin Categories: cs.CV

The Gap

Existing inverse rendering of urban scenes falls into two camps with complementary weaknesses. Physically-based methods (e.g., NeRF, 3D Gaussian Splatting with BRDF layers) reconstruct explicit scene properties — geometry, reflectance, lighting — but suffer from reconstruction artifacts (floaters, missing geometry) and rendering noise, especially in areas with sparse views or complex reflections. Generative video models (e.g., video diffusion) can produce impressively realistic novel views, but they offer no explicit control: you cannot relight a scene or insert a dynamic object with consistent shadows because they lack a physical world model.

The gap is clear: no existing method simultaneously gives you (1) explicit, consistent scene properties (needed for editing, simulation), (2) high-quality rendered videos free of artifacts, and (3) controllability over lighting and objects. BRDFusion proposes to fuse both approaches — use the physical model for structure and control, then let the generative model clean up the artifacts.

        +---------------------------+
        | Gap: Need both physical   |
        | consistency AND high-     |
        | quality renders, with     |
        | control.                  |
        +----------+----------------+
                   |
                   v
+------------------+----------------------+
| Assumption: Generative priors can       |
| denoise/fix physical rendering artifacts |
| without destroying explicit properties.  |
+------------------+----------------------+
                   |
                   v
+------------------+----------------------+
| Method: BRDFusion pipeline:             |
| 1) Inverse rendering -> explicit maps   |
| 2) Forward rendering -> noisy video     |
| 3) Generative model denoises -> clean   |
|    video with controllable edits.       |
+------------------+----------------------+
                   |
                   v
+------------------+----------------------+
| Evidence: Outperforms baselines on      |
| real & synthetic scenes; supports       |
| relighting, night sim, object editing.  |
+------------------+----------------------+
                   |
                   v
+------------------+------+
| Conclusion: Fusing      |
| physics + generation    |
| is the path forward for  |
| controllable, high-      |
| quality urban rendering.|
+-------------------------+

The Increment

One sentence: Before BRDFusion, you had to choose between physically consistent but artifact-ridden renders and high-quality but uncontrollable generative videos; after BRDFusion, you get both — explicit scene properties with high-quality, artifact-free rendering under full control.

Core Mechanism

BRDFusion has three main stages: inverse rendering, forward rendering, and generative refinement.

First, given a set of posed RGB videos of an urban scene, the system runs an inverse rendering pipeline that recovers explicit scene properties: geometry (point cloud or mesh), surface reflectance (diffuse albedo, roughness, metallic), and HDR lighting (in the form of environment maps or spherical harmonics). This is a standard optimization — it minimizes a photometric loss between rendered and observed pixel colors.

Second, once the scene properties are extracted, the system can perform forward rendering for any novel viewpoint and any lighting condition (including user-modified lighting). The output is a video sequence, but it contains artifacts: floaters, noise in specular regions, temporal inconsistency because the inverse rendering is imperfect.

Third — this is the novelty — a video diffusion model takes the noisy physically-rendered video as input and “cleans” it. The diffusion model is conditioned on the raw rendered frames, so it preserves the global scene structure and lighting modifications, but refines local details, removes artifacts, and improves temporal coherence. The key insight: the generative model does not hallucinate arbitrary content; it only corrects the defects that physics alone could not resolve, effectively acting as a learned denoiser with strong spatial-temporal priors.

+-------------------+       +------------------+       +-------------------+
| Inverse Renderer  |       | Forward Renderer |       | Generative Model  |
| (optimizes scene  |       | (renders video   |       | (video diffusion  |
|  properties from  |       |  from explicit   |       |  denoises and     |
|  input videos)    |       |  scene + light)  |       |  restores details)|
|                   |       |                  |       |                   |
| Input: videos     |       | Input: scene     |       | Input: noisy      |
| Output: geometry, | ----> |  properties +    | ----> |  rendered video   |
|  albedo, rough,   |       |  new light/cam   |       | Output: clean     |
|  envmap           |       | Output: noisy    |       |  video (with      |
+-------------------+       |  rendered video  |       |  control)         |
                            +------------------+       +-------------------+

Structural metaphor: a sculptor working on a marble block with a chisel, then a detailer with fine sandpaper.

Think of the inverse rendering stage as the sculptor’s rough chisel. The sculptor (physical model) has a block of marble (the scene) and chips away until a rough human figure emerges. This figure has correct proportions and pose (physical consistency), but its surface is coarse, with chisel marks and small cracks (rendering artifacts). This rough sculpture is the explicit scene property representation — you know the geometry, the material, where the light comes from.

Now the forward rendering is like the sculptor pressing the rough figure into a clay mold and then pulling a plaster cast. You can vary the mold (viewpoint) and you can mix different pigments into the plaster (lighting changes), so you get a series of casts that reflect your desired modifications. But every cast inherits the chisel marks and cracks from the original rough sculpture.

Finally, the generative model is the detailer with fine sandpaper and a small brush. They don’t change the pose or proportions (they preserve the structure and lighting modifications). They just sand down the rough edges, fill the cracks, polish the surface — removing artifacts and noise. The result is a smooth, museum-quality statue that still has the exact pose and modifications the sculptor intended.

This metaphor carries the load: chisel → rough explicit properties (physical, artifact-prone); mold/cast → forward rendering (controllable but noisy); sandpaper → generative denoising (non-destructive refinement). Without the sandpaper, you show a pockmarked statue. Without the chisel, you have no control over pose or material.

Key Concepts

• Inverse Rendering vs Forward Rendering: Inverse rendering is “what scene properties could produce these images?” — it’s an optimization that guesses geometry, materials, and lights from observations. Forward rendering is “given known scene properties and a camera/light configuration, what image would result?” — it’s a deterministic simulation using the rendering equation. BRDFusion uses inverse to extract properties (so you have control) and forward to generate novel views under modified conditions.

• Video Diffusion as Conditional Denoiser: A video diffusion model learns to gradually add noise to videos and then reverse the process to generate clean videos from noise. BRDFusion conditions it on the physically-rendered video (the noise source is artifacts, not Gaussian noise). The model is trained to take a corrupted (artifactual) video and output a cleaner version, using a temporal attention layer to maintain consistency across frames. This is different from text-to-video generation — here the model is not generating from scratch, but restoring a video that already has correct large-scale structure.

• Temporal Consistency: When rendering a video frame by frame independently, small errors in geometry or lighting cause flickering between frames. BRDFusion’s video diffusion model explicitly models temporal dependencies via 3D convolutions or temporal attention, so it smooths out these flickers — analogous to a video stabilization algorithm that works on content, not just motion vectors.

Framework Shift

Before (mainstream approach):
+-----------------------------+          +-------------------------------+
| Inverse Rendering (NeRF     |          | Generative Novel View         |
| / 3DGS + BRDF)              |          | Synthesis (Video Diffusion)   |
|                              |          |                               |
| Output: explicit scene      |          | Output: realistic video       |
| properties (geometry, mat,  |          | but NO control (can't relight,|
| light)                       |          | can't edit)                   |
| Then forward render:        |          |                               |
| artifacts + noise           |          |                               |
+-----------------------------+          +-------------------------------+
Two separate worlds, cannot combine.

After (this paper):
+---------------------------------------------------------------------+
|                    BRDFusion Pipeline                                |
|                                                                      |
| Inverse Rendering -> explicit scene -> Forward Render -> noisy vid   |
|                       (you can edit scene/light here)                |
|                                          |                           |
|                                          v                           |
|                               Generative Denoiser -> clean video     |
|                               (preserves edits, removes artifacts)   |
+---------------------------------------------------------------------+
One pipeline: physics for structure + control, generation for quality.

One sentence: From “pick either physical control with artifacts OR high quality without control” to “have both via a two-stage chisel-and-sandpaper pipeline”, the core shift is using generative models as artifact removers rather than content creators, preserving controllability while dramatically improving visual quality.

Expert Assessment

Problem choice: Real gap. The tension between physical inverse rendering (artifact-prone but controllable) and generative synthesis (high-quality but uncontrollable) is a central bottleneck for urban scene editing and simulation. This is not manufactured — industry practitioners (autonomous driving, robotics) explicitly need both.

Method maturity: Clever insight, not brute force. The idea of using diffusion as a post-process denoiser for physics-based renders is elegant and simple in retrospect. The approach is more “architectural insight” than “massive compute.” But there is an overlooked simpler approach: could a lightweight neural denoiser (like an image-to-image conditional U-Net trained on paired synthetic/noisy renders) achieve similar results with lower complexity? The paper does not fully explore this baseline. The video diffusion model is heavy; a lighter model might work.

Experimental integrity: Fair baselines (NeRF, 3DGS, etc.). Quantitative metrics (PSNR, LPIPS, FVD) are standard and the improvements are visible. Two potential red flags: (1) The comparisons are mostly on relative quality — the paper does not show systematic ablation of the generative model’s capacity (e.g., what if you use a small denoiser vs. full diffusion?). (2) The “novel-view relighting” and “night simulation” results look impressive but might cherry-pick frames where the generative model introduces plausible but incorrect physics (e.g., adding fake shadows that didn’t exist in the physical model). They should show failure cases.

Writing quality: Clear and well-structured. The motivation is strong. The weak section is the related work — it’s a laundry list without a coherent narrative of how each prior method fails relative to the BRDFusion approach. Rewriting related work to actually tell the story of the gap would elevate the paper significantly.

Verdict: strong accept — a simple, effective fusion of physics and generation that fills a real gap, despite some minor evaluation concerns.

Takeaways

• Denoising as a service: Any inverse rendering method that produces noisy intermediates can slap a generative denoiser on top. This is a transferable trick: recover whatever physics-based representation you need, render your edited content, then run a general video diffusion model conditioned on the render to remove artifacts.
• Conditional diffusion with preservation: The key design choice is conditioning the diffusion model on the raw render (not on text). This ensures the generative model does not change the content you want to control. Practitioners can use this trick in any domain where they have a deterministic but flawed simulator and want to produce clean output without losing control.
• Chisel-and-sandpaper pipeline architecture: The two-stage inverse→forward (chisel) then denoising (sandpaper) pattern can be applied to other disciplines: e.g., physics simulation of fluids (rough) enhanced by a learned super-resolver; audio synthesis with a deterministic synthesizer cleaned by a neural vocoder.