Thinking in Blender: From Pixels to Editable 3D Code with Vision-Language Models

Paper: 2606.02580 Authors: Guangzhao He, Rundong Luo, Wei-Chiu Ma, Hadar Averbuch-Elor Categories: cs.CV

The Gap

Inverse graphics aims to reconstruct images as editable 3D scenes. Current approaches rely on specialized 2D/3D foundation models (like NeRF, Gaussian Splatting), differentiable renderers, or multi-view images. These methods produce reconstructions that are hard to edit—you get a 3D representation, but modifying the material of a single object or adjusting lighting requires expertise and often manual intervention. The constraint: existing methods optimize in implicit or continuous spaces that don’t map cleanly to the discrete, symbolic operations artists use in 3D software.

This paper asks: can a general-purpose vision-language model reconstruct a scene directly as executable Blender code from a single image, without domain-specific 3D priors?

Problem: Single image -> editable 3D scene
   |
   v
Existing constraint: Specialized models + implicit representations
   |                  (hard to edit, not symbolic)
   v
Assumption: VLMs understand both visual structure AND code syntax
   |
   v
Method: Staged refinement in Blender code space
   |      (geometry -> materials -> composition -> lighting)
   v
Evidence: Quantitative metrics + downstream editing tasks
   |
   v
Conclusion: Task decomposition enables executable inverse graphics

The Increment

One sentence: Before this paper, inverse graphics produced hard-to-edit implicit 3D representations; after, a single image becomes an editable Blender script you can modify like any code.

Core Mechanism

The method treats inverse graphics as staged code generation. Given an input image, a vision-language model generates Blender Python code in four sequential stages: (1) geometry construction (primitives, meshes), (2) material assignment (colors, textures, reflectance), (3) spatial composition (object placement, camera view), and (4) lighting setup (light sources, environment maps). Each stage refines the previous stage’s output by comparing rendered results against the input image.

Data flows like this: the VLM receives the input image plus the current Blender code state, proposes code modifications, executes them in Blender, renders the scene, compares the rendering to the target, and iterates. The VLM operates as both programmer and critic—it writes code, evaluates visual discrepancies, and debugs. Refinement loops within each stage until convergence or iteration limit, then the next stage begins with the accumulated code.

The staged structure mirrors how artists build 3D scenes: rough shapes first, then materials, then layout, then lighting. Each stage constrains the search space—you don’t simultaneously optimize all factors. The VLM uses visual feedback (rendered vs. target) to guide code edits, but unlike gradient-based methods, edits are discrete symbolic operations (add_cube, set_metallic, move_light) that remain human-readable and editable.

Structural metaphor: Think of it as reverse-engineering a recipe by tasting the dish. You start with a photo of a meal. Stage 1: identify the main ingredients (is that chicken or tofu?). Stage 2: figure out seasonings and cooking methods (fried or baked? salty or sweet?). Stage 3: arrange the plating (what’s in front, what’s garnish?). Stage 4: adjust lighting for the food photo (harsh or soft light?). At each stage, you cook a test version, compare it to the photo, adjust the recipe, and lock in that part before moving to the next. The final output isn’t just a similar-looking dish—it’s a recipe anyone can follow, modify, and reuse.

Key Concepts

• Executable inverse graphics: Traditional inverse graphics produces 3D representations (voxels, meshes, radiance fields) optimized to match an image. Executable inverse graphics produces a program—code that, when run, generates the 3D scene. The difference: a program is symbolic, editable, and compositional. You can change line 47 to swap a sphere for a cube, or adjust a parameter to make something shinier. It’s the difference between getting a JPEG of a chart versus the Excel file that made it. The program is the representation, not just a path to one.

• Staged refinement: Instead of optimizing all scene factors simultaneously (geometry, materials, lighting), the method decomposes the problem into sequential stages with clear dependencies. Geometry comes first because you can’t assign materials to objects that don’t exist. Materials come before lighting because reflectance properties determine how light interacts. This isn’t just a convenience—it’s a regularization strategy. By forcing the model to commit to geometry before tweaking materials, you prevent it from “cheating” by compensating for wrong shapes with wrong colors. Each stage reduces the hypothesis space for the next.

• VLM as agentic reconstructor: The vision-language model acts as an agent that observes (compares renderings), plans (proposes code changes), acts (modifies Blender script), and iterates. It’s not generating code in one shot—it’s debugging in a loop. The VLM’s dual capability (visual understanding + code generation) bridges the pixel-symbolic gap. It sees that an object is too dark, understands that darkness comes from material properties or lighting, and translates that understanding into a specific code edit. The agent has no specialized 3D inductive bias—it reasons from scratch using visual-linguistic priors.

Framework Shift

Before (NeRF/3DGS pipeline):        After (SEIG):
                                    
Input image                         Input image
    |                                   |
    v                                   v
Multi-view or pose                  VLM + Blender runtime
    |                                   |
    v                                   v
Differentiable render           Stage 1: Code -> Geometry
    |                                   |
    v                                   v
Gradient descent                Stage 2: Code -> Materials
    |                                   |
    v                                   v
Implicit 3D (NeRF/Gaussian)     Stage 3: Code -> Composition
    |                                   |
    v                                   v
Hard to edit manually           Stage 4: Code -> Lighting
                                        |
                                        v
                                    Blender script
                                        |
                                        v
                                    Editable 3D scene

One sentence: From continuous optimization in implicit space to discrete symbolic refinement in code space, the core shift is making the reconstruction itself a human-readable program.

Expert Assessment

Problem choice: Real gap. Inverse graphics has produced impressive neural reconstructions, but the editing bottleneck is underexplored. Artists want scene graphs and parameters, not million-dimensional latent vectors. The timing is right—VLMs are capable enough to attempt this, but the problem isn’t artificially constructed to showcase them.

Method maturity: Clever insight with practical limitations. The staged decomposition is well-motivated and the choice to work in code space is bold. However, relying purely on VLM reasoning without any 3D inductive bias is both a strength (generality) and a weakness (efficiency). The method likely struggles with complex geometry or subtle lighting that requires precise spatial reasoning. The paper would benefit from ablating the staging—does the order matter? What if stages overlap?

Experimental integrity: Baselines are reasonable (NeRF, 3DGS, text-to-3D methods) but not perfectly matched—those methods use multi-view or 3D priors, which is somewhat unfair. However, the point is to show VLMs alone can do this, so it’s defensible. The metrics (pixel-level, perceptual, semantic) cover different fidelity dimensions, which is good. Missing: failure case analysis. When does this completely fall apart? Transparent objects? Mirrors? Fine text? The downstream applications (relighting, editing) are shown qualitatively but not measured rigorously.

Writing quality: The paper is clear in motivation but light on technical depth. The staging strategy is described at a high level, but implementation details are sparse—what prompt templates guide each stage? How many iterations per stage? How does the VLM know when to stop refining? The related work section conflates several research threads (inverse graphics, program synthesis, VLM agents) without clearly positioning this work. The results section would be stronger with explicit discussion of where the method fails and why.

Verdict: weak accept — promising direction that demonstrates VLMs can perform executable inverse graphics, but needs deeper analysis of failure modes, ablations on staging strategy, and more rigorous evaluation of downstream editability to be truly convincing.

Takeaways

Practitioners can steal the staged refinement pattern for any problem where you’re generating complex structured outputs from vision. If you’re building a vision-to-X system (vision-to-CAD, vision-to-recipe, vision-to-floorplan), consider decomposing into stages with clear dependencies rather than generating everything at once. Each stage locks in one aspect and constrains the search space for the next.

The idea of using code as the representation space is transferable beyond 3D. If your domain has a programmatic API (design software, robotics, data pipelines), you can frame generation as code synthesis and leverage VLMs’ dual vision-code capability. The rendered output becomes your evaluation signal—execute the code, compare the result to the goal, refine.

Concrete technique: When using VLMs for iterative generation, give the model access to execution feedback (run the code, show the output) rather than asking it to plan everything upfront. The observe-propose-execute-iterate loop is more robust than one-shot generation, especially for complex structured tasks where correctness is verifiable.