Articraft: Programming Articulated 3D Assets at Scale with LLM Agents

Paper: 2605.15187 Authors: Matt Zhou, Ruining Li, Xiaoyang Lyu, Zhaomou Song, Zhening Huang, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi, Shangzhe Wu Categories: cs.CV, cs.GR, cs.RO

The Gap

Articulated 3D assets—objects with moving parts like doors, drawers, or robot arms—are essential for robotics simulation, VR, and training vision models. But existing datasets are tiny (hundreds of objects) and narrow (a few categories). Manual creation doesn’t scale. Generative models like text-to-3D struggle with articulation constraints: joints must align precisely, parts must not interpenetrate, motion ranges must be physically plausible. Prior work either generates static meshes (no articulation) or uses procedural generation with hand-coded rules (limited diversity, requires expert knowledge per category).

The core problem: articulated asset generation requires both geometric creativity and structural precision. Neural generators lack the latter; rule-based systems lack the former.

Problem: Need 10K+ diverse articulated assets
   |
   v
Existing approaches fail:
   |
   +---> Neural generators: creative but structurally invalid
   |
   +---> Rule-based: valid but narrow/manual
   |
   v
Key insight: Articulation = executable specification
   |
   v
Method: LLM writes programs in domain SDK
   |
   +---> SDK: high-level primitives (parts, joints, tests)
   |
   +---> Harness: validates, returns structured feedback
   |
   v
Evidence: 10K assets, 245 categories, outperforms baselines
   |
   v
Conclusion: Code generation > direct generation for structured 3D

The Increment

One sentence: Before this paper, generating diverse articulated 3D assets required either manual modeling or narrow procedural rules; after, an LLM agent writes programs against a domain SDK to produce 10K+ validated assets across 245 categories.

Core Mechanism

Articraft reframes articulated asset generation as a code generation problem. Instead of asking an LLM to output mesh coordinates or joint parameters directly, it asks the LLM to write a Python program that constructs the asset using a domain-specific SDK. The SDK provides high-level primitives: add_part() to define geometry, add_joint() to specify articulation, add_test() to validate constraints. The LLM writes a program, the harness executes it in a sandboxed environment, validates the resulting asset (collision checks, joint limits, kinematic feasibility), and returns structured feedback. If validation fails, the LLM revises the program based on the feedback. This loop continues until the asset passes all checks or a retry limit is reached.

The SDK abstracts away low-level details like URDF syntax or mesh file management. The LLM operates at the level of “add a rectangular base, attach a cylindrical leg with a revolute joint at this position.” The harness handles rendering, physics validation, and error reporting. Crucially, the LLM can write tests as part of the program—assertions about joint ranges, collision-free motion, or part dimensions—which the harness executes to catch errors early.

User prompt: "Generate a desk lamp"
   |
   v
LLM writes program:
   base = add_part("cylinder", ...)
   arm = add_part("box", ...)
   joint = add_joint("revolute", base, arm, ...)
   add_test("arm rotates 180 degrees without collision")
   |
   v
Harness executes program:
   |
   +---> Geometry instantiation
   |
   +---> Joint validation (limits, axes)
   |
   +---> Collision checking
   |
   +---> Test execution
   |
   v
Validation result:
   |
   +---> Pass: return asset (URDF + meshes)
   |
   +---> Fail: return structured error
             |
             v
          LLM revises program (loop)

Think of Articraft like a construction site with a foreman and inspector. The LLM is the foreman who reads blueprints (the user prompt) and writes work orders (programs) in a standardized format. The SDK is the set of prefabricated components—beams, joints, fasteners—that the foreman can specify without worrying about raw materials. The harness is the inspector who checks whether the assembled structure meets building codes (no collisions, joints work, tests pass) and issues violation reports if not. The foreman revises the work order based on the inspector’s feedback until the structure passes inspection. The key insight: by restricting the foreman to prefab components and giving the inspector clear checklists, you get both creativity (the foreman can design novel structures) and reliability (the inspector ensures they’re buildable).

Key Concepts

• Domain-specific SDK: A programming interface tailored to a specific problem domain, exposing high-level operations that encapsulate complex low-level details. In Articraft, the SDK provides functions like add_part(shape, dimensions, material) and add_joint(type, parent, child, axis, limits). This is different from a general-purpose 3D library (like Blender’s Python API) because it bakes in domain knowledge: joints automatically handle coordinate frame transformations, parts have semantic labels (base, handle, lid), and the SDK enforces constraints (e.g., a revolute joint requires an axis). The benefit: the LLM doesn’t need to know how to compute rotation matrices or write URDF XML; it just calls add_joint("revolute", ...) and the SDK handles the rest. Analogy: instead of giving someone a hammer and nails (low-level tools), you give them IKEA furniture with labeled parts and an instruction manual (high-level interface).

• Structured feedback loop: A validation-and-revision cycle where errors are reported in a machine-readable format that guides the next iteration. When Articraft’s harness detects a collision, it doesn’t just say “collision detected”; it returns \{"error": "collision", "parts": ["arm", "base"], "joint_state": \{...\}\}. The LLM uses this structured data to diagnose the issue (the arm is too long, or the joint axis is wrong) and revise the program accordingly. This is more effective than natural language feedback (“something is wrong with the arm”) because it’s precise and actionable. The loop terminates when all validation checks pass or a retry limit is hit. This pattern—generate, validate, revise—is common in program synthesis but novel in 3D asset generation, where most systems generate once and hope for the best.

• Programmatic asset representation: Representing a 3D asset as executable code rather than as data (mesh files, parameter vectors). A desk lamp isn’t stored as vertices and faces; it’s stored as a Python function that, when executed, constructs the lamp by calling SDK functions. This has two advantages: (1) Composability: you can parameterize the program (lamp height, arm length) and generate variations by changing arguments. (2) Interpretability: you can read the program to understand the asset’s structure, which is harder with a raw mesh. The tradeoff: programs are less compact than meshes and require an execution environment. But for generation, programs are a better intermediate representation because they expose the generative process, making it easier to debug and refine.

Framework Shift

Before (neural/procedural generation):

User prompt --> [Generator] --> 3D asset
                    |
                    v
              (black box)
              - Neural: creative but invalid
              - Procedural: valid but narrow

After (Articraft):

User prompt --> [LLM] --> Program (SDK calls)
                           |
                           v
                      [Harness]
                           |
                           +---> Execute program
                           |
                           +---> Validate (geometry, physics, tests)
                           |
                           +---> Structured feedback
                           |
                           v
                      Valid asset <-- (loop if invalid)

From end-to-end generation to generate-validate-revise with a domain interface, the core shift is treating asset creation as a programming task where the LLM writes against a constrained API and an automated inspector ensures correctness.

Expert Assessment

Problem choice: Real and well-motivated. Articulated assets are a genuine bottleneck in robotics and embodied AI—existing datasets are orders of magnitude too small. The problem sits at the intersection of 3D vision, robotics, and program synthesis, which is a productive place to be. Not manufactured.

Method maturity: Clever insight (code generation for structured 3D) executed with solid engineering. The SDK design is the key contribution—it’s not trivial to identify the right abstraction level. Too low-level (raw URDF) and the LLM drowns in details; too high-level (just category names) and you lose diversity. The harness with structured feedback is well-designed. However, the method is somewhat brute-force: it relies on the LLM’s general coding ability rather than learning a 3D-specific prior. A hybrid approach (LLM + learned geometry module) might be more sample-efficient.

Experimental integrity: Baselines are fair (PartNet-Mobility, other procedural generators, general-purpose coding agents like GPT-4 with raw URDF). The human evaluation (quality, diversity, validity) is appropriate since there’s no ground truth for “good articulated asset.” The 10K dataset is impressive in scale. One concern: the paper doesn’t deeply analyze failure modes—what categories does Articraft struggle with? Are there systematic biases (e.g., overrepresentation of simple hinge joints)? The ablation on SDK design is useful but could be more thorough (e.g., impact of test-writing capability).

Writing quality: Clear and well-structured. The SDK design section is excellent. The related work could be tighter—it spends too much time on general text-to-3D methods that aren’t directly comparable. The results section would benefit from more failure analysis and a breakdown of asset complexity (number of parts, joint types) across categories. The supplementary material is thorough.

Verdict: strong accept — Addresses a real problem with a novel and effective method, produces a valuable dataset, and opens a new direction (LLM-based procedural generation with domain SDKs) that will likely inspire follow-up work.

Takeaways

For practitioners building generation systems: The SDK + harness pattern is broadly applicable. If you’re generating structured outputs (code, configs, 3D models, circuits), don’t ask the LLM to produce the raw format directly. Design a domain-specific interface that (1) abstracts low-level details, (2) encodes domain constraints, and (3) enables programmatic validation. Then wrap it in a harness that executes, validates, and returns structured feedback. This is more reliable than end-to-end generation and more scalable than hand-coded rules.

For robotics researchers: Articraft-10K is immediately useful for training manipulation policies, testing sim-to-real transfer, and benchmarking perception systems. The diversity (245 categories) is a step change from PartNet-Mobility (few dozen categories). The programmatic representation means you can generate variations on-the-fly for curriculum learning or domain randomization.

For 3D vision researchers: The result suggests that code is a better intermediate representation than geometry for generation tasks requiring structural constraints. This likely applies beyond articulated assets—architectural layouts, mechanical assemblies, molecular structures. The key is identifying the right SDK primitives for your domain.

Specific technique to steal: The test-writing capability. Letting the LLM write assertions as part of the generation program (e.g., “joint rotates 90 degrees,” “no part exceeds bounding box”) catches errors early and makes the validation loop more efficient. This is underused in program synthesis and could apply to other domains (data pipelines, config generation).