PAR3D: Teaching 3D Vision Models to See Parts, Not Just Objects

Paper: 2606.06485 Authors: Shaohui Dai, Yansong Qu, You Shen, Shengchuan Zhang, Liujuan Cao Categories: cs.CV

The Gap

Existing 3D multimodal large language models (3D-MLLMs) handle vision-language tasks like VQA, captioning, and referring segmentation. But they’re stuck at the object level — they see “chair” but not “chair leg” or “armrest”. This matters because robots interacting with environments need to reason about parts: “grab the door handle” not “grab the door”, “sit on the seat cushion” not “sit on the chair”.

Prior work (3D-LLM, Chat-3D v2, LEO) unified object-level scene understanding but ignored part-level structure. The bottleneck: no training data with part annotations, and representations optimized for whole-object semantics can’t distinguish fine-grained components.

Problem: 3D-MLLMs can't ground parts
    |
    v
Assumption: Part-aware representations + hierarchical queries can model object-part structure
    |
    v
Method: Part-Aware 3D Repr Learning + Hierarchical Segmentation Query Gen
    |
    v
Evidence: ScenePart dataset shows +12% part-level accuracy over baselines
    |
    v
Conclusion: Unified model handles both object and part tasks without degrading object performance

The Increment

One sentence: Before this paper, 3D-MLLMs saw scenes as collections of whole objects; after, they parse scenes hierarchically into objects and their constituent parts.

Core Mechanism

PAR3D injects part-level awareness into a 3D-MLLM through two mechanisms working in tandem. First, Part-Aware 3D Representation Learning enriches the vision encoder by training it on synthetic scenes where each object is annotated with part-level labels (e.g., a table has legs, top, drawer). This pretraining teaches the feature extractor to distinguish part boundaries inside objects. Second, Hierarchical Segmentation Query Generation creates a two-level query system: when asked to segment “the drawer”, it first generates an object query to locate the table, then spawns part queries within that object’s region to isolate the drawer.

Input: 3D Scene Point Cloud + Text Query ("segment the chair leg")
    |
    v
[Part-Aware Vision Encoder] ---> Part-enriched 3D features
    |                              (points know their part identity)
    v
[LLM Backbone] ---> Generates object query token + part query token
    |
    v
[Hierarchical Query Decoder]
    Object Query: locates "chair" region
         |
         v
    Part Queries: search within chair region for "leg"
    |
    v
Output: 3D mask highlighting chair leg points

Think of it like a warehouse robot finding a specific screw. The old system (object-centric) gets you to the right shelf (“find the toolbox”). The new system (part-aware) opens the toolbox and locates the Phillips head screws in compartment 3. The part-aware encoder is like labeling every compartment during warehouse setup. The hierarchical queries are like a two-step search: “which toolbox?” then “which compartment inside?”

The metaphor does real work: just as the robot can’t search compartments without first opening the right toolbox, PAR3D can’t generate part queries without first grounding the parent object. And just as pre-labeled compartments speed up search, part-aware features let the decoder directly attend to part boundaries instead of relearning them from scratch.

Key Concepts

• Part-Aware Representation: Imagine training an image classifier that sees cats. A standard encoder learns “furry, pointy ears, whiskers” as a blob. A part-aware encoder learns “left ear at position X, right ear at Y, tail at Z”. For 3D scenes, this means each point feature encodes not just “this belongs to a chair” but “this belongs to a chair leg, specifically the front-left leg”. Technically, during pretraining, the model solves a part segmentation task on synthetic data where every object is decomposed into semantic parts (ShapeNet parts + procedural scenes). The loss forces features to cluster by part identity, so the learned representation naturally separates part boundaries. When you later ask it to segment “table leg”, the features already have leg-ness baked in, rather than having to infer it purely from language grounding.

• Hierarchical Query Generation: In standard referring segmentation, a model generates a single query vector to attend to target pixels. PAR3D generates a tree of queries: a coarse object query and several fine-grained part queries. Think of it as a spotlight that first lights up the entire chair (object query), then within that illuminated region, smaller spotlights pick out the armrest, seat, backrest (part queries). Crucially, part queries are conditioned on the object query’s output — they only search within the spatial region the object query activates. This is implemented as a two-stage decoder: stage 1 runs object queries through cross-attention with scene features, stage 2 spawns part queries that cross-attend only to features within the object query’s top-K activated regions. This hierarchy handles the compositional structure: “left armrest” is defined relative to “chair”, not relative to the entire scene.

• ScenePart Dataset: A synthetic training corpus addressing the data scarcity problem. Real-world 3D scans rarely have part annotations (too expensive to label manually). ScenePart procedurally generates scenes by placing ShapeNet objects (which have part annotations) into room layouts, then synthesizes language instructions like “segment the drawer in the desk on the left”. It’s 10K scenes with 50K part-level referring expressions. This is the training fuel that makes part-aware learning feasible without massive annotation cost. The tradeoff: synthetic data has limited visual diversity (clean geometry, no clutter), so the model must generalize from synthetic pretraining to real-world scenes. The paper shows this sim-to-real transfer works reasonably well but leaves room for domain adaptation.

Framework Shift

Before (object-centric 3D-MLLMs):        After (PAR3D):

Scene ---> [3D Encoder]                  Scene ---> [Part-Aware Encoder]
             |                                        |
             v                                        v
        Object Features                          Object + Part Features
             |                                        |
             v                                        v
        [LLM] ---> Query                        [LLM] ---> Object Query
             |                                        |     +
             v                                        v     Part Queries
        [Decoder]                                [Hierarchical Decoder]
             |                                        |    /  |  \
             v                                        v   v   v   v
        Object Mask                           Object Mask + Part Masks

Flat representation, single-level query    Hierarchical representation, tree of queries

One sentence: From flat object features processed by single-level queries, to hierarchical part-enriched features processed by a tree of object and part queries.

Expert Assessment

Problem choice: Real gap. Embodied AI genuinely needs part-level grounding — a robot can’t “open the drawer” if it can’t distinguish the drawer from the desk body. Prior work’s object-centricity was a known limitation (acknowledged in LEO, Chat-3D v2 papers). This isn’t manufactured; it’s the next logical step in 3D scene understanding’s progression from detection → segmentation → part decomposition.

Method maturity: Solid engineering rather than deep insight. Part-aware pretraining is straightforward transfer of 2D part segmentation techniques (PartImageNet) to 3D. Hierarchical queries are a clean adaptation of cascaded detection (Cascade R-CNN). The novelty is integration, not invention. That said, integration is non-trivial — the hierarchical query design is elegant, and the part-aware loss is well-designed. No simpler approach being overlooked; you can’t skip the pretraining or hierarchy without performance collapse (ablations confirm this).

Experimental integrity: Baselines are fair but not exhaustive. Compares against 3D-LLM, Chat-3D v2, LEO on their own benchmarks, then introduces ScenePart for part-level eval. Missing: comparison with specialized part segmentation methods (PartSLIP, PartDistill) adapted to 3D, and evaluation on real-world part-annotated data (PartNet, AKB-48). The synthetic-to-real gap is acknowledged but not deeply probed. Ablations are thorough (each component justified). Numbers look clean: +12% part-level accuracy, +5% object-level, no degradation on standard benchmarks. Red flag: ScenePart test set comes from same synthetic distribution as training data, so generalization to real scenes is underexplored.

Writing quality: Section 3.2 (method) is dense — the hierarchical query generation description conflates architecture and training procedure, making it hard to parse on first read. Figure 2’s diagram doesn’t clearly show the two-stage query flow. The ablation study (Section 4.3) is excellent and could be promoted to main body. Related work undersells the connection to 2D part segmentation literature (PartImageNet, Pascal-Part). Rewriting Section 3.2 with clearer stage separation and a better diagram would elevate clarity significantly.

Verdict: Weak accept — competent execution of a real problem, but incremental rather than transformative. It’s the “next step” work that moves the field forward without redefining it. The ScenePart dataset is a contribution that outlives the method itself. If you’re building embodied AI systems, you’ll want to read this. If you’re researching vision-language foundations, you can wait for the follow-ups.

Takeaways

Hierarchical query design: The two-stage query mechanism (coarse object query → spatially-conditioned part queries) is a reusable pattern. Apply it anywhere you have nested structure: document layout analysis (page → section → paragraph), biological imaging (organ → tissue → cell), UI understanding (app → widget → subcomponent). The key insight: condition fine-grained queries on coarse query outputs to constrain search space.

Synthetic data as pretraining fuel: When annotation cost is prohibitive, procedural generation of training data (like ScenePart) can bootstrap models that later fine-tune on real data. Transferable to other domains: medical imaging (synthetic lesions), robotics (simulated manipulation), CAD (procedural assemblies).

Part-aware pretraining loss: Adding a part segmentation auxiliary task during vision encoder pretraining improves feature granularity without architecture changes. Cheap way to boost fine-grained recognition in any vision model. Directly transferable to 2D segmentation, video understanding, or any task where objects have compositional structure.