LaSSM: Efficient 3D Instance Segmentation with Local Aggregation and State Space Models

Paper: 2602.11007 Authors: Lei Yao, Yi Wang, Yawen Cui, Moyun Liu, Lap-Pui Chau Categories: cs.CV

Abstract

LaSSM addresses two critical bottlenecks in query-based 3D instance segmentation: inefficient query initialization and computationally expensive attention mechanisms. The method introduces a hierarchical semantic-spatial query initializer that leverages superpoints with both semantic and spatial information for comprehensive scene coverage. A novel coordinate-guided state space model (SSM) decoder then refines these queries through local aggregation and spatial dual-path processing. LaSSM achieves first place on ScanNet++ V2, outperforming previous methods by 2.5% mAP while using only one-third of the computational resources, demonstrating significant efficiency gains without sacrificing accuracy.

Key Contributions

• Hierarchical Semantic-Spatial Query Initializer: Derives query sets from superpoints by jointly considering semantic cues and spatial distribution, enabling better scene coverage and faster convergence compared to random or learned initialization strategies
• Coordinate-Guided SSM Decoder: Replaces computationally intensive attention mechanisms with state space models that progressively refine queries through local aggregation and spatial dual-path processing
• Local Aggregation Scheme: Restricts the model’s focus to geometrically coherent regions, preventing the incorporation of noisy information from distant or irrelevant points
• State-of-the-Art Efficiency: Achieves top performance on ScanNet++ V2 with 67% fewer FLOPs than previous best methods, while maintaining competitive results across multiple benchmarks

Technical Methodology

The LaSSM architecture consists of three main components working in sequence. First, the backbone network processes raw point clouds to extract multi-scale features. These features are then fed into the hierarchical semantic-spatial query initializer, which generates initial queries by analyzing superpoint representations. Unlike previous methods that use random sampling or learned embeddings, this initializer explicitly considers both the semantic content and spatial layout of the scene.

The core innovation lies in the coordinate-guided SSM decoder. Traditional query-based methods rely on cross-attention mechanisms that compute pairwise relationships between all queries and all points, leading to quadratic complexity. LaSSM replaces this with a more efficient approach: each decoder layer first performs local aggregation to gather information from spatially nearby points, then applies a spatial dual-path SSM block to capture dependencies within the query set.

The local aggregation operates by restricting each query’s receptive field to a geometrically coherent neighborhood, defined using coordinate information. This design choice is motivated by the observation that instance boundaries in 3D scenes typically correspond to spatial discontinuities. The spatial dual-path SSM block processes queries along two complementary paths—one focusing on spatial relationships and another on feature similarities—before merging the results.

Experimental Results

LaSSM demonstrates exceptional performance across multiple benchmarks. On ScanNet++ V2, it achieves 42.8% mAP, surpassing the previous best method by 2.5 points while requiring only 33% of the computational cost (measured in FLOPs). This efficiency gain is particularly significant for large-scale scene understanding applications where computational resources are constrained.

The method also shows strong generalization across different datasets. On ScanNet, it achieves 58.9% [email protected], competitive with state-of-the-art methods but with substantially lower computational requirements. Similar trends are observed on ScanNet200 (32.1% mAP), S3DIS (67.2% [email protected]), and ScanNet++ V1 (40.5% mAP), validating the approach’s robustness across varying scene complexities and annotation densities.

Ablation studies reveal that both the hierarchical query initializer and the SSM decoder contribute significantly to performance. Removing the semantic-spatial initialization reduces mAP by 3.2 points, while replacing the SSM decoder with standard attention decreases performance by 2.1 points while increasing FLOPs by 2.8x. The local aggregation scheme proves particularly important, with its removal causing a 1.8 point drop in mAP due to the incorporation of noisy information from distant regions.

Implications and Future Directions

LaSSM’s efficiency gains open new possibilities for real-time 3D scene understanding in resource-constrained environments such as mobile robotics and augmented reality applications. The 3x reduction in computational cost compared to attention-based methods makes it feasible to deploy sophisticated instance segmentation on edge devices without sacrificing accuracy.

The success of state space models in this domain suggests broader applicability beyond 3D vision. The coordinate-guided SSM architecture could potentially be adapted to other structured prediction tasks where spatial relationships play a crucial role, such as video instance segmentation or multi-object tracking.

However, several challenges remain. The method’s reliance on superpoint preprocessing introduces a dependency on the quality of initial oversegmentation. Future work could explore end-to-end learning of hierarchical representations or adaptive superpoint generation based on scene complexity. Additionally, while LaSSM excels at large-scale scenes, its performance on small objects or thin structures could be further improved through multi-scale processing or boundary refinement modules.

Takeaways

1. LaSSM achieves state-of-the-art 3D instance segmentation on ScanNet++ V2 (42.8% mAP) with 67% fewer FLOPs than previous methods, demonstrating that efficiency and accuracy are not mutually exclusive
2. Hierarchical semantic-spatial query initialization provides better scene coverage and faster convergence than random or learned initialization strategies by explicitly considering both semantic content and spatial distribution
3. State space models can effectively replace attention mechanisms in 3D vision tasks, offering linear complexity while maintaining the ability to capture long-range dependencies through coordinate-guided processing
4. Local aggregation schemes that restrict receptive fields to geometrically coherent regions prevent the incorporation of noisy information and improve instance boundary prediction
5. The method’s strong performance across multiple benchmarks (ScanNet, ScanNet200, S3DIS, ScanNet++ V1/V2) validates its generalization capability and robustness to varying scene complexities