TideGS: Training Billion-Scale 3D Gaussian Splatting via Out-of-Core Optimization

Paper: 2605.20150 Authors: Chonghao Zhong, Linfeng Shi, Hua Chen, Tiecheng Sun, Hao Zhao, Binhang Yuan, Chaojian Li Categories: cs.CV, cs.PF

The Gap

3D Gaussian Splatting (3DGS) represents scenes as millions of 3D Gaussian primitives, each carrying position, color, opacity, and covariance parameters. Prior systems hit a hard wall: commodity GPUs with 24GB memory can only hold ~11 million Gaussians in-memory, while large-scale scenes (city blocks, building interiors) demand hundreds of millions or billions of primitives for photorealistic quality. Existing out-of-core approaches (GigaGS) barely reach 100M Gaussians and suffer from naive parameter swapping that ignores 3DGS training’s spatial and temporal structure.

The core insight: 3DGS training is sparse and trajectory-conditioned. Each camera viewpoint activates only a small fraction of Gaussians (those visible in the frustum). GPU memory doesn’t need to hold all parameters—it can serve as a working-set cache.

Problem: Billion-scale 3DGS training
   |
   v
Observation: Training is sparse (per-view activation) + trajectory-structured
   |
   v
Assumption: GPU memory = working-set cache, not parameter store
   |
   v
Method: SSD-CPU-GPU hierarchy + spatial blocking + trajectory-adaptive streaming
   |
   v
Evidence: 1B+ Gaussians on 24GB GPU, best quality among single-GPU baselines
   |
   v
Conclusion: Out-of-core 3DGS training is viable at billion-primitive scale

The Increment

One sentence: Before TideGS, single-GPU 3DGS training was limited to ~11M Gaussians in-memory or ~100M with naive out-of-core swapping; after TideGS, practitioners can train 1B+ Gaussians on commodity hardware by exploiting training sparsity and camera trajectory structure.

Core Mechanism

TideGS organizes the parameter table as a three-level hierarchy: SSD (persistent storage) → CPU DRAM (staging buffer) → GPU VRAM (working set). The key is block-virtualized geometry: the scene is partitioned into spatial blocks aligned with SSD page boundaries (e.g., 4KB). Each block contains Gaussians within a 3D bounding box. During training, only blocks intersecting the current camera frustum are loaded into GPU memory.

The system runs a hierarchical asynchronous pipeline with three stages operating concurrently: (1) SSD-to-CPU prefetching loads predicted blocks into CPU memory, (2) CPU-to-GPU transfer moves active blocks to GPU, (3) GPU computes gradients and updates parameters. Stages overlap via double-buffering: while the GPU trains on iteration *t, the CPU prepares data for iteration t+1, and the SSD prefetches for t+2.

The breakthrough is trajectory-adaptive differential streaming. Instead of transferring entire blocks every iteration, TideGS tracks which blocks were active in the previous iteration and computes the delta: new blocks entering the frustum are loaded, old blocks exiting are evicted. For camera trajectories with high temporal coherence (common in real captures), the working-set delta is small—often 10-20% of the full working set—dramatically reducing I/O bandwidth.

Iteration t-1:  [Blocks A B C D] in GPU
                      |
                      v
Iteration t:    Camera moves slightly
                      |
                      v
                Frustum now covers [B C D E]
                      |
                      v
                Delta: Evict A, Load E
                      |
                      v
                Transfer only E (not B C D E)

Structural metaphor: Think of TideGS as a library with a reading room. The SSD is the archive (millions of books), CPU DRAM is the librarian’s cart (staging area), and GPU VRAM is the reading room (limited desks). A researcher (camera) requests books (Gaussians) based on their current topic (viewpoint). The librarian predicts the next topic from the research trajectory and pre-fetches relevant books. Crucially, when the researcher shifts topics slightly, the librarian doesn’t clear the entire reading room—they only swap out books no longer needed and bring in new ones. The reading room stays mostly full, and the researcher never waits.

Key Concepts

• Block-virtualized geometry: Instead of storing Gaussians as a flat array, TideGS groups them into spatial blocks (e.g., 8×8×8 meter cubes). Each block is a contiguous chunk in storage, aligned with SSD page size. This transforms random Gaussian access into sequential block I/O, exploiting SSD’s high sequential bandwidth (~3GB/s) while avoiding random-access penalties (~100MB/s). When a camera frustum intersects a block’s bounding box, the entire block is loaded. This trades some memory overhead (loading invisible Gaussians within active blocks) for massive I/O speedup.

• Trajectory-adaptive differential streaming: Camera trajectories in 3DGS training exhibit temporal coherence—consecutive frames often overlap 70-90% in visible Gaussians. TideGS maintains a working-set tracker that records which blocks were active in iteration *t-1. At iteration t, it computes the frustum-block intersection and diffs against the previous working set. Only the delta (new blocks minus evicted blocks) is transferred. For a smooth trajectory, this reduces per-iteration I/O from ~500MB (full working set) to ~50MB (10% delta), keeping the pipeline saturated without stalling the GPU.

• Hierarchical asynchronous pipeline: Training, data transfer, and prefetching happen in parallel across three hardware tiers. The GPU trains on iteration *t while the CPU prepares iteration t+1 and the SSD prefetches iteration t+2. This requires careful synchronization: the CPU must finish staging before the GPU needs the data, and the SSD must prefetch before the CPU requests it. TideGS uses a trajectory predictor (simple linear extrapolation of camera motion) to guess future frustums and issue prefetch commands early. If the prediction is wrong, the system falls back to on-demand loading, but correct predictions (>80% hit rate in experiments) hide I/O latency entirely.

Framework Shift

Before (in-memory 3DGS):              After (TideGS):

  All Gaussians in GPU VRAM            SSD: Full parameter table
         |                                  |
         v                                  v (prefetch)
  Train on visible subset              CPU: Staging buffer
         |                                  |
         v                                  v (delta transfer)
  Update all parameters                GPU: Working set only
                                            |
                                            v
                                       Train on visible subset
                                            |
                                            v
                                       Update working set
                                            |
                                            v (writeback delta)
                                       Sync to SSD/CPU

One sentence: From “fit all parameters in GPU memory” to “stream working sets through a memory hierarchy,” the core shift is treating GPU VRAM as a cache rather than a parameter store.

Expert Assessment

Problem choice: This is a real gap. 3DGS has proven itself for high-quality novel view synthesis, but scaling to large scenes (city-scale, building interiors) is blocked by memory. The problem sits at the intersection of graphics and systems—prior work focused on algorithmic improvements (better Gaussian initialization, adaptive densification) but ignored the memory bottleneck. TideGS addresses the right constraint at the right time.

Method maturity: The core insight (training sparsity + trajectory structure) is elegant, not brute force. However, the execution leans heavily on engineering: block alignment, prefetch heuristics, double-buffering. The trajectory predictor is simplistic (linear extrapolation)—more sophisticated predictors (LSTM, attention over past frames) could improve hit rates. The paper doesn’t explore failure modes: what happens with erratic camera motion or scenes with poor spatial locality? The method is mature enough for smooth trajectories but brittle for adversarial cases.

Experimental integrity: Baselines are fair. The paper compares against standard 3DGS (in-memory), GigaGS (prior out-of-core), and ablations of TideGS components. The scenes (Mip-NeRF 360, Tanks and Temples) are standard benchmarks. Numbers are credible: 1.2B Gaussians on 24GB GPU, 28.5 PSNR on a large scene. One red flag: the paper doesn’t report training time. Out-of-core systems are slower than in-memory—by how much? The omission suggests the overhead is non-trivial (likely 2-3× slower based on I/O bandwidth). Also, the comparison with GigaGS is slightly unfair—GigaGS wasn’t designed for billion-scale, so beating it by 10× isn’t shocking.

Writing quality: The paper is well-structured but verbose. Section 3 (method) could be condensed by 30%—too much space on obvious details (e.g., explaining what a frustum is). The ablation study (Section 4.3) is strong, but the main results (Section 4.2) bury the lead: the quality-vs-scale tradeoff should be a single figure, not scattered across tables. The related work section is thorough but reads like a literature dump. Rewriting Section 4.2 to lead with a single “scaling curve” figure (Gaussians vs PSNR vs memory) would elevate the paper significantly.

Verdict: Weak accept — Solid systems contribution with clear impact, but the method is more engineering than insight, and the evaluation omits key metrics (training time, failure modes). The work will be cited for enabling billion-scale 3DGS, but it’s not a paradigm shift.

Takeaways

For practitioners building large-scale ML systems: The trajectory-adaptive differential streaming idea transfers directly to any training loop with temporal coherence. If your model processes sequences (video, time-series, RL trajectories) and the working set changes slowly between steps, track the delta and transfer only incremental changes. This applies to distributed training (reduce all-reduce bandwidth), model serving (cache embeddings), and data pipelines (prefetch next batch based on trajectory).

For 3DGS researchers: Block-virtualized geometry is a clean abstraction. Even if you’re not doing out-of-core training, organizing Gaussians into spatial blocks improves cache locality during rendering and simplifies frustum culling. The paper’s spatial partitioning scheme (octree-based blocking) is reusable.

For systems researchers: The hierarchical asynchronous pipeline is textbook design—nothing novel, but the execution is clean. The key lesson: when building out-of-core systems, exploit domain structure (sparsity, locality, predictability) rather than treating the problem as generic parameter swapping. TideGS succeeds because it’s tailored to 3DGS training, not because it’s a general-purpose out-of-core framework.