Canvas360: Teaching AI to Think in 360 Degrees

Paper: 2607.08765 Authors: Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi Categories: cs.CV

The Gap

Panoramic image generation is booming, but the field has a dirty secret: existing models treat equirectangular panoramas like flat images, then act surprised when the outputs look warped. Prior work like Diffusion360 and PanoDiffusion made strides in text-to-panorama generation, but they hit two walls. First, there’s no large-scale, high-quality dataset specifically designed for in-context panoramic tasks — meaning tasks where you give the model an input panorama plus instructions (like “replace the sky” or “extend the left side”). Second, standard diffusion architectures ignore the geometric reality of equirectangular projection: objects near the poles get stretched, left and right edges must seamlessly wrap around, and depth cues behave differently than in planar images.

Canvas360 attacks both problems simultaneously — a new dataset (Canvas360Dataset, 1M samples) and a geometry-aware architecture that bakes equirectangular physics into the model’s DNA.

Problem                     Assumption              Method                  Evidence              Conclusion
|                           |                       |                       |                     |
No large-scale              Geometry can be         Canvas360Dataset        1M paired samples     Unified in-context
paired panoramic            learned from flat       (1M pairs)              + parallel depth      panorama framework
training data for           images alone            + velocity circular     + circular padding    surpasses prior
in-context tasks            |                       padding                 + similarity loss     methods on fidelity
|                           |                       + similarity loss       |                     and task coverage
Models ignore               v                       regularization         v
equirectangular             Standard diffusion      |                       Improved FAED metric
geometry (wrap,             fails on 360-degree     v                       + competitive SOTA
stretch, depth)             content                 Geometry-aware          on standard metrics
                            |                       representations         |
                            v                       that capture distortion |
                            Poor geometric          and global coherence
                            consistency

The Increment

One sentence: Before this paper, panoramic generation had no dedicated in-context training corpus and treated 360-degree geometry as an afterthought; after, there’s a 1M-sample benchmark dataset and an architecture that learns equirectangular structure from the ground up.

Core Mechanism

Canvas360 operates in two stages. Stage one is geometry-aware pretraining: the model learns to generate panoramas while simultaneously predicting depth, but here’s the twist — it does this in parallel rather than sequentially. The depth branch and the RGB branch share the diffusion backbone, so the model is forced to internalize the relationship between 3D structure and 2D panoramic appearance.

The architecture introduces two key geometric operations. Velocity circular padding ensures that the diffusion process respects the spherical topology — when the model denoises pixels at the left edge, it “sees” what’s happening at the right edge (and vice versa), because in a real panorama those edges are physically adjacent. Similarity loss regularization penalizes the model when features at the equator and poles behave identically, pushing it to learn position-dependent representations that account for equirectangular stretching.

Stage two is task-specific fine-tuning via token-level concatenation. For any downstream task (style transfer, inpainting, outpainting, editing), the input context and target output are concatenated at the token level in a single sequence. The geometry-aware priors from stage one transfer cleanly because the model already understands panoramic structure — now it just needs task conditioning.

Stage 1: Geometry-Aware Pretraining
====================================
Input Panorama --> [Shared Diffusion Backbone]
                        |           |
                        v           v
               [RGB Branch]   [Depth Branch]
                        |           |
                        v           v
                  RGB Output    Depth Output
                        |           |
                        +-----+-----+
                              |
                    Similarity Loss (reg)
                    Circular Padding (wrap edges)

Stage 2: Task-Specific Fine-Tuning
====================================
[Input Tokens] ++ [Task Tokens] ++ [Target Tokens]
                    |
                    v
            [Frozen Backbone + LoRA]
                    |
                    v
            Generated Panorama

Think of it like training a cartographer. In stage one, you don’t just hand them a blank globe and say “paint it.” You make them simultaneously draw the terrain AND build a relief map, so they understand that mountains near the equator look different from mountains near the poles. You also stitch the left and right edges of their canvas together, so they can never cheat by treating the map as a rectangle. In stage two, you give them specific commissions: “redo this map in watercolor style,” “fill in the missing ocean section,” “extend the coastline.” Because they already understand how a globe works, each new task is just a variation — not a fundamental challenge. The circular padding is the stitched canvas edges; the parallel depth is the relief map; the similarity loss is the corrective feedback that says “no, that’s not how projection works near the poles.”

Key Concepts

• Equirectangular Projection: Imagine peeling an orange and flattening the peel onto a table. The center (equator) stays roughly proportional, but the top and bottom (poles) get horribly stretched. This is how panoramas are stored as flat images. Most image models don’t know this — they treat every pixel equally. Canvas360 teaches the model that a pixel near the pole “means” something different than a pixel at the equator. Concrete example: a person standing at the equator might be 100 pixels tall, but the same person standing near the pole would be stretched to 200 pixels — the model needs to understand this isn’t a height change, it’s a projection artifact.

• Velocity Circular Padding: In standard image generation, the model pads the edges with zeros or reflections. But a panorama’s left edge IS the right edge — they’re the same physical point. Velocity circular padding makes the diffusion process wrap around: noise schedules and denoising at column 0 directly see column W-1, and vice versa. This prevents the “seam” artifact that plagues most panoramic generators. It’s like the difference between painting on a flat canvas versus painting on the inside of a cylinder — the brush strokes at the edges must connect.

• In-context Generation: Rather than training separate models for each task, you give the model the input as context tokens and ask it to complete the output. It’s the same philosophy as language models doing few-shot learning — the “task” is specified by the example, not by a task-specific head. For panoramas, this means one model handles style transfer, inpainting, outpainting, and editing simply by changing what tokens you feed it.

Framework Shift

Before (mainstream approach):           After (this paper):
                                       
[Flat Image Model]                      [Geometry-Aware Backbone]
        |                                       |
        v                                       |
   Panorama Task? -- No special                |
   treatment for wrap/stretch    -->  [Circular Padding + Depth]
        |                                       |
        v                                       |
   Train separate model                 [Token Concatenation]
   per task (style, inpaint,                     |
   outpaint, edit)                               v
        |                                 Unified Model
        v                                 (all tasks)
   No shared dataset
   for in-context panorama

From task-specific flat-image hacks to a unified geometry-native panorama framework, the core shift is treating the sphere as a sphere, not a rectangle with inconvenient edges.

Expert Assessment

Problem choice: This is a real gap. Panoramic generation has lagged behind standard image generation precisely because the field lacked both proper training data and architectures that respect equirectangular geometry. The 1M-sample dataset alone is a meaningful contribution — prior work either used small curated sets or scraped low-quality data. The problem sits at the intersection of 3D vision and generative modeling, a sweet spot that’s been underserved.

Method maturity: The individual techniques (circular padding, parallel depth prediction, similarity loss) are incremental rather than revolutionary — each has appeared in some form in other contexts. But the assembly is thoughtful. The velocity circular padding is the cleverest piece; it’s a simple idea that solves a real, visible problem. The similarity loss feels more like a regularization trick than a fundamental insight. I worry that the “unified framework” claim is somewhat overstated — token-level concatenation is the standard in-context recipe, not a novel contribution.

Experimental integrity: The FAED metric (panorama-specific FID variant) is reported prominently, which is appropriate since standard FID doesn’t capture panoramic geometry. Baselines include relevant prior work. However, the paper would benefit from more ablation on the individual components — how much does circular padding alone contribute versus the depth branch versus the similarity loss? The numbers look solid but not jaw-dropping on standard metrics; the FAED improvement is where the story gets compelling.

Writing quality: The paper reads clearly but cuts corners in the related work section — prior panoramic generation methods are summarized rather than critically positioned. The dataset section could use more detail on quality filtering and diversity metrics. The method section is well-structured; the two-stage framing is clean and easy to follow. If I could rewrite one section, it would be the experiments: add ablations, show failure cases, and be more explicit about what FAED actually measures geometrically.

Verdict: weak accept — The dataset is genuinely useful, the geometric-awareness angle is well-motivated, but the technical novelty is assembled from known ingredients rather than introducing fundamentally new machinery.

Takeaways

Three concrete things worth stealing:

1. Circular padding for any periodic domain: If your data has wrap-around topology (panoramas, cylindrical scans, temporal cycles), circular padding in the diffusion process is a low-effort, high-impact fix. This transfers directly to any domain where boundaries are artificial.

2. Parallel auxiliary tasks as geometric regularizers: Forcing the model to predict depth alongside RGB during pretraining is a form of multi-task learning that encodes 3D structure without requiring 3D supervision at inference time. This idea transfers to medical imaging (predict segmentation alongside generation) or satellite imagery (predict elevation alongside appearance).

3. Build the dataset, own the problem: The 1M-sample Canvas360Dataset may be the most lasting contribution. In nascent subfields, creating a high-quality benchmark often matters more than any single model. If you’re working in a data-starved niche, investing in data curation can be higher-leverage than architectural innovation.