RoPEMover: Depth-Aware Object Relocation via Positional Embeddings

Paper: 2606.27332 Authors: Ipek Oztas, Duygu Ceylan, Aybars Bugra Aksoy, Aysegul Dundar Categories: cs.CV

The Gap

Single-image object relocation requires more than simple inpainting: you must handle occlusions, reveal previously hidden regions, and update scene effects like shadows and reflections.
Existing approaches — GAN-based image editing, diffusion inpainting, or object-aware compositing — either treat the task as 2D cut-and-paste or learn a global warp, which destroys scene-level consistency. They often produce blurry boundaries, unrealistic shadows, or objects that float rather than sit in the scene.
This paper identifies a root cause: prior methods do not explicitly model the geometry of the displacement. The authors notice that diffusion transformers (DiT) use rotary positional embeddings (RoPE) to encode spatial layout. If you can manipulate those embeddings in a depth-aware way, you can induce controlled, geometry-consistent motion without retraining the entire model.

[Single-image object relocation]
           |
           v
+-------------------------------------+
| Prior: GAN / inpainting / warping   |
| fail on shadows, occlusions, depth  |
+-------------------------------------+
           |
           v
+------------------------------------------+
| Key insight: RoPE in DiT defines a       |
| structured spatial field (2D grid)        |
+------------------------------------------+
           |
           v
+----------------------------------------------+
| Our step: extend RoPE with depth => 3D       |
| spatial field; move object by shifting its   |
| RoPE coordinates                             |
+----------------------------------------------+
           |
           v
+---------------------------------------------+
| Evidence: SOTA on benchmarks (identity,     |
| shadow, region generation) with minimal     |
| real data (synthetic + parameter-efficient  |
| fine-tuning)                                |
+---------------------------------------------+
           |
           v
+---------------------------------------------+
| Conclusion: explicit RoPE manipulation      |
| enables scene-aware object relocation       |
+---------------------------------------------+

The Increment

One sentence: Before this paper, moving an object in a single image meant hoping the model would infer the right geometry; after this paper, you can explicitly tell the model where to place the object in 3D space by editing its positional embeddings, guaranteeing consistency.

Core Mechanism

The method builds on a diffusion transformer (DiT) that uses Rotary Position Embeddings (RoPE) in its attention layers. RoPE assigns a 2D position to each image patch via rotation matrices — think of it as a coordinate grid embedded in the model’s internal activations.

The core pipeline is:

1. Input: single image + object mask + target displacement (vector or depth map).
2. Depth estimation: a lightweight depth predictor extracts a per-pixel depth map.
3. Compute original RoPE: for each patch, the standard 2D RoPE is computed from its (x, y) coordinates.
4. Depth-aware extension: the depth value is used to modulate the rotation angles — shallow patches get a different rotation than deep ones, encoding 3D structure.
5. Relocation: for patches belonging to the object, the 2D (x, y) is replaced by the target location, while depth is kept consistent (or updated if the object moves closer/farther). The new RoPE is computed for those patches.
6. Inject into DiT: the attention layers use the modified RoPE. Patches with changed RoPE cause the model to interpret them as being at the new location, while the background patches retain their original embeddings.
7. Denoising: the diffusion process generates the output image, where the object appears at the new location, occlusions are filled with plausible content (from the background patches that were previously blocked), and shadows/illumination are updated because the 3D positions changed.

Input image + mask + depth (estimated)
         |
         v
+--------------------+
| Compute 2D RoPE    |
| for all patches    |
+--------+-----------+
         |
         v
+----------------------------+
| For object patches:        |
|   - new (x', y') = old +   |
|     displacement            |
|   - depth-aware RoPE:      |
|     rotate angle *= f(depth)|
+----------------------------+
         |
         v
+--------------------------------+
| Replace original RoPE in DiT's |
| attention layers                |
+--------------------------------+
         |
         v
+--------------------------------+
| Run diffusion denoising        |
| (parameter-efficient fine-tuned|
| on synthetic + few real images)|
+--------------------------------+
         |
         v
+--------------------------------+
| Output: relocated object       |
| with coherent shadows /        |
| revealed regions               |
+--------------------------------+

Structural metaphor: Think of a stage play with movable props.

• Stage grid = RoPE: every spot on stage has a unique coordinate (row, column). Actors (image patches) know where they stand by reading their coordinates.
• Depth dimension = distance from the audience: a prop that moves upstage shrinks in relative size and interacts differently with light.
• Our edit = we tell a specific actor “you now stand at seat D-5 instead of B-3”, and also adjust their “depth tag” to match the new distance.
• Diffusion model = the stage crew that, given these instructions, rebuilds the entire scene around the moved prop – shifting shadows, revealing what was behind it, and ensuring the lighting matches the new position.

Without depth, the crew would plop D-5 onto a flat grid, ignoring whether the prop should be smaller or cast shadows differently. With depth-aware RoPE, the crew—through the pre-trained model’s statistics—makes the scene geometrically plausible.

Key Concepts

• Rotary Position Embedding (RoPE): A way to encode positions in transformer attention by rotating queries and keys with a position-dependent angle. The rotation preserves the *relative distance between tokens: two tokens that are 5 positions apart always have the same rotation difference, regardless of absolute location. This lets attention “know” where things are relative to each other. In images, RoPE forms a 2D grid – each patch gets a unique combination of x-rotation and y-rotation. Manipulating these rotations is equivalent to relocating the patch’s “identity” in the spatial field.

• Depth-aware RoPE: Standard RoPE is 2D, but the real world is 3D. The paper adds a depth modulation: the rotation angle for each patch is multiplied by a function of its depth value. Shallow objects (higher depth value) have larger effective rotations, mimicking perspective foreshortening. This ensures that when you move an object, the model treats its size, shadow, and occlusion boundary according to its 3D position, not just a 2D shift.

• Parameter-efficient fine-tuning: Instead of retraining the entire DiT model from scratch (costly and data-hungry), the authors freeze most weights and only adapt a small set of parameters—e.g., low-rank adapters (LoRA) or specific attention projections. This allows the model to learn the depth-aware RoPE manipulation without forgetting its general image generation abilities, while needing only a small dataset of synthetic (rendered) objects plus a handful of real images for fine-tuning.

Framework Shift

Before (mainstream approach):          After (this paper):

[Input image]                          [Input image]
         |                                      |
         v                                      v
[GAN / inpainting model]            [Depth estimator + RoPE calculator]
         |                                      |
         v                                      v
[Object moved, but often              [Compute depth-aware RoPE]
  unrealistic shadows,                         |
  object blends poorly]                       v
                                       [Modify RoPE for object patches]
                                                |
                                                v
                                       [Parameter-efficient DiT with modified RoPE]
                                                |
                                                v
                                       [Output: geometry-consistent result]

One sentence: From *implicitly learning the transformation from data to explicitly editing the spatial coordinate system inside a pretrained transformer, the core shift is leveraging the model’s own positional encoding as an actionable knob for 3D-aware editing.

Expert Assessment

Problem choice: Real gap. Single-image object relocation is a long-standing and practical challenge in photo editing, content creation, and advertising. Existing SOTA (e.g., object-aware inpainting with depth guidance) still fails on consistent shadows and occlusion. The paper identifies a clear bottleneck — lack of explicit spatial manipulation in diffusion-based editing — and hits it directly.

Method maturity: Clever insight, not brute force. The idea of manipulating RoPE is elegant because it reuses the model’s internal representation instead of adding ad‑hoc modules. However, it relies on a separate depth estimator, which adds one more source of error. Also, the synthetic training data (likely rendered 3D objects on backgrounds) may cause a domain gap. The parameter-efficient fine-tuning is wise but the method’s robustness to out-of-distribution scenes is not fully validated in the abstract.

Experimental integrity: The paper claims SOTA across all metrics on standard benchmarks (the abstract mentions object motion benchmarks). If the baselines include recent methods like ObjectStitch, LaMa, or Stable Diffusion + ControlNet, and the numbers hold, then the experiments are solid. One red flag: the phrase “parameter-efficient fine-tuning” often means the comparison is not apple-to-apple — they might not compare with fully fine-tuned versions of the same model. Need to check the paper details.

Writing quality: The abstract is dense but clear. However, they mention “extend 2D RoPE into a depth-aware formulation” without explaining the *formula or training procedure — likely cut for space. A rewritten experimental section that walks through an example failure case from baselines and shows how RoPEMover handles it would elevate the paper. Also, a diagram of the depth-aware RoPE computation (matrix or rotation math) is missing.

Verdict: Strong accept — The core concept is original, well-motivated, and produces visible improvements. The paper opens up a new family of spatial editing methods by showing how to directly manipulate position encodings in diffusion models. The synthetic data limitation is acceptable for a first demonstration.

Takeaways

• Explicit coordinate editing in transformers: If you’re working with any transformer that uses positional embeddings (ViT, DiT, etc.), you can treat those embeddings as a control signal. Replace the positions of certain tokens to move or reorder them — this applies far beyond object relocation, e.g., image layout generation, video frame interpolation, or 3D scene manipulation.
• Depth-aware RoPE design: The idea of modulating a position encoding by depth (or any continuous spatial variable) is general. You can extend it to optical flow, disparity, or semantic height for other editing tasks.
• Synthetic + small real fine-tuning pipeline: The paper demonstrates that with a handful of real images (likely <100) and thousands of synthetic renders, you can teach a large diffusion model a new spatial skill. This recipe of “synthetic ground truth + LoRA fine-tuning” is directly transferable to any geometry-aware image task (e.g., insertion, removal, scaling).
• Shadow consistency as a byproduct of 3D-aware position: The method doesn’t explicitly model shadows — it emerges because the model, seeing the object at a new 3D coordinate, generates correct lighting as a result of the pretrained diffusion statistics. This suggests that proper spatial encoding can elicit physically plausible behaviors from a model without explicit physics training.