TokenLight: Controlling Image Lighting Like Adjusting Sliders on a Mixing Board

Paper: 2604.15310 Authors: Sumit Chaturvedi, Yannick Hold-Geoffroy, Mengwei Ren, Jingyuan Liu, He Zhang, Yiqun Mei, Julie Dorsey, Zhixin Shu Categories: cs.CV, cs.GR

The Gap

Existing image relighting methods fall into two camps. Traditional inverse rendering approaches (like those using neural radiance fields) require explicit 3D reconstruction and material decomposition — expensive, brittle, and often failing on complex real-world scenes. Recent learning-based methods (diffusion models, GANs) can relight images but offer only coarse control: you can say “make it brighter” or “change the light direction,” but you can’t independently adjust intensity while keeping color constant, or move a light source along a continuous 3D path while maintaining other attributes.

The core problem: lighting is entangled. When you change one aspect (say, light position), existing methods inadvertently alter others (intensity, color, shadows). There’s no “unmixing board” where each lighting attribute gets its own slider.

Problem: Entangled lighting control in image relighting
   |
   v
Assumption: Lighting can be factorized into independent attributes
            that can be controlled separately
   |
   v
Method: Encode each attribute (intensity, color, position, etc.)
        as discrete tokens in a conditional diffusion model
   |
   v
Evidence: Quantitative metrics (PSNR, LPIPS) + qualitative results
          showing independent control + emergent understanding of
          geometry/occlusion without explicit supervision
   |
   v
Conclusion: Tokenized attributes enable precise, disentangled
            lighting control while maintaining photorealism

The Increment

One sentence: Before this paper, you could relight images but not independently control multiple lighting attributes; after, you have a “mixing board” where each lighting factor (intensity, color, 3D position, diffuse level, ambient) gets its own continuous control knob.

Core Mechanism

The method treats relighting as conditional image generation using a latent diffusion model. Instead of conditioning on a single monolithic “lighting descriptor,” it introduces attribute tokens — discrete embeddings for intensity, color, 3D light position, diffuse level, and ambient illumination. Each token is learned during training and acts as a control handle.

Training happens on a large synthetic dataset (500K images) where ground-truth lighting parameters are known, supplemented by 5K real captures. The model learns to map combinations of attribute tokens to their visual effects. Crucially, tokens are designed to be orthogonal: changing the intensity token shouldn’t require relearning how color affects the scene.

At inference, you select tokens for each attribute (e.g., intensity=0.8, color=warm, position=top-left) and feed them to the diffusion model. The model generates the relit image. Because tokens are discrete but interpolatable, you get continuous control — you can smoothly transition between “intensity=0.5” and “intensity=0.9” by interpolating their token embeddings.

Input Image --> [Encoder] --> Latent z
                                |
Attribute Tokens:               v
[Intensity] [Color] [Position] [Diffuse] [Ambient]
     |         |         |         |         |
     +----+----+----+----+----+----+----+----+
          |
          v
    [Cross-Attention Layers in Diffusion UNet]
          |
          v
    Denoising Process (T steps)
          |
          v
    [Decoder] --> Relit Image

Think of it like a mixing board in a recording studio. Each track (attribute) has its own fader. The intensity fader controls volume, the color fader controls EQ, the position fader controls stereo panning. You can push one fader without touching others. The mixing board (diffusion model) knows how to blend these independent signals into a coherent output (relit image). The magic: the board learned the “physics” of how light behaves — occlusion, shadows, material interaction — just by seeing enough examples, without anyone explicitly teaching it ray tracing.

Key Concepts

• Attribute Tokens: Instead of encoding lighting as a continuous vector (like [0.7, 0.3, 0.5, …] where dimensions are entangled), the method uses discrete learned embeddings. Imagine a vocabulary where each “word” represents a specific lighting state: “intensity_high,” “color_warm,” “position_top_left.” During training, the model learns these words and their visual meanings. At test time, you compose a “sentence” by picking words (tokens) for each attribute. The key insight: discrete tokens are easier to disentangle than continuous vectors because they force the model to learn distinct, non-overlapping representations. It’s like having separate LEGO bricks (tokens) instead of a blob of clay (continuous vector) — you can swap one brick without reshaping the whole structure.

• Emergent Geometry Understanding: The model is never explicitly told “this pixel is occluded” or “this surface is glossy.” Yet it learns to cast shadows correctly, handle occlusion when lights move behind objects, and even relight transparent materials plausibly. How? The synthetic training data provides enough examples of light-geometry interactions that the diffusion model internalizes these patterns. It’s like learning grammar by reading thousands of sentences without studying grammar rules — the structure emerges from exposure. This is remarkable because traditional relighting requires explicit 3D reconstruction; here, the model builds an implicit understanding purely from 2D supervision.

• Continuous Control via Discrete Tokens: Sounds contradictory, right? Tokens are discrete (you have “intensity_0.5” and “intensity_0.6” as separate tokens), but their embeddings live in a continuous space. So you can interpolate: take 70% of the “intensity_0.5” embedding and 30% of “intensity_0.6” to get “intensity_0.53.” This gives you smooth, continuous control while maintaining the disentanglement benefits of discrete representations. It’s like having preset radio stations (discrete) but being able to fine-tune between them (continuous).

Framework Shift

Before (mainstream approach):        After (this paper):

Input Image                          Input Image
     |                                    |
     v                                    v
[Inverse Rendering]                 [Encoder]
     |                                    |
Geometry + Materials                 Latent z
     |                                    |
     v                                    v
[Forward Rendering]              [Attribute Tokens]
with new lighting                 (independent controls)
     |                                    |
     v                              [Diffusion Model]
Relit Image                         (learned physics)
                                         |
Explicit 3D pipeline                     v
Brittle, slow                       Relit Image
                                    
                                    Implicit understanding
                                    Fast, robust

One sentence: From explicit 3D reconstruction followed by forward rendering, to learned implicit lighting manipulation where attribute tokens act as independent control knobs in a diffusion model that has internalized light-geometry interactions.

Expert Assessment

Problem choice: Real gap. Existing relighting methods either require expensive 3D reconstruction or provide only coarse control. The demand for fine-grained, independent lighting control in image editing is genuine — photographers and VFX artists need this. The paper sits at the intersection of two trends: diffusion models for image generation and disentangled representations for controllable synthesis.

Method maturity: Clever insight with solid execution. The attribute token idea is elegant — it sidesteps the entanglement problem by design rather than brute force. However, the reliance on synthetic data (500K images) raises questions about generalization. The 5K real captures help, but it’s a band-aid. A simpler approach might be to use existing inverse rendering for synthetic data generation but skip it at inference — which is essentially what they do, but they don’t explore whether cheaper synthetic data (fewer scenes, simpler geometry) would suffice.

Experimental integrity: Baselines are fair (prior diffusion-based relighting, traditional inverse rendering). Quantitative metrics (PSNR, LPIPS, SSIM) are standard. The qualitative results are impressive — lights inside objects, transparent materials — but these are cherry-picked examples. The paper lacks failure case analysis. What happens when you push tokens to extreme combinations not seen in training? The user study (preference test) is small (20 participants) but adequate for this type of work.

Writing quality: The paper is well-structured, but the method section is dense. The attribute token design (how many tokens per attribute? how are they initialized?) is buried in supplementary material. The “emergent geometry understanding” claim is bold but under-explained — a dedicated ablation showing what the model learns about occlusion/materials would strengthen this. The related work section is thorough but could be trimmed.

Verdict: Weak accept — the core idea (attribute tokens for disentangled lighting control) is novel and the results are strong, but the reliance on large-scale synthetic data and lack of failure analysis prevent this from being a strong accept. It’s a solid contribution that advances the field but leaves room for follow-up work on data efficiency and robustness.

Takeaways

Discrete tokens for continuous control: The trick of using discrete embeddings (tokens) but interpolating them for continuous control is transferable. If you’re building any controllable generation system (audio, video, 3D), consider tokenizing your control attributes instead of using continuous vectors. It forces disentanglement and makes interpolation more interpretable.

Synthetic data + small real data: The 500K synthetic + 5K real recipe is a practical pattern. Generate cheap synthetic data with ground-truth labels, then fine-tune on a small real dataset to bridge the sim-to-real gap. This is especially useful when real data is expensive (e.g., multi-view captures, motion capture).

Emergent understanding from supervision: The model learns geometry and material interactions without explicit supervision — just from seeing enough examples. This suggests that for tasks where explicit modeling is hard (physics simulation, occlusion reasoning), a large dataset with implicit structure might be enough for a neural network to internalize the rules. The key: your training data must cover the diversity of interactions you want the model to handle.

Cross-attention for control: Using cross-attention to inject control signals (attribute tokens) into a diffusion model is now standard, but this paper shows how to structure those controls for independence. If you’re building a conditional diffusion model, think carefully about how your conditioning vectors are organized — entangled vectors lead to entangled outputs.