Color by Coordinates: Finding HSL Structure in FLUX's Latent Space

Paper: 2603.12261 Authors: Mateusz Pach, Jessica Bader, Quentin Bouniot, Serge Belongie, Zeynep Akata Categories: cs.LG, cs.AI, cs.CV

The Gap

Text-to-image models like FLUX.1 are powerful but opaque. When you want to change the color of a generated object, your options are basically: re-prompt and hope, or fine-tune the model. Neither is satisfying. Re-prompting is unreliable — the model might shift composition, lighting, or style along with the color. Fine-tuning is expensive and doesn’t generalize. Inversion-based editing methods (like prompt-to-prompt or null-text inversion) give you more control but still operate at the text or attention level, not at the level of “what encodes redness.”

The deeper problem is that nobody really knew how color is represented inside these models. The VAE latent space — the compressed representation that sits between the pixel image and the diffusion process — was treated as a black box. Prior interpretability work on diffusion models focused mostly on the U-Net or transformer attention patterns, not on the VAE’s internal geometry.

This paper asks: is there geometric structure in the VAE latent space that corresponds to human-interpretable color attributes? And if so, can you manipulate it directly?

Problem: Color control in T2I models is unreliable or expensive
    |
    v
Assumption: VAE latent space has interpretable geometric structure
    |
    v
Method: PCA + circular statistics on latent vectors -> find HSL subspace
    |
    v
Evidence: LCS predicts color accurately + direct manipulation changes color
    |
    v
Conclusion: Color is encoded in a structured, manipulable subspace (LCS)

The Increment

One sentence: Before this paper, color in diffusion model latents was a mystery you worked around; after it, there’s a named, geometric subspace you can reach in and turn like a dial.

Core Mechanism

The authors start by encoding a large set of images with known color properties into FLUX.1’s VAE. They then look at the resulting latent vectors and ask: do images with similar hues cluster together? They apply PCA to the latent space and find that a small number of principal components — the Latent Color Subspace (LCS) — capture most of the variance in color. Crucially, hue turns out to live on a circle in this subspace (two PCA components form a 2D plane where hue angle corresponds to actual hue), while saturation and lightness map to scalar directions.

Once the subspace is identified, prediction is straightforward: project a latent onto the LCS basis, read off the coordinates, convert to HSL. Control works in reverse: take a target HSL value, compute the corresponding LCS coordinates, and shift the latent by the difference. No gradient, no training, no diffusion steps — just linear algebra on the compressed representation.

The verification is two-pronged. First, they show the LCS can predict the color of an image from its latent alone, with high accuracy. Second, they show that editing the latent along LCS directions changes the image color in the expected way while leaving other attributes (texture, composition, identity) largely intact.

Input image
    |
    v
[VAE Encoder] -> latent z (16 x H x W)
    |
    v
[Project onto LCS basis] -> (hue_cos, hue_sin, saturation, lightness)
    |
    +-- Prediction path: convert coords -> HSL -> compare to ground truth
    |
    +-- Control path: compute delta from target HSL
                          |
                          v
                     z' = z + delta * LCS_basis
                          |
                          v
                     [VAE Decoder] -> edited image

Think of the VAE latent space as a very high-dimensional filing cabinet. Each drawer is a dimension, and every image is a folder stuffed into a specific location. The filing cabinet looks chaotic — 16 dimensions per spatial location, millions of possible combinations. But the authors discover that if you look at just a few specific drawers (the LCS), all the color information is neatly organized there. Hue is filed in a circular pattern in drawers 1 and 2, saturation is in drawer 3, lightness in drawer 4. The rest of the drawers hold other stuff (texture, shape, semantics), but color lives in its own corner.

The manipulation is like editing a file without opening the whole cabinet. You know exactly which drawers hold color, so you reach in, adjust the coordinates, and close the drawer. The image changes color, but everything else stays put because you didn’t touch the other drawers.

Key Concepts

• Latent Color Subspace (LCS): A low-dimensional subspace within the VAE’s latent representation where color information is concentrated. It’s discovered via PCA: the first few principal components of the latent vectors, when computed over a dataset of images with known colors, turn out to encode hue, saturation, and lightness in a structured way. The “subspace” means you can project any latent vector onto these components and read off its color.

• Circular Hue Encoding: Hue (red, orange, yellow, green, blue, purple) is a circular attribute — after purple comes red again. The LCS encodes this as a 2D circle: two principal components form a plane, and the angle in that plane corresponds to hue. This is elegant because it respects the topology of color space: colors that are perceptually close (red and orange) are geometrically close in the latent space.

• Training-Free Control: Once the LCS is identified, you can manipulate color without retraining or fine-tuning the model. You compute the target color’s coordinates in the LCS, compute the delta from the current latent’s projection, and add that delta back to the latent. Then decode. No gradients, no optimization, just linear algebra. This is fundamentally different from methods that require backpropagation through the diffusion process.

Framework Shift

Before (mainstream approach):        After (this paper):

  Want to change color                 Want to change color
        |                                    |
        v                                    v
  Re-prompt + hope                     Encode image -> latent z
        OR                                   |
  Fine-tune model                            v
        OR                               Project onto LCS
  Inversion + editing                        |
  (all expensive/unreliable)                 v
                                       Read current HSL
                                             |
                                             v
                                       Compute delta to target HSL
                                             |
                                             v
                                       z' = z + delta * LCS_basis
                                             |
                                             v
                                       Decode -> edited image
                                       (fast, deterministic, precise)

From trial-and-error or expensive optimization to direct geometric manipulation, the core shift is: color is not a black-box property but a structured, accessible subspace.

Expert Assessment

Problem choice: Real and well-motivated. Color control in T2I models is a known pain point, and the lack of interpretability in VAE latents is a genuine gap. The timing is right — as these models get deployed for creative tools, users need reliable control knobs.

Method maturity: PCA on latent vectors is not novel, but the discovery that it reveals HSL structure is genuinely interesting. The circular hue encoding is elegant and suggests the VAE learned something meaningful about color perception, not just pixel statistics. The training-free manipulation is a direct consequence of the structure, not a separate contribution.

Experimental integrity: The key experiments are: (1) does LCS predict color accurately? (2) does editing LCS change color as expected? (3) does it preserve other attributes? All three need quantitative metrics and user studies. The paper would be stronger with ablations showing what happens if you use different PCA components or different projection methods.

Writing quality: The title and abstract are clear. The methods section (not fully visible) likely needs more detail on how the LCS basis is computed and how the delta is applied. The failure cases section is critical — when does this break? (Likely: images with complex lighting, transparent objects, or ambiguous color boundaries.)

Verdict: accept — a clean discovery with practical utility, though the novelty is more in the finding than the method.

Takeaways

The PCA-for-interpretability pattern is underused. Whenever you have a learned latent space (VAE, autoencoder, embedding model), try PCA on a dataset with known ground-truth attributes. You might find that the top components correspond to interpretable dimensions.

The circular encoding insight is worth remembering: if your attribute is circular (hue, angle, time-of-day), check whether the latent space encodes it as a 2D circle. If it does, you can manipulate it cleanly. If it doesn’t, the model might not have learned the right structure.

The training-free control pattern (identify subspace, project, edit, decode) is broadly applicable. Any time you have a generative model with a structured latent space, ask whether you can manipulate specific attributes by editing the latent directly rather than going through the full generation process.