SEAL: Teaching Diffusion Models to Personalize Stickers Without Memorizing Backgrounds

Paper: 2604.26883 Authors: Changhyun Roh, Yonghyun Jeong, Jonghyun Lee, Chanho Eom, Jihyong Oh Categories: cs.CV

The Gap

Diffusion-based personalization methods like DreamBooth and Textual Inversion can generate custom concepts from reference images. But when you only have one reference image—common in sticker personalization—test-time fine-tuning (TTF) methods overfit catastrophically. They produce visual entanglement (background artifacts get baked into the learned concept) and structural rigidity (the model memorizes the exact pose/composition and can’t follow new prompts). Existing methods treat the reference image as a monolithic unit, failing to separate “what the thing is” from “where it is” and “what’s around it.”

The sticker domain amplifies this problem because prompts often demand explicit attribute changes (“make it angry,” “put it underwater”) while keeping identity fixed. No existing benchmark systematically evaluates whether a method can disentangle identity from context.

Problem: Single-image overfitting
    |
    v
Assumption: Spatial/structural constraints during adaptation
            can separate identity from context
    |
    v
Method: SEAL (semantic attention loss + token splitting + layer restriction)
    |
    v
Evidence: StickerBench evaluation shows improved identity preservation
          + contextual controllability vs baselines
    |
    v
Conclusion: Explicit spatial guidance during TTF prevents entanglement

The Increment

One sentence: Before SEAL, single-image personalization memorized backgrounds and poses; after SEAL, it learns identity while staying controllable across contexts.

Core Mechanism

SEAL is a plug-and-play adaptation module that wraps around existing personalization pipelines (DreamBooth, Custom Diffusion, etc.) without touching their U-Net backbones. It operates during the test-time fine-tuning phase, applying three constraints:

Semantic-guided Spatial Attention Loss monitors cross-attention maps between text tokens and image regions. It penalizes attention leakage—when the learned concept token attends to background regions instead of the foreground subject. This forces the embedding to encode only the subject, not its surroundings.

Split-merge Token Strategy breaks the concept into two tokens during training: one for coarse identity, one for fine details. At inference, they merge back into a single token. This prevents the single token from becoming a “kitchen sink” that encodes pose, background, and identity all at once.

Structure-aware Layer Restriction freezes deeper U-Net layers during adaptation. Early layers handle high-level semantics (what the thing is), while deep layers encode spatial structure (where things are). By restricting adaptation to early layers, SEAL prevents the model from memorizing reference-specific compositions.

Reference Image
    |
    v
[Foreground Segmentation] --> Attention Mask
    |
    v
Test-time Fine-tuning:
    |
    +---> [Semantic Attention Loss] ---> Penalize background attention
    |
    +---> [Split Token: coarse + fine] ---> Prevent monolithic encoding
    |
    +---> [Layer Restriction: freeze deep] ---> Block structure memorization
    |
    v
Adapted Embedding --> Inference (tokens merge) --> Controllable Generation

Think of SEAL like teaching a portrait artist from a single photo. A bad student memorizes the photo’s background, lighting, and pose—ask them to draw the person in a different setting, and they can’t. SEAL is the instructor standing over their shoulder: “Focus only on the face. Ignore the chair behind them. Don’t memorize how they’re sitting—learn what makes them them.” The attention loss is the instructor tapping the canvas when the student’s eyes wander to the background. The split tokens are like sketching the face in two passes—rough structure first, then details—so the student doesn’t conflate “this person’s nose” with “this person sitting at this angle.” The layer restriction is forbidding the student from practicing perspective drawing during this lesson, because that’s where pose memorization sneaks in.

Key Concepts

• Visual Entanglement: When a model learns a concept from limited data, it can’t distinguish signal from noise. If your only photo of a dog shows it on a beach, the model might encode “dog” as “dog + sand + waves.” Later, when you prompt “dog in a forest,” you get palm trees in the background. This happens because the embedding vector becomes a compressed representation of the entire image, not just the subject. SEAL fixes this by explicitly supervising where the model is allowed to look during training—attention maps must focus on the foreground mask, or the loss function penalizes the model.

• Structural Rigidity: Diffusion models generate images by iteratively denoising, guided by cross-attention between text and image features. Deep U-Net layers encode spatial relationships—“the head is above the body,” “the tail curves left.” When fine-tuning on one image, these layers memorize the reference’s exact layout. Prompt “dog jumping” but the model still generates “dog sitting” because the spatial prior is frozen. SEAL restricts adaptation to early layers, which handle semantic identity (“this is a corgi”) without encoding pose. The deep layers stay generic, preserving spatial flexibility.

Framework Shift

Before (standard TTF):              After (SEAL):

Reference Image                     Reference Image
    |                                   |
    v                                   v
Fine-tune entire U-Net          Segment foreground
    |                                   |
    v                                   v
Single concept token            Split tokens (coarse + fine)
    |                                   |
    v                                   v
Embedding = identity            Attention loss: focus on mask
          + background                  |
          + pose                        v
    |                               Freeze deep layers
    v                                   |
Generate: memorized layout              v
                                Embedding = identity only
                                    |
                                    v
                                Generate: flexible layout

From monolithic overfitting to disentangled adaptation, the core shift is explicit spatial supervision during test-time fine-tuning.

Expert Assessment

Problem choice: Real gap. Single-image personalization is a practical constraint (users don’t have 20 photos of their custom sticker character), and the entanglement/rigidity issues are well-documented in prior work. The sticker domain is a smart testbed—attribute edits are explicit and frequent, making failure modes obvious.

Method maturity: Clever but not groundbreaking. The components (attention loss, token splitting, layer freezing) are individually known techniques. The contribution is recognizing that combining them addresses a specific failure mode. The “plug-and-play” framing is honest—it genuinely doesn’t require retraining base models. However, the method introduces hyperparameters (which layers to freeze, attention loss weight) that likely need per-dataset tuning.

Experimental integrity: StickerBench is the paper’s strongest asset—a large-scale dataset with structured six-attribute tags (Appearance, Emotion, Action, Camera, Style, Background) enables systematic evaluation of disentanglement. The baselines (DreamBooth, Custom Diffusion, etc.) are fair. Quantitative metrics (identity preservation, contextual controllability) are appropriate. One concern: no ablation showing whether all three components are necessary, or if two would suffice. The paper claims “consistent improvement” but doesn’t report variance across runs or subjects.

Writing quality: The abstract and introduction are clear. The method section buries the lead—Figure 2 (the architecture diagram) should appear earlier. The “semantic-guided spatial attention loss” is explained in dense math notation before the intuition is given; reversing this order would help. The related work section is thorough but could be trimmed. The experimental section is well-structured, but the qualitative comparisons (Figure 5) cherry-pick favorable examples without showing failure cases.

Verdict: weak accept — Solid incremental contribution with a useful dataset, but the method is a combination of known techniques rather than a conceptual leap. The experimental evaluation is competent but not exhaustive.

Takeaways

Attention supervision as a regularizer: If you’re fine-tuning a model on limited data and want to prevent overfitting to spurious correlations, add a loss term that penalizes attention to irrelevant regions. This generalizes beyond stickers—think few-shot object detection, medical imaging with noisy backgrounds, or any domain where you have segmentation masks.

Layer-wise adaptation: Not all layers should be fine-tuned equally. Early layers encode “what,” deep layers encode “where.” If you want to preserve spatial flexibility, freeze the deep layers. This is a simple heuristic that transfers to other architectures (ViTs, CNNs).

Structured evaluation datasets: StickerBench’s six-attribute schema is a template for building benchmarks in other personalization domains. The key insight: vary one attribute (emotion, action) while holding identity fixed, then measure whether the model can follow the variation. This is more informative than generic “quality” scores.

Token splitting for disentanglement: If a single embedding is encoding too much (identity + context), split it into multiple tokens during training and merge at inference. This is a cheap way to increase representational capacity without architectural changes.