Teaching Diffusion Models to Say 'No': A Geometric Approach to Linguistic Negation

Paper: 2603.06533 Authors: Taewon Kang, Ming C. Lin Categories: cs.CV

The Gap

Diffusion models can generate “a dog playing in the park” beautifully, but ask for “a park with no dogs” and you’ll likely still get dogs. The problem isn’t that models don’t understand negation — it’s that they treat “no X” as just another semantic direction to follow, like “more X” or “blue X”. Prior work tried negative prompting (subtracting embeddings), attention masking, or fine-tuning on negation datasets. These approaches either fail on compositional cases (“no red cars and no blue trucks”) or require expensive retraining for each model.

The core issue: existing methods treat negation as a feature to learn rather than a constraint to enforce. They ask “what does ‘not X’ look like?” when they should ask “what space excludes X?”

Problem: Diffusion models generate "not X" by learning patterns
         |
         v
Assumption: Negation is a geometric constraint, not a semantic feature
         |
         v
Method: Project guidance updates onto feasible region (convex set)
         |
         v
Evidence: Handles multi-negation, scope ambiguity, graded semantics
         |
         v
Conclusion: Training-free negation via constrained optimization

The Increment

One sentence: Before, negation was a prompt engineering trick that often failed; after, it’s a geometric constraint that provably excludes unwanted content.

Core Mechanism

The method has three components working in sequence. First, it takes the standard classifier-free guidance update — the direction diffusion wants to move to satisfy your prompt. Second, it constructs a convex feasible set representing “all states that don’t contain the negated concept”. Third, it projects the guidance update onto this feasible set, ensuring the model never steps into forbidden territory.

Here’s the data flow: at each diffusion timestep, you have a noisy latent state. Classifier-free guidance computes an update direction by comparing conditional and unconditional predictions. Instead of applying this update directly, NEGATE checks if it would violate negation constraints. If so, it finds the closest valid direction that respects the constraint — geometrically, this is a projection onto a convex set. The projected update is then applied to the latent.

The constraint set itself is built from CLIP embeddings. For “no dogs”, the method defines a halfspace in embedding space: all points whose similarity to “dog” is below a threshold. Multiple negations create an intersection of halfspaces, which remains convex. This convexity is crucial — it guarantees the projection is unique and computationally tractable.

Diffusion Step t:
                                    
  Noisy Latent (x_t)                Constraint Set C
       |                            (convex region in
       |                             embedding space)
       v                                   |
  CFG Guidance -----> [Projection] <------+
   (unconstrained       onto C
    update direction)      |
                           v
                    Constrained Update
                           |
                           v
                    Next Latent (x_t-1)

Think of it like a self-driving car navigating with a map of forbidden zones. Standard diffusion is like giving the car a destination and letting it drive straight there — it might cut through restricted areas. NEGATE adds geofencing: the car still aims for the destination, but whenever its planned route enters a forbidden zone, it automatically adjusts to the nearest legal path. The forbidden zones (convex sets) are defined by what you don’t want (negated concepts), and the adjustment (projection) happens at every timestep, ensuring the car never violates constraints even as the route evolves.

Key Concepts

• Classifier-Free Guidance as a Vector Field: In diffusion models, each denoising step computes two predictions — one conditioned on your prompt, one unconditional. The difference between these predictions is a vector pointing toward “more like the prompt”. Standard practice scales this vector and adds it to the unconditional prediction. NEGATE reinterprets this: the vector is a proposed direction in semantic space, and you can modify it before applying. Concrete example: if the unconditional model predicts “generic outdoor scene” and the conditional predicts “park with dogs”, the guidance vector points toward “add dogs”. NEGATE intercepts this vector and asks “does this violate ‘no dogs’?” If yes, it rotates the vector to point toward “park features without dogs”.

• Convex Projection as Constraint Enforcement: A convex set is a region where any line segment between two points in the region stays inside the region. Projecting onto a convex set means finding the closest point in the set to your current location. Why this matters: if you define “no dogs” as “CLIP similarity to ‘dog’ < threshold”, that’s a halfspace (convex). “No dogs and no cats” is the intersection of two halfspaces (still convex). Projection onto this intersection gives you the nearest valid semantic direction. Concrete example: imagine a 2D plane where x-axis is “dog-ness” and y-axis is “cat-ness”. Your constraint is x<0.3 and y<0.3 (the lower-left quadrant). If guidance wants to move you to (0.8, 0.2), projection snaps you to (0.3, 0.2) — maximum dog-ness allowed while respecting the constraint.

• Scope-Sensitive Negation via Structured Constraints: Natural language negation has scope: “not all dogs are brown” differs from “all dogs are not brown”. Diffusion models typically ignore this. NEGATE handles scope by constructing hierarchical constraint sets. For “not (red car)”, the constraint applies to the entire object. For “car that is not red”, the constraint applies only to the color attribute. The method uses linguistic parsing to determine scope, then builds constraint sets accordingly. Concrete example: “a scene without any animals” creates a global constraint on the entire frame. “A dog that is not running” creates a local constraint on the dog’s action attribute, allowing the dog to exist but restricting its motion semantics.

Framework Shift

Before (mainstream approach):        After (this paper):

Prompt: "no dogs"                    Prompt: "no dogs"
    |                                    |
    v                                    v
Embed "no dogs"                      Parse negation structure
    |                                    |
    v                                    v
Generate toward                      Define feasible set C:
embedding target                     {z : sim(z, "dog") < tau}
    |                                    |
    v                                    v
Hope model learned                   At each step:
what "no X" means                    - Compute CFG update u
    |                                - Project u onto C
    v                                - Apply projected u
Output (often has dogs)                  |
                                         v
                                    Output (provably no dogs)

From learning negation as a semantic pattern to enforcing negation as a geometric constraint, the core shift is treating “not X” as a boundary condition rather than a target embedding.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Negation failure is a well-documented pain point in diffusion models, and it’s not just an academic curiosity — users constantly complain about it in practice. The problem sits at the intersection of linguistics and generative modeling, which is underexplored. However, the framing as “linguistic negation” might be overselling it — this is really about constraint satisfaction in semantic space, which is broader and more useful than just handling “not”.

Method maturity: Clever insight with solid geometric foundations. The convex projection idea is elegant and theoretically grounded. But I’m skeptical about the CLIP embedding threshold — how do you set tau? The paper doesn’t deeply address this, and in practice, threshold sensitivity could be a major issue. Also, the method assumes CLIP embeddings cleanly separate concepts, which breaks down for abstract or compositional negations. There’s a simpler approach being overlooked: just use attention masking with better heuristics. The authors dismiss this too quickly.

Experimental integrity: The benchmark suite is valuable — isolating negation failure modes is smart. But the baselines are weak. They compare against vanilla CFG and basic negative prompting, not against recent work like Attend-and-Excite or other spatial control methods. The video experiments are limited (only a few examples shown), and there’s no user study to validate whether “negation compliance” actually improves perceived quality. The quantitative metrics (CLIP similarity to negated concept) are reasonable but not sufficient — you need human eval to catch failure modes.

Writing quality: The paper front-loads heavy formalism (convex optimization, Lagrangian duals) that obscures the core idea. Section 3 should start with the intuition and geometric picture, then add math. The related work section is thorough but reads like a literature dump rather than a narrative. The biggest missed opportunity: no ablation on the constraint set construction. What happens if you use different embedding models? Different similarity metrics? These questions are critical for practitioners but left unanswered.

Verdict: weak accept — solid idea with theoretical grounding, but experimental validation is incomplete and the method’s practical robustness is unclear.

Takeaways

The big steal here is the framing: treat unwanted content as a geometric constraint, not a semantic target. This generalizes beyond negation. If you’re building any conditional generation system, ask “what’s the feasible set?” instead of “what’s the target embedding?” Concretely, you can adapt this to safety filtering (project away from NSFW regions), style control (constrain to a style manifold), or multi-objective generation (intersection of multiple constraint sets).

The convex projection trick is also portable. Anytime you have a guidance signal that might violate constraints, project it onto the feasible set before applying. This works for any differentiable constraint you can express as a convex set. The code pattern is simple: compute update, check constraint, project if needed, apply.

One warning: the method assumes your constraint is well-defined in embedding space. If your concept is fuzzy or context-dependent, the constraint set becomes ill-defined and projection fails. Don’t expect this to magically handle “no inappropriate content” — you need crisp, separable concepts for it to work.