CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generation

Paper: 2602.22150 Authors: YuXin Song, Yu Lu, Haoyuan Sun, Huanjin Yao, Fanglong Liu, Yifan Sun, Haocheng Feng, Hang Zhou, Jingdong Wang Categories: cs.CV

Abstract

Unified conditional image generation faces a fundamental challenge: different tasks require fundamentally incompatible internal representations. Semantic synthesis tasks demand high-level conceptual understanding, while spatial manipulation tasks require precise localization cues. This paper introduces CoLoGen, a unified diffusion framework that addresses the concept-localization representational conflict through progressive curriculum learning. The framework employs a three-stage training strategy that first establishes foundational conceptual and localization capabilities, then adapts these abilities to diverse visual conditions, and finally refines their integration for complex instruction-driven generation. At its core, the Progressive Representation Weaving (PRW) module dynamically routes features to specialized experts and stably merges their outputs across training stages. Experimental results demonstrate competitive or superior performance across editing, controllable generation, and customized generation tasks.

Key Contributions

• Identifies and formalizes the concept-localization representational conflict as a core challenge in unified conditional image generation
• Proposes a three-stage progressive curriculum learning strategy that systematically builds and reconciles dual representational capabilities
• Introduces the Progressive Representation Weaving (PRW) module for dynamic feature routing and stable expert integration
• Achieves state-of-the-art or competitive performance across multiple generation tasks including editing, controllable generation, and customization

Methodology: Progressive Curriculum Learning

CoLoGen’s training strategy follows a carefully designed three-stage curriculum that mirrors human learning progression from fundamentals to complex applications.

Stage 1: Foundation Building establishes core capabilities by training separate conceptual and localization pathways. The conceptual branch learns semantic understanding through text-to-image generation tasks, while the localization branch develops spatial precision through structure-conditioned generation. This separation prevents early-stage interference between competing objectives.

Stage 2: Conditional Adaptation introduces diverse visual conditions (edges, depth maps, segmentation masks) to both pathways. The PRW module begins routing features to appropriate experts based on task requirements, learning which representations best serve different conditioning modalities.

Stage 3: Synergistic Refinement tackles complex instruction-driven tasks that require simultaneous conceptual understanding and spatial precision. The PRW module dynamically balances contributions from both expert pathways, enabling seamless integration for tasks like “change the dog to a cat while preserving pose.”

The Progressive Representation Weaving module operates through learned routing weights $\alpha_t$ and $\beta_t$ that evolve across stages:

$\mathbf{h}_{\text{fused}} = \alpha_t \cdot \mathbf{h}_{\text{concept}} + \beta_t \cdot \mathbf{h}_{\text{local}} + \gamma_t \cdot \mathbf{h}_{\text{shared}}$

where $\alpha_t + \beta_t + \gamma_t = 1$ ensures stable training dynamics.

Experimental Results and Analysis

CoLoGen demonstrates strong performance across three major evaluation domains:

Image Editing: On instruction-based editing benchmarks, CoLoGen achieves superior semantic fidelity while maintaining structural consistency. The framework successfully handles complex edits requiring both conceptual changes and spatial preservation, outperforming specialized editing models in multi-constraint scenarios.

Controllable Generation: When conditioned on structural inputs (pose, depth, edges), CoLoGen matches or exceeds ControlNet-style approaches while maintaining better semantic coherence. The dual-pathway architecture prevents the common issue where strong spatial conditioning degrades semantic quality.

Customized Generation: For subject-driven generation tasks, CoLoGen effectively balances identity preservation with creative variation. The progressive training enables smooth interpolation between strict replication and conceptual adaptation based on instruction specificity.

Ablation studies reveal that removing the progressive curriculum leads to 15-20% performance degradation, confirming that staged learning is essential for resolving representational conflicts. The PRW module contributes 8-12% improvement over naive feature concatenation approaches.

Architectural Insights and Design Principles

The success of CoLoGen stems from several key architectural decisions that address the representational conflict at multiple levels.

Expert Specialization: Rather than forcing a single encoder to handle all conditioning types, CoLoGen maintains specialized pathways that develop distinct inductive biases. The conceptual expert learns hierarchical semantic features optimized for text-image alignment, while the localization expert develops fine-grained spatial attention patterns.

Dynamic Integration: The PRW module’s learned routing mechanism adapts to task requirements at inference time. For pure semantic tasks, it upweights conceptual features; for spatial tasks, it emphasizes localization cues. This flexibility eliminates the need for task-specific model variants.

Stable Training Dynamics: The progressive curriculum prevents gradient conflicts that arise when optimizing contradictory objectives simultaneously. Early-stage separation allows each pathway to establish stable feature spaces before integration begins.

The framework’s modular design also enables efficient scaling: new conditioning modalities can be incorporated by training additional expert branches without retraining the entire model.

Takeaways

1. Unified image generation requires explicitly addressing the concept-localization representational conflict rather than assuming a single representation can serve all tasks
2. Progressive curriculum learning that mirrors human skill acquisition—fundamentals first, then adaptation, finally integration—proves more effective than joint training from scratch
3. Dynamic feature routing through specialized experts outperforms monolithic architectures for heterogeneous conditional generation tasks
4. The three-stage training strategy (foundation → adaptation → refinement) provides a principled framework applicable beyond image generation to other multi-task learning scenarios
5. Balancing expert specialization with stable integration mechanisms is crucial for maintaining both task-specific performance and cross-task generalization