PSDesigner: Mimicking Human Creative Workflow for Automated Graphic Design

Paper: 2603.25738 Authors: Xincheng Shuai, Song Tang, Yutong Huang, Henghui Ding, Dacheng Tao Categories: cs.CV

The Gap

Existing AI design tools take shortcuts. Text-to-image models like DALL-E or Midjourney generate pixel outputs — beautiful, but not editable in professional tools like Photoshop. MLLMs can suggest layouts or color schemes, but they don’t produce the layered, vector-based files designers actually work with. The gap: no system bridges user intent to production-ready, editable design files through a workflow that mirrors how professionals actually design.

Prior approaches either (1) generate raster images with no layer structure, or (2) use template-filling that lacks creative flexibility. Neither captures the iterative, tool-driven process where designers collect assets, compose layers, adjust properties, and refine elements.

Problem: User intent → ??? → Editable design file
         (existing: pixels or rigid templates)
              |
              v
Assumption: Professional workflow = asset collection + tool operations
              |
              v
Method: Train on PSD files with operation traces
        Build system with asset retrieval + tool execution
              |
              v
Evidence: CreativePSD dataset (operation-annotated PSDs)
          Outperforms baselines on design quality metrics
              |
              v
Conclusion: Workflow emulation > direct generation

The Increment

One sentence: Before this paper, AI generated design images; after, AI executes design operations on editable files like a human designer would.

Core Mechanism

PSDesigner has three main components working in sequence. First, an asset collector retrieves theme-relevant images, fonts, and graphics based on user instructions — think of it as the research phase where a designer gathers materials. Second, a planner (built on an MLLM) interprets the user’s intent and decides what operations to perform: add this image as a layer, apply this filter, adjust that text property. Third, an executor translates these high-level decisions into actual Photoshop API calls that manipulate a PSD file.

The key innovation is training on CreativePSD, a dataset where each design file is paired with the sequence of operations that created it. Instead of learning “what good design looks like” from pixels, the model learns “what good designers do” from action traces. The MLLM sees examples like: “User wants a vintage poster → Designer adds sepia filter → Designer overlays texture → Designer adjusts layer opacity to 70%.” This operation-level supervision teaches the system to think in tools, not pixels.

Data flows like this: user instruction → asset collector fetches materials → planner generates operation sequence → executor applies operations to PSD → output is an editable, layered design file.

User Intent
    |
    v
[Asset Collector] ---> (images, fonts, graphics)
    |                           |
    v                           v
[MLLM Planner] <--- CreativePSD training data
    |                  (operation traces)
    v
(operation sequence: add_layer, apply_filter, adjust_opacity...)
    |
    v
[Executor] ---> Photoshop API calls
    |
    v
Editable PSD File
(layers, vectors, adjustable properties)

Think of PSDesigner as a cooking assistant that learned by watching chefs, not by looking at finished dishes. A regular image generator is like showing the assistant photos of meals and asking it to recreate them — it might produce something that looks right, but you can’t adjust the seasoning or swap ingredients. PSDesigner instead learned the recipe: “sauté onions first, add garlic after 2 minutes, deglaze with wine.” When you ask for a design, it doesn’t paint a picture; it follows a procedure. The asset collector is the pantry run, the planner is reading the recipe and deciding what to do next, and the executor is the hands doing the actual cooking. The output isn’t a photo of food — it’s a dish you can still modify, with each ingredient on its own layer.

Key Concepts

• Operation Trace Supervision: Instead of training on input-output pairs (text → image), the model learns from input-process-output triples (text → operation sequence → image). Imagine teaching someone to play chess by showing them not just winning positions, but the move-by-move game records of grandmasters. The model doesn’t just learn what good design looks like; it learns the decision tree of how to get there. In practice, this means the training data includes annotations like “Step 1: create new layer, Step 2: insert asset_042.png, Step 3: apply gaussian blur radius=5” — the actual Photoshop operations a designer performed.

• Tool-Use Capability: The system doesn’t generate pixels; it calls functions. Think of it like the difference between drawing a circle by hand versus calling circle(x, y, radius) in code. PSDesigner has a vocabulary of design operations (add layer, adjust hue, apply mask, etc.) and learns when to invoke each one. This is why the output is editable — it’s not a rendered image but a structured file where each element is a separate, modifiable object. The MLLM acts as a controller that issues commands to a design API, much like how you’d use a programming language to manipulate a document.

• CreativePSD Dataset: This is the secret sauce. Existing design datasets are just collections of finished images. CreativePSD contains PSD files (Photoshop’s native format with full layer information) plus the operation history that created them. It’s like having not just a codebase, but the Git commit history showing how it was built. The dataset covers diverse scenarios (posters, social media graphics, ads) and styles (minimalist, vintage, corporate), giving the model a broad vocabulary of design procedures to learn from.

Framework Shift

Before (mainstream approach):        After (this paper):

Text prompt                          Text prompt
     |                                    |
     v                                    v
[Text-to-Image Model]              [Asset Collector]
     |                                    |
     v                                    v
Pixel image                          Material library
(flat, uneditable)                       |
                                         v
                                   [MLLM Planner]
                                    (trained on
                                   operation traces)
                                         |
                                         v
                                   Operation sequence
                                         |
                                         v
                                    [Executor]
                                         |
                                         v
                                   Layered PSD file
                                   (editable, structured)

From generating outputs to executing procedures, the core shift is treating design as a sequence of tool operations rather than a pixel prediction problem.

Expert Assessment

Problem choice: This is a real gap. Professional designers don’t work with flat images; they need editable files. The problem sits at the intersection of creative AI and practical tooling — a space that’s been underserved because it requires both generative modeling and software engineering chops.

Method maturity: The approach is clever but not revolutionary. Using operation traces for supervision is smart, and the modular architecture (collector + planner + executor) is clean. However, the reliance on an MLLM for planning feels like it could be a bottleneck — MLLMs are expensive to run and can be unpredictable. I’d want to see ablations on whether a smaller, specialized model could handle the planning step.

Experimental integrity: The baselines seem fair (comparing against template-based methods and direct generation approaches), but I’m skeptical about the evaluation metrics. “Design quality” is subjective, and user studies with 20 participants aren’t enough to draw strong conclusions. The paper would benefit from more rigorous quantitative metrics — maybe measuring how often the generated operations match expert designers’ choices on held-out tasks.

Writing quality: The paper front-loads motivation well, but the technical sections get dense. The description of CreativePSD’s construction is buried in supplementary material when it should be front and center — that dataset is the main contribution. The results section also conflates “our system works” with “our approach is necessary” — more ablations would clarify which components actually matter.

Verdict: weak accept — Solid execution on a practical problem, but the evaluation could be more rigorous and the technical novelty is incremental rather than groundbreaking.

Takeaways

The big idea you can steal: train on process, not just outcomes. If you’re building any system that needs to produce structured, editable outputs (code, CAD files, music scores), consider collecting datasets with operation traces rather than just final artifacts. The shift from “learn to generate X” to “learn to construct X” unlocks editability and interpretability.

Concretely, the three-component architecture (retrieval + planning + execution) is a clean pattern for tool-use systems. You can apply this to other creative domains: a music composition system that retrieves samples, plans arrangement operations, and executes DAW commands; a 3D modeling system that fetches assets, plans transformations, and executes Blender API calls.

The CreativePSD dataset construction method is also transferable. If you’re in a domain where experts use software tools, you can instrument those tools to log operation histories and build a dataset of “how experts work” rather than just “what experts produce.” This is more labor-intensive than scraping finished outputs, but it teaches models to think in the domain’s native operations.