Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Paper: 2609.22086 Authors: Hongyang Du, Lan Yan, Christian Flores, Asim Kadav Categories: cs.AI, cs.CV

The Gap

Professional graphic design—creating structured, layered posters, vector branding, or promotional banners—is a challenging long-horizon agentic task. Unlike software engineering, where unit tests, type checkers, and linters provide an objective programmatic oracle, graphic design offers no executable oracle for success. Aesthetic balance, typographic hierarchy, and brand fidelity cannot be captured by a boolean unit test.

When autonomous agents attempt to manipulate complex design suites (with 200+ distinct tool APIs for kerning, layer blending, vector warping, and spatial alignment), they stumble. Fine-tuning models on noisy human design telemetry is brittle and risks catastrophic forgetting; relying purely on in-context reasoning forces the model to reinvent complex multi-step design recipes from scratch on every user prompt.

   THE CHALLENGE OF AGENTIC GRAPHIC DESIGN

   User Prompt / Creative Brief
                |
                v
   +---------------------------------------------+
   | Frontier LLM (e.g. Claude-Sonnet-4 / Opus)   |
   +---------------------------------------------+
                |
                | Interacts across 230+ design tools
                v
   [ No Programmatic Oracle (No Unit Tests!) ]
   [ High Aesthetic Ambiguity & Layer Coupling ]
                |
                v
   Existing Solutions:
   - Supervised Fine-Tuning -> Brittle, expensive, forgets base abilities
   - Zero-Shot In-Context   -> Fails on long-horizon tool composition
                |
                v
   METHOD: Designer-RSI (Procedural Memory Evolution)
   Frozen LLM + External Procedural Memory Bank
   - Widening: Acquires new subroutines for uncovered tasks
   - Deepening: Refines existing recipes against failed traces
   - Replay Gate: Admits updates ONLY if zero regression on past wins

The Increment

One sentence: Designer-RSI keeps the underlying frontier model entirely frozen and evolves an external natural-language procedural memory across 230+ professional design tools, using widening and deepening under a strict replay gate to raise GenEval2 execution success from 72.7% to 99.3% across 1,406 real user briefs without human labeling or weight updates.

Core Mechanism

The framework introduces two complementary continuous evolution mechanisms governed by a safety gate:

1. Memory Widening (Coverage Expansion): When user traffic surfaces recurrent subtasks that lack an existing procedure in the skill bank, the system identifies common interaction motifs and distills them into new modular procedural skills (e.g., “create a drop-shadowed typographic badge with aligned circular badges”).
2. Memory Deepening (Failure-Driven Refinement): For existing procedures that occasionally fail due to parameter sensitivity or tool ordering bugs, the agent analyzes paired success and failure trajectories, revising pre-conditions, tool argument bounds, or fallback recovery steps.
3. Matched Replay Gate: To prevent skill drift and regression, any candidate addition or edit must pass a re-execution test against a frozen historical replay suite. Changes that fix a bug but cause a previously mastered design brief to fail are rejected immediately.

   PROCEDURAL MEMORY EVOLUTION DYNAMICS

   User Brief ---> Procedural Memory Bank ---> 230+ Design Tools
                          ^
                          | [Evolution Loop]
   +-------------------------------------------------------------+
   | 1. Widening: Discovers missing recipes from unhandled briefs|
   | 2. Deepening: Diagnoses and patches failing edge cases      |
   | 3. Replay Gate: Test on historical suite (No Regression!)   |
   +-------------------------------------------------------------+

The structural metaphor is a professional design studio’s internal handbook of standard operating procedures (SOPs).

• The base model is a brilliant senior designer.
• The 230 tools are the complex dials and panels in Figma or Adobe Illustrator.
• Instead of sending the designer back to university to retrain their brain every time a new client arrives (weight fine-tuning), the studio maintains an evolving SOP Binder.
• When an intern encounters a task nobody wrote down before (“how to vectorize a hand-drawn logo with transparent SVG clipping”), they write a new chapter (Widening).
• When a client complains that text clipped on small mobile screens, the team updates that specific chapter with a warning note and an explicit margin rule (Deepening).
• The Replay Gate is the studio director: before printing the updated handbook, they test the revised instructions on last year’s top 100 client projects to verify that fixing the new bug didn’t break an old favorite.

Key Concepts

• Procedural Memory vs. Parametric Memory: Parametric memory lives in frozen neural network weights; procedural memory is externalized into executable, human-readable natural language workflows that can be inspected, pruned, and incrementally version-controlled.
• Matched Replay Gating: A non-regression filter that evaluates candidate skill modifications against historically validated tasks to enforce monotonic performance gains.
• Unverifiable Feedback Adaptation: Learning effectively in domains where reward signals are sparse, noisy, or heuristic rather than mathematically certifiable.

Framework Shift

Before (Static Prompting or Model Fine-Tuning):
  New Design Tasks -> Fine-tune weights or prompt from scratch
  -> Risk of catastrophic forgetting, expensive training, fragile edge cases
  -> Agent repeatedly stumbles over identical tool ordering traps

After (Designer-RSI Continual Procedural Adaptation):
  Frozen Frontier LLM + Self-Evolving Procedural Skill Bank
  -> Bank expands from 76 to 139 skills over 5 unsupervised rounds
  -> GenEval2 success surges from 72.7% to 99.3%
  -> Updates are modular, interpretable, and rollback-safe

From “treating agent knowledge as static in-context instructions or opaque weight updates,” the core shift is treating procedural expertise as a modular, living library of validated tool-usage recipes that evolves from production traffic.

Expert Assessment

Problem choice: Exceptional. Graphic design is one of the ultimate tests of multi-modal agency because it bridges high-level semantic intent with precise pixel-level coordinate manipulation across massive tool spaces.

Method maturity: The combination of widening (breadth) and deepening (depth) under an explicit replay gate is sound systems engineering. It directly addresses the stability-plasticity dilemma without touching model weights.

Experimental integrity: Rigorous. Evaluated across 1,406 real-world user briefs and 1,869 graded trajectories over five iterative rounds. Testing against both Claude-Sonnet-4 and Claude-Opus-4.6 across four specialized design benchmarks shows that the procedural bank transfers across model generations.

Writing quality: Concrete and transparent. Ablations convincingly demonstrate that widening and deepening are strictly complementary (+49% alone vs +58.5% together).

Verdict: strong accept — A stellar paradigm for post-deployment agent adaptation in domains where reward models are fuzzy or nonexistent.

Takeaways

• When building agents for complex software suites with hundreds of tools, avoid fine-tuning weights; maintain an external, modular procedural memory of tool composition recipes.
• Implement an explicit replay gate for agent prompts: never update a prompt or system skill without automatically testing it against historical successful traces.
• Separate memory growth into coverage (widening to new sub-problems) and robustness (deepening edge-case recovery).