COMIC: Teaching AI Agents to Write Sketch Comedy Through Studio Role-Play

Paper: 2603.11048 Authors: Susung Hong, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz Categories: cs.CV, cs.AI, cs.CL, cs.MA, cs.NE

The Gap

Video generation has made impressive strides in technical quality—realism, temporal coherence, motion dynamics. But these systems optimize for perceptual fidelity, not creative content. When it comes to comedy, prior work hits a wall: humor is subjective, context-dependent, and lacks clear optimization targets. You can’t just maximize “funniness” like you minimize reconstruction error. Previous attempts at creative content generation either relied on template-filling (rigid, unfunny) or single-pass LLM generation (no refinement, no quality control). Multi-agent systems existed but weren’t applied to iterative creative refinement with learned subjective metrics.

Problem: Comedy generation needs subjective quality evaluation
   |
   v
Assumption: Real viewer preferences can train humor critics
   |
   v
Method: Multi-agent studio simulation + YouTube-trained critics
   |
   v
Evidence: Outputs approach professional sketch quality
   |
   v
Conclusion: Learned subjective metrics enable creative AI systems

The Increment

One sentence: Before this paper, automated video generation optimized technical quality but couldn’t iteratively refine creative content using learned subjective metrics; after, we have a multi-agent system that mimics production workflows and uses viewer-preference-trained critics to generate comedy sketches.

Core Mechanism

The system deploys a population of specialized AI agents, each representing a role in a production studio: writers, directors, critics, editors. The process unfolds in iterative rounds. First, multiple writer agents independently generate sketch concepts from character references. These compete in a tournament where critic agents—trained on a corpus of YouTube comedy videos and their engagement metrics—evaluate humor quality. Winning concepts advance to script development, where writer agents flesh out full scripts with dialogue, stage directions, and comedic beats.

The scripts then feed into a video generation pipeline that produces visual content. Here’s where it gets interesting: the critic agents evaluate the generated videos, providing feedback that flows back to earlier stages. This creates a refinement loop. Poor-performing sketches get revised or discarded. High-performing elements get amplified in subsequent generations. The critics aren’t using hand-coded rules—they’ve learned what makes comedy work by analyzing thousands of YouTube videos, correlating content features with viewer engagement (likes, comments, watch time).

The key architectural choice is competition plus iteration. By generating multiple candidates and selecting winners, the system explores a wider creative space than single-pass generation. By iterating with feedback, it refines ideas rather than settling for first drafts. The learned critics provide the selection pressure that drives quality improvement.

Character Refs --> [Writer Agents] --> Sketch Ideas (multiple)
                         |                    |
                         |                    v
                         |            [Critic Agents]
                         |                    |
                         |                    v
                         |            Selected Ideas
                         |                    |
                         v                    v
                   [Script Dev] <------ Feedback Loop
                         |
                         v
                   Full Scripts
                         |
                         v
                   [Video Gen]
                         |
                         v
                   Generated Videos
                         |
                         v
                   [Critic Eval] -----> Refinement Signal

Think of this like a TV writers’ room crossed with natural selection. In a real writers’ room, multiple people pitch ideas (variation), the showrunner picks what resonates (selection), those ideas get developed into scripts (refinement), and audience feedback guides future decisions (fitness function). COMIC automates this entire workflow. The writer agents are the pitching writers, generating diverse concepts. The critic agents are the showrunner, using learned taste to pick winners. The iteration loop is the development process, where scripts get punched up based on what’s working. The YouTube-trained critics are the audience proxy, providing the selection pressure that evolves better comedy over generations. Without the critics, you’d have random variation with no direction. Without iteration, you’d have one-shot generation with no refinement. The metaphor isn’t decorative—each component of the method maps directly to a role in the creative process.

Key Concepts

• Learned Subjective Metrics: Traditional ML optimizes objective functions—minimize loss, maximize accuracy. But creative tasks need subjective quality assessment. How do you teach an AI what’s funny? COMIC’s approach: analyze a large corpus of comedy videos on YouTube, extract features (timing, setup-punchline structure, character dynamics, visual gags), and correlate these with viewer engagement signals (likes, comments, retention). The critic agents learn a function mapping content features to “funniness” as revealed by real human preferences. This is different from hand-coded heuristics (too rigid) or human-in-the-loop evaluation (doesn’t scale). It’s learning taste from data. Concrete example: if videos with rapid-fire dialogue and physical comedy get higher engagement than slow-burn observational humor, the critics learn to favor that style. The metric isn’t perfect—humor is cultural and contextual—but it’s grounded in actual viewer behavior rather than arbitrary rules.

• Multi-Agent Competition: Instead of one AI generating one output, COMIC spawns multiple writer agents that independently create sketch concepts. These compete in a tournament evaluated by critics. Why does this work? Diversity plus selection. A single agent might get stuck in local optima—generating variations on the same mediocre idea. Multiple agents explore different regions of the creative space. Some will fail spectacularly, but a few will stumble on genuinely funny concepts. The critics provide selection pressure, amplifying what works and discarding what doesn’t. This mirrors evolutionary algorithms but applied to creative content. The competition isn’t just for efficiency—it’s fundamental to quality. Without it, you’d need one agent to be consistently brilliant. With it, you need the population to occasionally produce brilliance, which is much easier.

• Iterative Refinement with Feedback Loops: Most generative systems are one-shot: input → model → output. COMIC adds cycles: output → evaluation → feedback → revised input → improved output. After generating videos, critics evaluate them and provide signals that flow back to script development. Poor jokes get rewritten. Weak character moments get strengthened. This is how human writers work—first drafts are rarely final. The feedback loop transforms the system from a generator into an optimizer. Each iteration climbs toward higher quality. The trick is having critics that provide useful gradients—signals that actually improve subsequent outputs. That’s why the YouTube-trained critics matter: they’ve learned what changes make comedy better, not just what makes it different.

Framework Shift

Before (mainstream approach):        After (this paper):

Input --> [Single Model] --> Video   Characters --> [Agent Population]
          (optimize for                              |
           technical quality)                        v
                                                [Competition]
                                                     |
                                                     v
                                                [Selection]
                                                     |
                                                     v
                                                [Refinement] <--+
                                                     |          |
                                                     v          |
                                                  Video    [Critics]
                                                            (learned
                                                             metrics)

From single-pass technical optimization to multi-agent iterative creative refinement, the core shift is treating content generation as an evolutionary process guided by learned subjective quality metrics rather than a direct mapping problem.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Comedy generation sits at the intersection of multiple hard problems: subjective quality evaluation, narrative structure, timing, cultural context. It’s also commercially relevant—content creation is expensive and labor-intensive. The problem sits at the frontier of creative AI, where technical quality is solved but creative quality remains elusive. Good choice.

Method maturity: The multi-agent architecture feels slightly overengineered—do you really need separate writer/director/editor agents, or is this anthropomorphizing for the sake of it? The real contribution is the YouTube-trained critics. That’s the clever insight. The rest is scaffolding. I’d want to see ablations showing that the multi-agent competition actually outperforms simpler approaches like generate-N-and-rerank. The iterative refinement is solid, but the paper doesn’t clarify how many iterations are needed or when to stop. Feels like there’s room for simpler baselines.

Experimental integrity: The abstract claims “approaching the quality of professionally produced sketches” and “state-of-the-art performance in video generation.” These are strong claims that need careful validation. How are they measuring “approaching professional quality”? Human evaluations? If so, what’s the evaluation protocol? Are raters comparing COMIC outputs to actual SNL sketches, or just rating them in isolation? For the video generation claim, what baselines are they comparing against? The paper needs to show fair comparisons with human-written sketches and existing video generation methods. Without seeing the full paper, I’m cautious about these claims. The YouTube-trained critics are interesting, but do they actually correlate with independent human judgments of humor? That’s the key validation.

Writing quality: The abstract is clear and ambitious, which is good. But it’s light on technical details—how exactly do the agents interact? What’s the architecture of the critics? How is the YouTube corpus processed? The phrase “loosely based on real production studio roles” is vague. Are the role distinctions meaningful, or just labels? The paper would benefit from being more precise about what each agent does and why that division of labor matters. The related work section probably needs to carefully position this against prior work in creative AI, multi-agent systems, and video generation to avoid overclaiming novelty.

Verdict: Weak accept — interesting problem and the learned subjective metrics are a genuine contribution, but the multi-agent architecture needs stronger justification and the experimental validation needs to be rigorous to support the strong claims.

Takeaways

Steal the learned subjective metrics approach: Training critics on real user engagement data (YouTube likes, comments, retention) to evaluate subjective quality is applicable far beyond comedy. Product design, music generation, storytelling—any domain where quality is subjective and you have user feedback data. The key insight: don’t hand-code quality metrics, learn them from revealed preferences.

Competition beats single-shot for creative tasks: Generate multiple candidates, evaluate with learned metrics, select winners. This is more robust than trying to make one model consistently brilliant. Applicable to any creative generation task: writing, design, music, architecture.

Iterative refinement needs good critics: Feedback loops only work if the critics provide useful gradients. If your evaluation metric is noisy or misaligned, iteration makes things worse. Invest in training critics that actually correlate with human judgment before building refinement loops.

Multi-agent systems for workflow simulation: The idea of mapping AI agents to human roles (writer, director, critic) is interesting for complex creative tasks. But be honest about whether the role distinctions are load-bearing or just anthropomorphic window dressing. Test if the multi-agent architecture actually outperforms simpler approaches.