BizGenEval: Testing If AI Can Actually Design Business Documents

Paper: 2603.25732 Authors: Yan Li, Zezi Zeng, Ziwei Zhou, Xin Gao, Muzhao Tian, Yifan Yang, Mingxi Cheng, Qi Dai, Yuqing Yang, Lili Qiu Categories: cs.CV

The Gap

Image generation models like DALL-E and Midjourney have gotten scary good at making beautiful landscapes and portraits. But ask them to generate a slide deck with specific text in the right places, or a bar chart with accurate labels, and they fall apart. The problem: existing benchmarks (COCO, ImageNet) only test natural image synthesis—pretty pictures with loose constraints. They don’t measure what matters for real commercial work: precise text rendering, multi-element layout control, attribute binding (this label goes with that bar), and domain knowledge (scientific figures need specific conventions).

The gap is structural. Current evaluation assumes “good image = photorealistic + aesthetically pleasing.” Commercial design needs “correct image = all constraints satisfied + functionally usable.” No one has systematically tested whether models can handle the multi-constraint, structured nature of business documents.

Problem: Models trained on natural images
         fail at commercial design tasks
              |
              v
Assumption: Commercial design = natural image synthesis
            + structured constraints + domain knowledge
              |
              v
Method: Build benchmark with 5 document types,
        4 capability dimensions, 20 tasks,
        400 prompts, 8000 verification questions
              |
              v
Evidence: 26 models tested, all show substantial gaps
          (best model ~60% on complex tasks)
              |
              v
Conclusion: Current models not ready for
            professional visual content creation

The Increment

One sentence: Before this paper, we thought image generation was “solved” for practical use; after, we know models can’t handle the structured constraints of real commercial design.

Core Mechanism

BizGenEval is a testing framework, not a model. It works like a standardized exam with multiple subjects. The benchmark covers five document types (slides, charts, webpages, posters, scientific figures) and tests four capabilities (text rendering, layout control, attribute binding, knowledge-based reasoning). Each combination creates a specific task—for example, “generate a bar chart with correct text labels” tests text rendering + charts.

For each task, human experts write prompts that specify multiple constraints: “Create a slide with title ‘Q4 Results’ in the top-left, a pie chart showing 40% blue and 60% red in the center, and footer text ‘Confidential’ in 8pt font.” Then they create checklist questions to verify each constraint: “Is the title ‘Q4 Results’?”, “Is it in the top-left?”, “Is the blue slice 40%?”, etc. Each image gets scored by how many checklist items it passes.

The data flow: prompt → model → generated image → human verification against checklist → pass/fail score for each constraint → aggregate metrics across tasks. The benchmark contains 400 carefully designed prompts and 8000 verification questions, all human-curated to ensure they test real commercial requirements.

Input Prompt                Generated Image           Verification
+-----------+              +-------------+           +------------+
| "Create   |              |             |           | Checklist: |
| slide:    |  ------>     |   [IMAGE]   |  ------>  | - Title?   |
| Title='X' |   Model      |             |   Human   | - Layout?  |
| Chart=... |              |             |           | - Colors?  |
+-----------+              +-------------+           +------------+
                                                            |
                                                            v
                                                     Score: 15/20

Think of BizGenEval as a driving test for image models. A regular benchmark is like testing if someone can steer a car on an empty road (natural image synthesis). BizGenEval is the full driving exam: parallel parking (layout control), reading street signs (text rendering), following GPS directions (attribute binding), and knowing traffic laws (domain knowledge). You need all four to drive safely, but most models only practiced steering. The test has five different courses (document types), each requiring all four skills in different combinations. The checklist questions are like the examiner’s clipboard—objective pass/fail criteria, not subjective judgment.

Key Concepts

• Attribute Binding: In natural images, if you say “red car next to blue house,” the model might swap colors or positions—annoying but not critical. In commercial design, if a chart legend says “Revenue = Blue” but the blue bar shows costs, the document is wrong and unusable. Attribute binding means ensuring each visual element has the correct properties and relationships specified in the prompt. It’s the difference between “close enough” and “exactly right.” Example: A poster with three product photos must show Product A in the top slot, Product B middle, Product C bottom—not randomly arranged.

• Multi-Constraint Satisfaction: Commercial documents have interdependent requirements. A slide needs the title in a specific font AND size AND position AND color, while also having a chart with correct data AND labels AND legend, plus footer text that doesn’t overlap anything. If any one constraint fails, the whole document fails. This is different from natural images where you can have a “mostly correct” sunset. The challenge: models trained with diffusion or autoregressive methods optimize for overall image quality, not constraint satisfaction. They’ll sacrifice text accuracy for better composition.

• Checklist Verification: Instead of asking humans “rate this image 1-10,” BizGenEval asks binary questions: “Is the title text ‘Q4 Results’? Yes/No.” “Is the pie chart 40% blue? Yes/No.” This eliminates subjective bias and makes evaluation reproducible. Each prompt has 20 checklist questions covering all specified constraints. The score is simply: (passed questions) / (total questions). This approach reveals that models often get 80% of constraints right but fail on critical details—which means the output is unusable.

Framework Shift

Before (mainstream approach):        After (this paper):

Natural Image Synthesis:             Commercial Design Evaluation:

  Prompt                               Prompt + Constraints
    |                                    |
    v                                    v
  Model                                Model
    |                                    |
    v                                    v
  Image                                Image
    |                                    |
    v                                    v
  Human rates                          Checklist verification
  "quality" (1-10)                     (binary pass/fail)
    |                                    |
    v                                    v
  Single score                         Constraint-level scores
  (subjective)                         (objective, diagnostic)

Focus: Aesthetic quality               Focus: Functional correctness
Metric: FID, CLIP score                Metric: Constraint satisfaction rate
Goal: Pretty images                    Goal: Usable documents

From aesthetic evaluation to functional verification, the core shift is measuring whether outputs satisfy explicit requirements rather than implicit quality standards.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has been celebrating FID improvements while ignoring that models can’t generate a usable PowerPoint slide. The timing is perfect—companies are trying to deploy these models for content creation and hitting walls. The problem sits at the intersection of vision and language, where most research focuses on understanding (VQA, captioning) rather than generation with constraints.

Method maturity: This is pure benchmark engineering, not algorithmic innovation. The insight is recognizing that commercial design is a different problem class than natural images. The execution is solid—five document types cover real use cases, four capability dimensions are well-chosen, and checklist verification is more rigorous than rating scales. However, the benchmark is static (400 prompts) and doesn’t test iterative refinement, which is how humans actually use these tools. Also, 8000 questions required massive human labor—not scalable for continuous evaluation.

Experimental integrity: Testing 26 models is thorough. The results are believable—best commercial APIs (GPT-4V, Gemini) score ~60% on complex tasks, open-source models ~40%. The paper shows per-task breakdowns, revealing that all models struggle with text rendering and attribute binding. One concern: the prompts are in English, but commercial design is global. Another: the checklist questions were written by the same people who wrote the prompts, which could introduce bias toward certain phrasings or constraint types.

Writing quality: The paper front-loads motivation well but buries the methodology. Section 3 (benchmark construction) should come before Section 2 (related work) so readers understand what they’re evaluating before comparing to other benchmarks. The results section is thorough but repetitive—showing per-task scores for 26 models creates a wall of numbers. A better approach: show aggregate scores in main text, per-task details in appendix. The discussion section is weak—it lists limitations but doesn’t propose solutions or research directions.

Verdict: weak accept — Addresses a real problem with solid execution, but the contribution is a benchmark (infrastructure) rather than a method (insight). Valuable for the community, but not groundbreaking.

Takeaways

If you’re building or evaluating generative models for practical applications, steal the checklist verification approach. Instead of asking “is this output good?”, decompose requirements into binary questions. This works beyond images—for code generation, ask “does it compile?”, “does it pass test X?”, “does it use library Y?”. For text generation, check factual claims individually rather than rating overall quality.

The four capability dimensions (text rendering, layout, attribute binding, knowledge reasoning) transfer to other structured generation tasks. If you’re generating UI mockups, CAD drawings, or data visualizations, test these separately. Models often excel at one (layout) while failing at another (text), and aggregate metrics hide this.

The broader lesson: “good at natural data” doesn’t mean “good at structured data.” If your application has explicit constraints, you need constraint-specific evaluation. Don’t assume a model that generates beautiful images can generate correct diagrams.