DV-World: Testing Data Visualization Agents in Real Enterprise Workflows

Paper: 2604.25914 Authors: Jinxiang Meng, Shaoping Huang, Fangyu Lei, Jingyu Guo, Haoxiang Liu, Jiahao Su, Sihan Wang, Yao Wang, Enrui Wang, Ye Yang Categories: cs.CL

The Gap

Existing data visualization benchmarks test agents in sanitized sandboxes: write Python code to generate a chart from clean CSV data, assume the user’s intent is crystal clear, and call it done. But real enterprise work looks nothing like this. Analysts work in Excel and Google Sheets, not Jupyter notebooks. They inherit broken dashboards that need diagnosis and repair. They get vague requests like “make it look better” and must probe for actual requirements. Prior benchmarks (ChartQA, PlotQA, MatPlotBench) focus on single-shot chart generation from perfect specifications—they ignore the messy lifecycle of professional visualization work.

Real-world gap:
  Sandbox confinement          Native environment grounding
       |                                |
       v                                v
  Code-only tasks    --->    Spreadsheet + code + repair
       |                                |
       v                                v
  Perfect intent     --->    Ambiguous requirements
       |                                |
       +--------------------------------+
                      |
                      v
              DV-World benchmark
                      |
       +--------------+--------------+
       |              |              |
   DV-Sheet      DV-Evolution   DV-Interact
   (native)      (adaptation)   (alignment)
       |              |              |
       v              v              v
   <50% performance across SOTA models

The Increment

One sentence: Before this paper, we tested visualization agents on toy chart-generation tasks; after, we have a benchmark that exposes their inability to handle real enterprise workflows involving spreadsheets, legacy code adaptation, and ambiguous user intent.

Core Mechanism

DV-World consists of 260 tasks split across three tracks that mirror professional visualization lifecycles. DV-Sheet (100 tasks) requires agents to manipulate native spreadsheet environments—creating charts and dashboards directly in Excel/Sheets, and diagnosing then repairing broken visualizations. DV-Evolution (100 tasks) hands agents a reference visualization artifact (Python/R/JavaScript code or a spreadsheet) and asks them to adapt it to new data, potentially requiring cross-language translation. DV-Interact (60 tasks) pairs agents with a user simulator that provides intentionally vague requirements, forcing the agent to ask clarifying questions before implementation.

DV-World architecture:

Input task --> Agent --> Action --> Environment --> Evaluation
                 |                       |              |
                 |                       |              |
         +-------+-------+       +-------+------+   +---+---+
         |       |       |       |       |      |   |       |
     DV-Sheet DV-Evo DV-Int   Excel  Python  R/JS  TvA  MLLM
         |       |       |       |       |      |   |       |
         v       v       v       v       v      v   v       v
    Create   Adapt   Clarify  Native  Code  Code  Num  Visual
    Repair   Restr.  Align    Env.    Env.  Env.  Acc. Sem.

Think of DV-World as a flight simulator for data visualization agents. Just as pilot training moved from textbook problems to full-cockpit simulators with engine failures and weather emergencies, this benchmark moves from “generate a bar chart” to “the CFO’s dashboard broke overnight and she needs it fixed before the board meeting.” The three tracks are different flight scenarios: DV-Sheet is flying the actual aircraft (native tools like Excel), DV-Evolution is transitioning between aircraft types (adapting a Matplotlib chart to work in R), and DV-Interact is dealing with air traffic control giving unclear instructions (ambiguous user requirements). Each scenario tests whether the agent has the operational competence to handle real professional turbulence, not just textbook maneuvers.

Key Concepts

• Native environment grounding: Most benchmarks let agents write Python code that runs in isolation—a sandbox where the only tool is matplotlib and the only output is a PNG file. Native grounding means the agent must work inside Excel or Google Sheets, manipulating cells, formulas, and chart objects through the application’s actual API. It’s the difference between writing a recipe and cooking in a real kitchen with real pots and a temperamental stove. The agent can’t just output code and walk away; it must navigate the tool’s constraints, handle its quirks, and produce artifacts that live in the user’s actual workspace.

• Cross-platform evolution: A company’s visualization assets accumulate over years in different languages and tools—Python scripts from the data science team, R code from statisticians, JavaScript dashboards from web developers, Excel templates from analysts. When new data arrives, someone must adapt these artifacts to fit. Cross-platform evolution tests whether an agent can take a Matplotlib chart and rewrite it in ggplot2, or convert an Excel dashboard to a Plotly web app, preserving the visual semantics while navigating completely different APIs and paradigms. It’s like asking a translator to not just convert words but maintain the rhetorical structure across languages with different grammars.

• Proactive intent alignment: Real users don’t arrive with perfect specifications. They say “make the sales chart more insightful” or “the colors look off.” A passive agent asks “what do you want?” and gets stuck. A proactive agent recognizes ambiguity, generates hypotheses about what the user might mean (do they want a different chart type? different aggregation? different color scheme?), and asks targeted questions to narrow the space. The user simulator in DV-Interact mimics this by withholding information until the agent asks the right questions—testing whether the agent can drive the conversation toward clarity rather than waiting to be spoon-fed requirements.

Framework Shift

Before (mainstream approach):        After (this paper):

User intent (perfect)                User intent (ambiguous)
       |                                     |
       v                                     v
   Specification                      Clarification loop
       |                                     |
       v                                     v
   Code sandbox                       Native environment
       |                              (Excel/Sheets/Code)
       v                                     |
   Generate chart                            v
       |                              Create/Repair/Adapt
       v                                     |
   PNG output                                v
       |                              Artifact in workspace
       v                                     |
   Visual similarity check                   v
                                      Numerical + semantic eval
                                      (TvA + MLLM rubrics)

Single-shot generation          <->   Full lifecycle workflow
Isolated sandbox                <->   Real tool integration
Assumed perfect intent          <->   Negotiated requirements

From isolated chart generation to integrated workflow simulation, the core shift is treating visualization as a professional practice with messy inputs and native tool constraints, not a clean code-generation problem.

Expert Assessment

Problem choice: This is a real gap. Anyone who’s worked in enterprise analytics knows the chasm between “write code to make a chart” and “fix the CFO’s broken dashboard.” The authors correctly identify that existing benchmarks test a narrow slice of visualization competence. The three-track design (native tools, adaptation, intent alignment) maps cleanly to actual pain points in professional workflows. Not manufactured.

Method maturity: The benchmark design is solid but the evaluation framework shows seams. Table-value Alignment (TvA) for numerical precision is straightforward. MLLM-as-a-Judge with rubrics for visual semantics is pragmatic but introduces noise—the paper doesn’t deeply validate whether GPT-4’s aesthetic judgments align with human experts. The user simulator for DV-Interact is rule-based, which limits realism. A more mature approach would include human-in-the-loop validation of the simulator’s behavior and inter-rater reliability studies for the MLLM judge.

Experimental integrity: Baselines are fair—they test Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, and open models like Qwen2.5-Coder. The <50% overall performance is striking and credible given the task difficulty. However, the paper doesn’t report variance across runs or provide error analysis showing *why models fail (is it API misuse? planning failures? inability to parse ambiguous intent?). The rubrics for MLLM evaluation are shown but not validated against human judgments. Numbers hold up but could use more scrutiny on failure modes.

Writing quality: The paper front-loads motivation well but rushes through technical details. Section 3 (benchmark construction) would benefit from concrete task examples earlier—readers shouldn’t have to wait until Section 4 to see what a DV-Sheet task actually looks like. The evaluation section conflates description of metrics with presentation of results, making it hard to parse. Rewriting Section 4 to separate “what we measure” from “what we found” would clarify the narrative. The related work section is thorough but could be tightened.

Verdict: weak accept — Real problem, useful benchmark, but evaluation framework needs validation and failure analysis is shallow. The contribution is the benchmark itself, not methodological innovation, and it succeeds at exposing a genuine capability gap in current models.

Takeaways

Practitioners building AI agents for data work can steal three things:

1. Native environment testing: Stop evaluating agents only on code generation. If your users work in Excel, test whether the agent can actually manipulate Excel, not just output Python that would theoretically work. The gap between “generates correct code” and “operates the tool” is larger than you think.

2. Clarification loops as a first-class capability: Don’t assume users will provide perfect specifications. Build and test the agent’s ability to recognize ambiguity and ask targeted questions. The DV-Interact track’s user simulator pattern (withhold information, reward good questions) is a template for testing this.

3. Cross-platform adaptation as a distinct skill: If your organization has visualization assets in multiple languages/tools, test whether agents can translate between them while preserving semantics. This is harder than single-language generation and requires different evaluation (not just “does it run?” but “does it convey the same information?”).

The broader lesson: benchmarks that ignore the messy reality of professional workflows will produce agents that fail in production. DV-World’s value is showing what “messy reality” looks like for data visualization.