MM-CondChain: A Benchmark That Finally Tests Whether Vision-Language Models Can Actually Follow Conditional Logic

Paper: 2603.12266 Authors: Haozhan Shen, Shilin Yan, Hongwei Xue, Shuaiqi Lu, Xiaojun Tang, Guannan Zhang, Tiancheng Zhao, Jianwei Yin Categories: cs.CV

The Gap

Existing multimodal benchmarks — VQA, MMBench, MMStar, and their cousins — are fundamentally single-shot: you look at an image, you answer a question. Even the ones that claim “compositional reasoning” mostly mean “the question has two parts.” What they don’t test is whether a model can evaluate a chain of conditional statements where each step depends on visual evidence, and where getting step 2 wrong means the entire execution path diverges.

This matters because real-world visual workflows — GUI automation, document processing, robotic task planning — are structured as conditional programs: “if X is visible AND Y has property Z, then do A, else do B.” Prior benchmarks treat each condition as independent and shallow. They don’t test depth (chained dependencies), they don’t test hard negatives (conditions that look true but aren’t), and they don’t verify correctness programmatically — they rely on human annotation that doesn’t scale.

Problem: MLLMs deployed in workflows need conditional visual reasoning
    |
    v
Prior benchmarks: single-step QA or shallow independent conditions
    |
    v
Assumption: deep chained conditionals require a different evaluation paradigm
    |
    v
Method: MM-CondChain -- layered conditions + VPIR + agentic synthesis
    |
    v
Evidence: best model scores 53.33 Path F1, drops sharply with depth
    |
    v
Conclusion: deep compositional reasoning is a real, unsolved capability gap

The Increment

One sentence: Before this paper, we had no way to rigorously test whether a vision-language model can follow a branching conditional program grounded in visual evidence — now we do, and the answer is: not very well.

Core Mechanism

The benchmark is built around what the authors call a “multi-layer reasoning chain.” Each instance is a small conditional program: layer 1 checks some visual condition (e.g., “is there a dialog box AND is its border color red?”), and the result — true or false — determines which branch layer 2 evaluates, and so on. The final answer isn’t just a label; it’s the execution path the model took through the chain. This is evaluated with Path F1, which gives partial credit for getting some layers right even if the final outcome is wrong.

The construction pipeline has three stages. A Planner LLM generates compositional conditions layer by layer, making sure each condition involves multiple visual predicates (object presence, attribute value, spatial relation). Each condition is then compiled into a Verifiable Programmatic Intermediate Representation (VPIR) — essentially a structured logical expression that can be mechanically checked against ground truth. A Composer then assembles the verified layers into a natural-language instruction that reads like a real workflow step. This pipeline runs across three domains: natural images (COCO-style), data charts, and GUI screenshots.

The hard negative construction is worth noting separately. For each true condition, the pipeline generates a near-miss variant — same structure, slightly wrong attribute or object — so models can’t shortcut by pattern-matching the surface form of the question.

Image + Domain
    |
    v
[Planner LLM]
    |  generates layer-by-layer compositional conditions
    v
[VPIR Compiler]
    |  condition -> structured logical expression
    |  mechanical verification against ground truth
    v
[Composer LLM]
    |  assembles verified layers into natural-language instruction
    v
Benchmark Instance:
  Layer 1: condition_A (AND/OR predicates) -> True/False
      |
      +-- True  -> Layer 2a: condition_B -> True/False -> ...
      |
      +-- False -> Layer 2b: condition_C -> True/False -> ...
                                              |
                                              v
                                         Final outcome + execution path

Think of it like a building inspection checklist that branches. An inspector walks into a building and checks: “Is the fire exit clearly marked AND is the sprinkler system green-tagged?” If yes, move to the structural check. If no, flag it and go to the remediation branch. The inspector’s job isn’t just to answer yes/no at the end — it’s to follow the right path through the checklist based on what they actually see. The Planner is the person who writes the checklist. The VPIR is the legal standard that defines what “clearly marked” means precisely enough to be audited. The Composer turns the legal standard back into plain English for the inspector. And Path F1 scores not just whether the inspector reached the right final verdict, but whether they took the right path to get there.

Key Concepts

• Verifiable Programmatic Intermediate Representation (VPIR): The core reliability trick. Instead of asking humans to judge whether a generated condition is correct, the pipeline compiles each condition into a formal logical expression (think: AND(EXISTS(dialog_box), EQUALS(border_color, red))). This expression can be evaluated mechanically against annotated ground truth. It’s the difference between asking “does this sentence sound right?” and running a unit test. The VPIR is what makes the pipeline scalable — you can generate thousands of conditions and verify them without human review of each one.

• Path F1: Standard accuracy would just check the final answer. But in a branching chain, a model might get the final answer right by accident (wrong path, lucky outcome) or get it wrong despite correct intermediate reasoning. Path F1 treats the execution path as a sequence and computes F1 over the sequence of correct layer evaluations. This gives a much more honest picture of whether the model is actually reasoning or just guessing at the end.

• Hard Negatives: For each condition in the benchmark, there’s a near-miss version — same visual scene, same question structure, but one predicate is subtly wrong (e.g., the dialog exists but the color is blue, not red). This forces models to actually evaluate each predicate rather than relying on heuristics like “dialogs usually appear with permission requests.” The sharp performance drop on hard negatives in the experiments is the most telling result in the paper.

Framework Shift

Before (mainstream approach):        After (this paper):

Image -> [Single Question]           Image -> [Layer 1 Condition]
             |                                      |
             v                              True /     \ False
         [Answer]                            /             \
                                    [Layer 2a]         [Layer 2b]
Image -> [Multi-part Question]              |               |
             |                          True/False      True/False
             v                              \               /
         [Answer A] + [Answer B]             \             /
         (independent, no branching)          v           v
                                         [Execution Path + Final Outcome]
                                         evaluated by Path F1

From flat question-answering to branching conditional execution, the core shift is: the model’s output is now a program trace, not a label.

Expert Assessment

Problem choice: This is a real gap, not a manufactured one. The motivation from GUI automation and visual workflows is genuine — anyone who has tried to use an MLLM to automate a multi-step UI task knows that conditional branching is exactly where these models fall apart. The paper sits at a natural intersection of program synthesis, visual grounding, and agent evaluation, which is where the field is clearly heading. Good timing.

Method maturity: The VPIR idea is the clever bit — using a formal intermediate representation to decouple generation from verification is a solid engineering insight that transfers well. The agentic synthesis pipeline is more “sensible engineering” than “novel insight,” but it works. The concern is that the VPIR expressiveness is never fully characterized — we don’t know what class of conditions it can and can’t represent, which matters if you want to extend this to more complex domains.

Experimental integrity: The baselines are reasonable — they test a range of open and closed models including GPT-4V class systems. The 53.33 Path F1 ceiling for the best model is a credible result, not suspiciously low or high. The ablations on depth and predicate complexity are the right experiments to run. One flag: the paper doesn’t report inter-annotator agreement for the human-verified portions of the pipeline, which makes it hard to assess how much noise is in the ground truth. Also, the three domains (natural images, charts, GUIs) are evaluated somewhat separately — a unified analysis of cross-domain transfer would strengthen the claims.

Writing quality: The pipeline description is the weakest section. The Planner/VPIR/Composer breakdown is clear in concept but the paper spends too much space on implementation details that belong in an appendix, and not enough on the formal properties of VPIR (what can it express? what are its limits?). Rewriting the VPIR section to include a formal grammar or at least a worked example with the actual logical expression would make the paper significantly more rigorous and reusable.

Verdict: weak accept — the benchmark fills a real gap and the VPIR construction trick is genuinely useful, but the paper undersells its own theoretical contribution and the experimental analysis stops just short of the depth needed to be fully convincing.

Takeaways

The VPIR pattern is the most transferable idea here. If you’re building any kind of synthetic benchmark where you need to verify that generated conditions are correct without human review of every instance, the “compile to a formal representation, verify mechanically, then render back to natural language” pipeline is a clean pattern worth stealing. It’s essentially test-driven data generation.

Path F1 as an evaluation metric for branching reasoning tasks is also worth adopting. If you’re evaluating any model on a task that has intermediate steps with dependencies, scoring only the final output throws away signal. Path F1 (or any sequence-level metric over the execution trace) gives you a much more diagnostic picture.

The hard negative construction strategy — generate near-miss variants by perturbing one predicate — is a simple but effective way to stress-test whether models are actually evaluating conditions or pattern-matching. Easy to adapt to other grounding tasks.