Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal LLMs

Paper: 2606.26079
Authors: Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli
Categories: cs.CL, cs.CV, cs.LG

The Gap

Existing MLLM benchmarks (e.g., MMBench, LLaVA-Bench) score each test item on exactly one canonical ordering. They treat the model’s answer as a static property of the content — if a question has four options, only one permutation is ever shown. But emerging evaluation guidelines (e.g., HELM, BIG-bench) demand “order invariance” as a baseline reliability property. The gap? No prior work systematically measures whether shuffling order-irrelevant dimensions (option order, evidence chunk order, document rank, image set order, mixed modality order) changes the answer. This paper fills that gap by building a dedicated audit framework.

[ASCII logic topology: Problem -> Assumption -> Method -> Evidence -> Conclusion]

Problem: "Does order irrelevant to content change MLLM answers?"
   |
   v
Assumption in prior work: "Order does not matter (so single ordering suffices)"
   |
   v
Method: Facet-Probe -- audit 5 order facets across 18 models
   |
   v
Evidence: Bayesian IRT separates ordering noise from per-facet bias; same-ordering control estimates decoder floor. Flip rates 24-50%.
   |
   v
Conclusion: None of 18 models is order-invariant. Prompt mitigation is modality-conditional and insufficient. Propose cross-ordering flip rate as standard metric.

The Increment

One sentence: Before this paper, the field assumed order invariance was safe to ignore; after this paper, we know that every MLLM audited fails it badly, and simple prompt fixes don’t transfer across modalities.

Core Mechanism

Facet-Probe operates in three layers: (1) Facet generation — for each test item, it creates multiple orderings across five facets: option order (permute answer choices), evidence-chunk order (reorder paragraphs in the context), document-rank order (reorder retrieved documents in RAG settings), image-set order (reorder images in a multi-image input), and mixed-modality order (e.g., text-image alternation). (2) Bayesian item-response model — fitted per model per facet, it estimates two latent parameters per item: a “true” correctness score, and a per-facet sensitivity score that captures how much the answer flips when that facet is perturbed. The model accounts for random noise and item difficulty. (3) Same-ordering control — running a Gemini model at temperature 0 on identical inputs multiple times isolates the decoder’s intrinsic stochasticity floor. Any flip rate above that floor is attributed to order sensitivity.

[ASCII diagram of method internal: components, data flow, operations]

  Base Item (canonical order)
         |
         v
   +-----------+     +-----------+
   | Facet Gen |<----| 5 Facets  |
   +-----------+     +-----------+
         |  (e.g., option shuffle, chunk reorder...)
         v
   +-------------------+
   | Multiple orderings |
   +-------------------+
         |
         +---------+---------+---------+---------+---------+
         |  Opt    |  Chunk  |  Doc    |  Img    |  Mixed  |
         +---------+---------+---------+---------+---------+
              |         |         |         |         |
              v         v         v         v         v
   +---------------------------------------------------------+
   |     Bayesian Item-Response Model                        |
   |     (per model per facet)                               |
   |     Input: answer correctness across orderings          |
   |     Output: true score + per-facet sensitivity          |
   +---------------------------------------------------------+
              |
              v
   +---------------------------------------------------------+
   |  Same-ordering control (Gemini temp=0, repeat 10x)     |
   |  = decoder noise floor                                 |
   +---------------------------------------------------------+
              |
              v
   Compare: observed flip rate vs floor -> ordering excess

Now let me explain using a load-bearing structural metaphor: Think of each MLLM as a judge in a courtroom. The evidence presented to the judge is a fixed set of documents, images, and options. But the clerk (the model’s input pipeline) has the discretion to rearrange the order of things: the order of witness statements (evidence chunks), the order of exhibits (documents), the placement of photos (image set), the arrangement of the multiple-choice options, and even whether testimony is given in text-then-image or image-then-text (mixed modality). A good, order-invariant judge should deliver the same verdict regardless of which clerk arranged the papers. Facet-Probe is like a contracted audit firm that hires the same judge multiple times, each time giving the same evidence but with the clerk using a different rearranging rule for one facet at a time. The audit then uses a statistical model (like a judge’s “predisposition thermometer”) to separate how much of the verdict flips are due to the judge’s inherent inconsistency (noise) versus how much they are systematically swayed by the ordering (facet bias). Finally, the auditor runs a calibration test — they ask the judge the exact same case twice in the exact same arrangement to see how often the judge contradicts themselves purely from internal randomness (the decoder floor). If the flip rates from the rearrangement tests are much higher than that baseline, then the judge truly cannot be trusted to be order-invariant.

Key Concepts

• Order sensitivity (flip rate): For a given model and facet, the proportion of test items where changing only the ordering (while keeping all content identical) leads to a different answer. Example: A question has four answer options. You show the model the options in order A,B,C,D and it answers “A”. You then permute the options to D,C,B,A and the model now answers “C”. That’s a flip. The flip rate is the percentage of items where such a change occurs. This measures how much the model’s decision depends on superficial arrangement rather than content.

• Bayesian Item-Response Theory (IRT): A statistical framework originally from educational testing, used here to separate two sources of variation: the item’s intrinsic difficulty (does the model get it right regardless of ordering?) and the facet’s bias (does a particular ordering of that facet systematically increase or decrease correctness?). The model learns a latent “true correctness” for each item, plus a per-facet shift parameter. This is crucial because raw flip rate conflates random noise with systematic biases; IRT teases them apart.

• Cross-ordering flip rate: The metric proposed as a standard reporting axis. It is the fraction of test items where the model’s answer changes across two random orderings of a specified facet (or across all facets jointly). Unlike a single-score accuracy, this captures reliability. The authors argue that every MLLM leaderboard should include this number alongside accuracy.

Framework Shift

Before (mainstream approach):        After (this paper):
+===================================+  +===================================+
| Single canonical ordering per     |  | Multiple orderings per facet      |
| test item.                        |  | (5 facets: opt, chunk, doc, img,  |
|                                   |  |  mixed).                          |
| Accuracy = correct / total        |  | Accuracy + cross-ordering flip    |
|                                   |  | rate.                             |
| Order sensitivity is ignored.     |  | Order sensitivity is a first-    |
|                                   |  | class property.                   |
| Conclusion: Model A 85% accurate  |  | Conclusion: Model A 85% accurate  |
| is good.                          |  | but flips on 40% of items under   |
|                                   |  | option shuffling — not good.      |
| If prompted with "think step by   |  | Prompt mitigation is modality-    |
| step", maybe order robustness     |  | conditional: fixes for text don't |
| improves.                         |  | transfer to image ordering.       |
+===================================+  +===================================+

One sentence: From treating order as irrelevant background noise to elevating it as a mandatory, measurable reliability dimension, the core shift is recognizing that shuffling what shouldn’t matter reveals hidden fragility.

Expert Assessment

Problem choice: Real gap. The field has been chasing top-of-the-leaderboard accuracy while ignoring elementary robustness. This paper identifies a blind spot that has been there since the first multimodal benchmark, and it matters because real-world deployment cannot guarantee a canonical ordering. The problem is at the right maturity level — now that we have capable MLLMs, it’s time to stress-test them.

Method maturity: Clever insight deployed with appropriate machinery. The Bayesian IRT is a good fit — it’s not overkill, and the same-ordering control is a simple but crucial sanity check. Simpler alternatives (e.g., just reporting raw flip rate) exist but would conflate noise with bias; the IRT separation adds genuine value. No red flags in design.

Experimental integrity: Solid. 18 models — frontier (GPT-4V, Gemini, Claude) and open-weight — provide comprehensive coverage. The same-ordering control with temperature 0 is a nice baseline. One concern: the mitigation experiments are limited to Gemini and a few prompt strategies; the conclusion that “prompt-level mitigation alone is unlikely to succeed” might be stronger if tested across more models and more prompt variants. But as a first audit, this is thorough. No obvious cherry-picking.

Writing quality: Clear and well-structured. The five-facet breakdown is logical, and the Bayesian modeling section is explained sufficiently for an ML audience. Where it cuts corners: the abstract claims “none of the 18 MLLMs we audit are order-invariant” but the reader must search for the exact flip rate breakdown by model in the appendix. The main paper could show a single summary table more prominently. If the authors rewrite to put the flip rates per model and per facet front and center (maybe Figure 1), the paper’s impact would increase dramatically.

Verdict: Strong accept — The problem is real, the methodology is sound, the results are alarming, and the proposed metric is actionable. Every MLLM practitioner should read this and add “cross-ordering flip rate” to their evaluation checklist.

Takeaways

Three concrete things to steal:

1. The five facet taxonomy — when auditing any system’s input order robustness, use this taxonomy (option, evidence chunk, document rank, image set, mixed modality) as a checklist. It generalizes beyond MLLMs to any model that consumes multiple input elements.

2. The same-ordering control trick — run the same input multiple times (temperature 0) to measure the decoder noise floor. Then any flip rate above that floor must come from ordering changes. This is a cheap, universally applicable sanity check.

3. The Bayesian IRT separation — if you have repeated measurements per item (different orderings), treat item difficulty and ordering sensitivity as latent variables. The IRT framework (or a simpler logistic regression with random effects) lets you attribute source of variation cleanly. Steal this for any reliability diagnostic.