A Ranking Approach for Measuring Calibration

Paper: 2609.13100 Authors: Anirban Chatterjee, Rina Foygel Barber Categories: stat.ME, cs.LG, stat.ML

The Gap

A model that reports probabilities makes a stronger claim than one that reports labels, and the property that justifies the claim is calibration: the true probability of the outcome exactly matches the forecasted probability f(X). So calibration is what makes a forecasted probability mean what it says.

In practice models are never perfectly calibrated, so the question becomes how to measure the miscalibration — and the paper notes why the answer matters: it is how you assess a model’s reliability. A miscalibration number is not a diagnostic detail; it is the quantity that tells a user whether to trust a probability.

And the standard measure has a known impossibility attached to it. The Expected Calibration Error (ECE) is the most widely used, and it is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. That is a strong statement about the target, not about an estimator: without assumptions, no method can guarantee an accurate estimate of ECE. So any practical approach is necessarily estimating something else, and the question becomes what — and whether that something still tracks ECE.

   CALIBRATION IS WHAT MAKES A PROBABILITY MEAN WHAT IT SAYS

   a model reporting PROBABILITIES makes a STRONGER CLAIM than one
   reporting LABELS
     the property justifying that claim is CALIBRATION:
       the TRUE probability of the outcome EXACTLY MATCHES the
       FORECASTED probability f(X)
        |
        v
   in practice models are NEVER PERFECTLY CALIBRATED
     -> the question becomes how to MEASURE the miscalibration
     <- and the answer matters because it is how you ASSESS A MODEL'S
        RELIABILITY
        -> a miscalibration number is NOT a diagnostic detail: it is
           the quantity that tells a user WHETHER TO TRUST A PROBABILITY

   [AND THE STANDARD MEASURE HAS A KNOWN IMPOSSIBILITY ATTACHED]
     EXPECTED CALIBRATION ERROR (ECE) is the MOST WIDELY USED
       -> and it is KNOWN TO BE IMPOSSIBLE TO ESTIMATE THE ECE WITH
          GUARANTEED ACCURACY IN AN ASSUMPTION-FREE SETTING
       <- a strong statement about the TARGET, not about an ESTIMATOR:
          without assumptions, NO METHOD can guarantee an accurate
          estimate of ECE
       -> so any practical approach is NECESSARILY ESTIMATING SOMETHING
          ELSE
       -> the question becomes WHAT -- and whether that something STILL
          TRACKS ECE

The Increment

One sentence: Before this paper, practitioners approximated an unestimable quantity with binned bins; after it, a measure based on comparing neighbouring predicted probabilities provides a better proxy, with theoretical guarantees and empirical support.

Core Mechanism

The proposed measure, rankECE, is based on comparing points with neighbouring values of the predicted probability f(X). The design choice is to compare neighbours in the ordering of predictions rather than within fixed bins of the prediction value — and that choice is what the impossibility result points toward. Since ECE cannot be estimated in general, the goal is a quantity that tracks it without needing assumptions, and neighbourhood comparison is a local operation that does not require the prediction distribution to have any particular shape.

And the paper makes a comparison against what is actually used. The claim is that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice. Two aspects of that framing are worth noting. The baseline is the approximation in practice rather than a theoretical competitor, so the comparison bears directly on what one should do differently. And the quantity is described as a proxy rather than as an estimate of ECE — which is the honest framing given the impossibility, since claiming to estimate ECE would contradict the known result.

The support is both theoretical and empirical: the paper states that theoretical guarantees and empirical results establish the improvement. For a measurement problem, having guarantees matters in a specific way: a proxy that happens to work on the datasets tested is weaker than one with a guarantee about when it tracks the target, because the user of a calibration measure is deciding whether to trust probabilities, and iteration over datasets does not settle that.

   THE PROPOSED MEASURE: RANKECE
     BASED ON COMPARING POINTS WITH NEIGHBOURING VALUES OF THE PREDICTED
     PROBABILITY f(X)
       <- the design choice: compare NEIGHBOURS IN THE ORDERING OF
          PREDICTIONS rather than WITHIN FIXED BINS of the prediction
          value
       <- and that choice is WHAT THE IMPOSSIBILITY RESULT POINTS TOWARD:
          since ECE CANNOT BE ESTIMATED IN GENERAL, the goal is a
          quantity that TRACKS IT WITHOUT NEEDING ASSUMPTIONS
          -> NEIGHBOURHOOD COMPARISON is a LOCAL OPERATION that does NOT
             REQUIRE THE PREDICTION DISTRIBUTION TO HAVE ANY PARTICULAR
             SHAPE

   AND THE COMPARISON IS AGAINST WHAT IS ACTUALLY USED
     rankECE PROVIDES A BETTER PROXY FOR ECE THAN BINNED APPROXIMATIONS
     TO ECE, WHICH ARE THE MOST COMMONLY-USED APPROXIMATIONS IN PRACTICE
       <- the BASELINE is THE APPROXIMATION IN PRACTICE rather than a
          THEORETICAL COMPETITOR
          -> the comparison bears DIRECTLY on WHAT ONE SHOULD DO
             DIFFERENTLY
       <- the quantity is described as a PROXY rather than as an
          ESTIMATE of ECE
          <- the HONEST FRAMING given the IMPOSSIBILITY: claiming to
             ESTIMATE ECE WOULD CONTRADICT THE KNOWN RESULT

   THE SUPPORT IS BOTH THEORETICAL AND EMPIRICAL
     theory AND experiments establish the improvement
       <- for a MEASUREMENT problem, having GUARANTEES matters in a
          SPECIFIC way:
          a proxy that HAPPENS TO WORK ON THE DATASETS TESTED is WEAKER
          than one with a GUARANTEE ABOUT WHEN IT TRACKS THE TARGET
       <- because the user of a calibration measure is DECIDING WHETHER
          TO TRUST PROBABILITIES, and ITERATION OVER DATASETS DOES NOT
          SETTLE THAT

Think of it as grading a set of archery targets by comparing each archer’s neighbours rather than by drawing arbitrary scoring rings. The standard approach draws rings at fixed distances and counts hits per ring — and its accuracy depends on how the rings happen to fall relative to where the arrows landed, which is a property of your binning, not of the archers. If you instead compare each archer to those who landed closest to them, you avoid the rings entirely: the comparison is local, it needs no assumption about where arrows tend to cluster, and it still tells you whether similar predictions come with similar outcomes. The paper frames the result as a better proxy rather than a better measurement for the same reason you would be careful with the analogy: the underlying quantity is not estimable in general, so what you can honestly claim is a relationship.

Key Concepts

• Calibration as the property that makes probabilities meaningful: the true outcome probability equals the forecast probability. Measuring its failure is what tells a user whether to trust the number.
• The ECE impossibility: no assumption-free guarantee of accurate estimation. It reframes the task from estimation to finding a proxy that tracks the target.
• Neighbourhood comparison instead of bins: comparing adjacent predictions in the ordering, which is local and requires no assumption about the prediction distribution’s shape.
• The practical baseline: binned approximations, the most commonly used approach. Comparing against what is actually deployed makes the result actionable.
• Proxy rather than estimate: the honest framing, since claiming to estimate ECE would contradict the known impossibility.
• Guarantees alongside experiments: for a measurement problem, a guarantee about when the proxy tracks ECE is what allows a trust decision, which data iteration alone cannot provide.

Framework Shift

Before (binned approximations of an unestimable target):
  measure miscalibration with ECE
  -> ECE cannot be estimated with guaranteed accuracy without assumptions
  -> practice approximates it with fixed bins
  -> bin boundaries shape the answer, and the approximation's fidelity
     depends on the prediction distribution

After (neighbourhood comparison with guarantees):
  rankECE compares points with neighbouring predicted probabilities
  -> local, needs no assumption about the distribution's shape
  -> a better proxy than binned approximations, theoretically and
     empirically
  -> framed as a proxy, not as an estimate of ECE

From approximating an unestimable quantity with fixed bins, to tracking it with a local comparison that carries guarantees, the core shift is that when the target cannot be estimated in general, the honest goal is a related quantity whose relationship to the target can be stated.

Expert Assessment

Problem choice: Excellent, and it targets a measure whose limitations are known but whose consequences have been under-managed. ECE is ubiquitous, and the impossibility result means every use of it is a use of an approximation — so improving the approximation is a contribution to essentially every calibration claim in the literature.

Method maturity: The design follows from the impossibility in the right way: if the target is not estimable, find a quantity that tracks it without assumptions, which is what motivates comparing neighbourhoods rather than fixed bins. Describing the result as a proxy rather than an estimate is the correct and non-obvious framing, since the tempting claim would have contradicted the paper’s own premise. Providing theoretical guarantees alongside empirical results is what makes the proxy usable for a trust decision rather than merely promising on tested datasets.

Experimental integrity: Choosing binned approximations as the baseline is the right comparison, because it is what practitioners use and therefore what the result would change. The limitation is that the guarantee is about the proxy’s relationship to ECE, not about recovering ECE itself — so a reader still cannot obtain a guaranteed ECE estimate, and the value of rankECE depends on how well the proxied relationship transfers to the regime of interest, such as heavily imbalanced outcomes or very small calibration samples. The paper is clear about framing the contribution as a proxy, which keeps the scope honest.

Writing quality: The contrast between bins and neighbourhoods is stated compactly and carries the intuition, which is what makes the method easy to grasp before the theory. Because the practical reader wants to know what to compute, a short statement of how rankECE is computed from a sample — the minimal recipe — would let someone adopt it without reconstructing it from the guarantees.

Verdict: accept — it takes a known impossibility for the field’s standard calibration measure and contributes a proxy that is better founded and better performing, framed honestly as a proxy rather than as the measurement it cannot be.

Takeaways

• Check whether your metric is estimable at all. A quantity that cannot be estimated without assumptions should be tracked by a proxy whose relationship to it is stated.
• Prefer local comparisons to fixed bins when the distribution is unknown. Bin boundaries shape the answer in ways that have nothing to do with what you are measuring.
• Compare against what is actually used. Reporting an improvement over the practical baseline is what makes a measurement contribution actionable.
• Watch the framing of an impossibility. Saying “proxy” instead of “estimate” is the difference between a sound claim and one that contradicts its own premise.