Model Agreement via Anchoring: A Unified Framework for Reducing Prediction Disagreement

Paper: 2602.23360 Authors: Eric Eaton, Surbhi Goel, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica Sorrell Categories: cs.LG, cs.AI

Abstract

This work addresses the fundamental problem of model disagreement in machine learning—when two models trained independently on different samples produce different predictions. The authors introduce a novel anchoring technique that provides theoretical bounds on the expected squared difference between predictions of independently trained models. The key insight is to anchor the analysis to the average of two models, enabling disagreement bounds for four widely-used algorithms: stacked aggregation, gradient boosting, neural architecture search, and regression trees. Remarkably, the framework shows that disagreement can be systematically reduced to zero by tuning natural parameters like ensemble size, iteration count, architecture size, or tree depth.

Key Contributions

• A general anchoring technique for bounding independent model disagreement without requiring coordination between training processes
• Theoretical disagreement bounds for stacked aggregation that vanish as $O(1/k)$ with the number of stacked models $k$
• Disagreement analysis for gradient boosting showing convergence with iteration count $k$
• Architecture search bounds demonstrating disagreement reduction with search space size $n$
• Regression tree disagreement bounds that decrease with tree depth $d$
• Generalization from one-dimensional squared loss to multi-dimensional regression with arbitrary strongly convex losses

The Anchoring Methodology

The core innovation lies in the anchoring approach. Rather than directly comparing two independently trained models $f_1$ and $f_2$, the authors introduce their average $\bar{f} = (f_1 + f_2)/2$ as an anchor point. The disagreement can then be decomposed as:

$\mathbb{E}[(f_1(x) - f_2(x))^2] = \mathbb{E}[(f_1(x) - \bar{f}(x))^2] + \mathbb{E}[(f_2(x) - \bar{f}(x))^2]$

This decomposition is powerful because each model’s deviation from the average can be bounded using standard learning-theoretic tools. The anchor $\bar{f}$ serves as a stable reference point that both models implicitly converge toward as training parameters increase.

For stacked aggregation, if each base model has disagreement bounded by $\Delta$, stacking $k$ models reduces disagreement to approximately $\Delta/k$. For gradient boosting, the iterative refinement process naturally drives both models toward similar solutions. In neural architecture search, larger search spaces increase the probability that both training runs discover similar high-performing architectures. For regression trees, deeper trees provide finer-grained partitioning that reduces prediction variance.

Algorithmic Applications

Stacked Aggregation: The paper proves that for any base model class, stacking $k$ independently trained models reduces disagreement by a factor of $k$. This provides theoretical justification for ensemble methods and explains why bagging and similar techniques improve model stability.

Gradient Boosting: The analysis shows that as boosting iterations increase, both models refine their predictions in similar directions, driven by the loss gradient. The disagreement bound decreases with the number of boosting rounds, explaining the empirical stability of gradient boosting methods like XGBoost and LightGBM.

Neural Architecture Search: When searching over architectures of size $n$, the probability that two independent searches discover similar optimal architectures increases with $n$. This provides insight into why larger architecture spaces can lead to more reproducible results despite the non-convex optimization landscape.

Regression Trees: For trees of depth $d$, the disagreement bound decreases exponentially with depth. Deeper trees create finer partitions of the input space, reducing the variance in predictions within each leaf node and thus reducing disagreement between independently trained trees.

Theoretical Implications

The anchoring framework reveals a fundamental connection between model capacity, training dynamics, and prediction stability. The results suggest that model disagreement is not merely a nuisance but a quantifiable property that can be controlled through principled algorithm design.

The generalization to strongly convex losses beyond squared error demonstrates the robustness of the anchoring technique. This suggests the methodology could extend to classification problems, structured prediction, and other learning settings where disagreement is a concern.

From a practical standpoint, these bounds provide guidance on hyperparameter selection. If low disagreement is desired—for instance, in safety-critical applications or when model predictions must be auditable—practitioners can tune parameters like ensemble size or tree depth with theoretical guarantees on the resulting stability.

Takeaways

1. Model disagreement can be systematically controlled through natural training parameters without requiring coordination between training runs
2. The anchoring technique provides a unified analytical framework applicable to diverse ML algorithms
3. Stacked aggregation reduces disagreement by $O(1/k)$ with ensemble size, providing theoretical backing for ensemble methods
4. Gradient boosting naturally drives models toward agreement through iterative refinement
5. Larger architecture search spaces and deeper regression trees both reduce prediction disagreement
6. The framework generalizes beyond squared loss to any strongly convex loss function, suggesting broad applicability