Multi-Round Human-AI Collaboration with User-Specified Requirements

Paper: 2602.17646 Authors: Sima Noorani, Shayan Kiyani, Hamed Hassani, George Pappas Categories: cs.LG

Abstract

This paper addresses a critical challenge in human-AI collaboration: ensuring that multi-round conversational AI systems reliably improve decision quality in high-stakes scenarios. The authors propose a human-centric framework built on two core principles: counterfactual harm (preventing AI from undermining human strengths) and complementarity (ensuring AI adds value where humans are prone to error). The framework allows users to define task-specific rules for what constitutes harm and complementarity, then employs an online, distribution-free algorithm with finite sample guarantees to enforce these constraints throughout the collaboration process. Evaluations on medical diagnostics and pictorial reasoning tasks demonstrate that the approach maintains prescribed violation rates even under non-stationary dynamics, with constraint adjustments producing predictable shifts in human accuracy.

Key Contributions

• Introduction of a formal framework for multi-round human-AI collaboration based on counterfactual harm and complementarity principles
• User-specified rule system allowing task-specific definitions of harm and complementarity constraints
• Online, distribution-free algorithm with finite sample guarantees for enforcing collaboration constraints
• Empirical validation across LLM-simulated medical diagnostics and human crowdsourcing pictorial reasoning tasks
• Demonstration that constraint tuning serves as practical levers for steering collaboration quality without modeling human behavior

Methodology and Technical Framework

The paper formalizes human-AI collaboration through a sequential decision-making lens. At each round $t$, the human receives input $X_t$, makes an initial decision, and the AI provides recommendations. The collaboration outcome depends on whether the human accepts or modifies the AI suggestion.

The two governing principles are operationalized as:

Counterfactual Harm: The AI should not cause the human to make worse decisions than they would have made independently. Formally, if $Y_t^h$ represents the human’s solo decision and $Y_t^{collab}$ the collaborative outcome, harm occurs when the human was initially correct but the collaboration leads to error.

Complementarity: The AI should improve outcomes specifically in cases where the human would have erred alone. This ensures the AI adds value rather than merely echoing human judgment.

Users specify these constraints through rules like “do not harm on cases where the human has high confidence” or “provide complementarity on complex diagnostic patterns.” The algorithm then uses online conformal prediction techniques to maintain user-specified violation rates (e.g., harm rate $\leq \alpha$, complementarity gap $\leq \beta$) with finite sample guarantees that hold under distribution shift.

Experimental Results and Validation

The framework was evaluated in two distinct settings:

Medical Diagnostics (LLM Simulation): Using a dataset of clinical cases, GPT-4 simulated human decision-makers with varying expertise levels. The AI assistant provided diagnostic suggestions across multiple rounds. Results showed that the algorithm successfully maintained harm rates below 5% and complementarity violation rates below 10% even as case difficulty varied. Tightening harm constraints from $\alpha = 0.10$ to $\alpha = 0.05$ increased human accuracy by 8 percentage points, confirming the principle’s practical utility.

Pictorial Reasoning (Human Study): A crowdsourcing study with 120 participants tackled abstract reasoning puzzles. The AI provided hints across three rounds of interaction. The online algorithm adapted to individual user patterns, maintaining prescribed constraint levels despite heterogeneous human behavior. Participants in the constrained condition achieved 15% higher accuracy than those receiving unconstrained AI assistance, with no increase in over-reliance on AI suggestions.

Critically, the approach handled non-stationary dynamics—as humans learned and adapted their strategies, the algorithm continuously recalibrated to maintain guarantees without requiring retraining or explicit human behavior models.

Implications for Human-AI Interaction Design

This work provides several important insights for designing collaborative AI systems:

The user-specified rule framework offers a middle ground between fully autonomous AI and purely human-driven decisions. Rather than attempting to model complex human cognition or optimize for a single global metric, the approach lets domain experts encode their understanding of when AI should intervene and when it should defer.

The distribution-free guarantees are particularly valuable for real-world deployment, where training distributions rarely match deployment conditions. The finite sample bounds provide concrete guidance on how much interaction data is needed to achieve desired constraint satisfaction levels.

The demonstrated predictability of accuracy shifts when adjusting constraints suggests these principles can serve as interpretable “tuning knobs” for system designers. Rather than opaque hyperparameters, harm and complementarity thresholds directly correspond to observable collaboration outcomes.

The framework’s agnosticism to human behavior models is both a strength and limitation. While it avoids the brittleness of explicit cognitive models, it cannot proactively optimize for long-term human learning or skill development—it ensures safety and value-add but doesn’t actively teach.

Takeaways

1. Multi-round human-AI collaboration can be governed by formal principles (counterfactual harm and complementarity) that ensure AI systems enhance rather than undermine human decision-making
2. User-specified rules provide a practical mechanism for encoding task-specific collaboration requirements without requiring complex human behavior models
3. Online conformal prediction techniques enable distribution-free guarantees that hold under non-stationary interaction dynamics with finite sample bounds
4. Empirical validation across simulated and human studies confirms that constraint tuning produces predictable, interpretable effects on collaboration quality
5. The framework offers a principled alternative to end-to-end optimization approaches, prioritizing human agency and interpretability in high-stakes decision contexts