Paper: 2603.24582 Authors: Biplab Pal, Santanu Bhattacharya Categories: cs.AI
Abstract
The paper introduces a Markov-style, measure-based way to assess agentic AI before deployment by combining reliability checks with oversight-cost analysis. It is tested on a large enterprise procurement event log, showing that state-level support can still hide substantial uncertainty in next-step actions. The authors report that richer state definitions improve alignment between predicted autonomy and observed step accuracy.
Key Contributions
- Formalizes workflow autonomy as a sequential decision problem with reliability and human-oversight constraints
- Defines state blind-spot mass, state-action blind mass, an escalation gate, and an oversight-cost identity
- Evaluates the framework on the BPI Challenge 2019 purchase-to-pay event log
- Shows that expanding the state from 42 to 668 variables increases state-action blind mass
- Demonstrates that predicted autonomy tracks observed step accuracy within a few percentage points on held-out data
Methodology
The paper frames agentic workflows as Markovian state transitions. Instead of treating autonomy as a vague property, it defines measurable quantities for how much of the next-step action space is unsupported by the current state and how much human oversight may be needed. That makes it possible to audit a workflow before deployment rather than only after failures appear.
A key part of the analysis is the choice of state representation. The authors compare a compact 42-variable state with a much richer 668-variable one. The broader state reveals more hidden uncertainty, which is exactly the kind of gap the framework is designed to surface.
Results
The empirical study uses a large procurement process log from BPI Challenge 2019. The central finding is that a system can look well-supported at the state level while still being uncertain about what action comes next. When the state is expanded, state-action blind mass increases notably, showing that the earlier abstraction had hidden important ambiguity.
The paper also reports that the model’s maximum action probability stays close to observed autonomous step accuracy on held-out data, with an average difference of 3.4 percentage points. That suggests the framework is useful for practical pre-deployment auditing.
Takeaways
- Agentic autonomy can be audited as a sequential decision problem rather than a vague high-level property
- The choice of state representation strongly affects how much hidden uncertainty the audit reveals
- Compact states can underestimate next-step ambiguity in real workflows
- Oversight cost should be analyzed together with reliability, not separately
- The framework provides a concrete way to compare predicted autonomy with observed behavior before deployment
论文: 2603.24582 作者: Biplab Pal, Santanu Bhattacharya 分类: cs.AI
摘要
本文提出一种基于马尔可夫和测度的方法,通过结合可靠性检查与监督成本分析来评估部署前的代理式AI。作者在一个大型企业采购事件日志上进行验证,表明即使状态层面看似充分支持,下一步动作仍可能存在显著不确定性。更丰富的状态定义能够改善预测自治与实际步骤准确率之间的一致性。
主要贡献
- 将工作流自治形式化为带有可靠性和人工监督约束的序列决策问题
- 定义了state blind-spot mass、state-action blind mass、escalation gate和oversight-cost identity
- 在BPI Challenge 2019的采购到付款事件日志上进行评估
- 证明将状态从42个变量扩展到668个变量会增加state-action blind mass
- 显示在留出数据上,预测自治与实际步骤准确率可在几个百分点内对齐
方法论
论文将代理式工作流建模为马尔可夫状态转移。作者没有把自治当成一个模糊的高层属性,而是定义了可测量的量,用于表示当前状态对下一步动作空间支持了多少,以及可能需要多少人工监督。这使得在部署前审计工作流成为可能,而不仅仅是在故障发生后回顾。
分析的关键在于状态表示的选择。作者比较了一个紧凑的42变量状态和一个更丰富的668变量状态。更宽的状态揭示了更多隐藏的不确定性,这正是该框架试图暴露的差距。
结果
实证研究使用了BPI Challenge 2019的大型采购流程日志。核心发现是,一个系统在状态层面看起来支持充分,但在下一步动作上仍可能非常不确定。当状态扩展后,state-action blind mass明显上升,说明更早的抽象掩盖了重要的歧义。
论文还报告说,模型的最大动作概率在留出数据上与实际自治步骤准确率相当接近,平均差距为3.4个百分点。这表明该框架对实际的部署前审计具有参考价值。
要点总结
- 代理式自治可以被视为序列决策问题来审计,而不只是模糊的高层性质
- 状态表示的选择会显著影响审计揭示出的隐藏不确定性
- 紧凑状态可能低估真实工作流中的下一步歧义
- 监督成本应与可靠性一起分析,而不是分开处理
- 该框架为部署前比较预测自治与实际行为提供了具体方法