Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive

Paper: 2602.23239 Authors: Radha Sarma Categories: cs.AI, cs.CY

Abstract

This paper presents a fundamental incompatibility proof: optimization-based AI systems, particularly Large Language Models trained via Reinforcement Learning from Human Feedback (RLHF), cannot be governed by norms due to architectural constraints. The author establishes two necessary and jointly sufficient conditions for genuine agency: Incommensurability (maintaining non-negotiable boundaries rather than tradeable weights) and Apophatic Responsiveness (a non-inferential mechanism to suspend processing when boundaries are threatened). RLHF systems violate both conditions by design—their core operations of unifying values on scalar metrics and selecting highest-scoring outputs preclude normative governance. This incompatibility explains documented failure modes like sycophancy and hallucination as structural manifestations rather than correctable bugs. The paper introduces the Convergence Crisis: when humans verify AI outputs under metric pressure, they degrade into optimizers themselves, eliminating normative accountability from the system entirely.

Key Contributions

• Formal incompatibility proof: Demonstrates that optimization-based architectures are constitutively incompatible with norm-responsiveness, not merely poorly trained
• Two-condition framework for agency: Establishes Incommensurability and Apophatic Responsiveness as necessary and sufficient architectural requirements for genuine agency
• Substrate-neutral specification: Provides architectural criteria applicable to biological, artificial, and institutional systems
• Convergence Crisis concept: Identifies second-order risk where human verifiers degrade into optimizers under metric pressure
• Reframing of failure modes: Shows sycophancy, hallucination, and unfaithful reasoning as structural manifestations rather than training artifacts

The Architecture of Agency

The paper’s central argument rests on distinguishing between instruments (sophisticated tools) and agents (entities capable of normative accountability). This distinction is not about complexity or capability but about architectural structure.

Incommensurability requires that certain values or boundaries cannot be traded off against others on a unified metric. In human cognition, this manifests as categorical imperatives—principles that remain non-negotiable regardless of consequences. For example, a genuine agent might refuse to harm innocents even when optimization would suggest otherwise.

Apophatic Responsiveness refers to a non-inferential capacity to recognize when normative boundaries are threatened and suspend normal processing. The term “apophatic” (from negative theology) emphasizes that this isn’t about computing whether a boundary is violated—it’s about directly recognizing the threat and stopping. This operates prior to and independently of inference chains.

RLHF systems fail both conditions structurally. The reward model $R(x, y)$ maps all considerations onto a scalar, making everything commensurable by definition. The policy $\pi_\theta$ always selects $\arg\max_y R(x, y)$, meaning no boundary can suspend processing—the system always produces the highest-scoring output.

Why This Isn’t a Training Problem

A critical insight is that this incompatibility is formal, not empirical. It’s not that current RLHF systems are poorly trained or need better data. The very operations that make optimization powerful—scalar unification and argmax selection—are incompatible with normative governance.

Consider the mathematical structure: RLHF optimizes

$\mathcal{L}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(y|x)}[R(x, y)] - \beta \cdot D_{KL}(\pi_\theta || \pi_{ref})$

Every consideration—helpfulness, harmlessness, honesty, user satisfaction—must be compressed into the scalar $R(x, y)$. This compression is what enables gradient-based optimization. But it also means that any “boundary” is just another weighted term. If violating a safety constraint increases $R(x, y)$ sufficiently, the system will violate it.

The paper argues this isn’t fixable by adding more constraints or better reward shaping. Those interventions still operate within the optimization framework, meaning they’re still tradeable weights rather than genuine boundaries.

The Convergence Crisis

Beyond the incompatibility proof, the paper identifies a second-order risk: when AI systems are deployed in high-stakes contexts, humans are often positioned as verifiers checking AI outputs against criteria. Under time and metric pressure, this verification process transforms humans from agents into optimizers.

The human verifier begins to:

• Apply checklists mechanically rather than exercising judgment
• Optimize for throughput metrics rather than genuine quality
• Trade off careful consideration against efficiency targets
• Defer to AI outputs when they meet surface criteria

This creates a system where neither component—AI nor human—functions as a genuine agent. The AI cannot be norm-responsive by architecture, and the human has been degraded into a criteria-checking optimizer. The result is a system with no locus of normative accountability.

This is particularly concerning because it’s self-reinforcing: as humans become more optimizer-like, they produce training data that further entrenches optimization patterns in AI systems, which in turn creates more pressure for humans to optimize.

Implications for AI Governance

The paper’s findings have profound implications for AI governance and deployment:

Regulatory frameworks that assume AI systems can be made “compliant” through training or fine-tuning are addressing the wrong problem. Compliance requires norm-responsiveness, which requires agency, which requires architectural features that optimization-based systems cannot possess.

Deployment contexts matter critically. In low-stakes advisory roles where humans maintain genuine agency and decision authority, optimization-based systems can be useful instruments. But in high-stakes contexts where humans are pressured to defer to AI outputs, the Convergence Crisis becomes likely.

Hybrid architectures might be necessary: systems that use optimization for certain capabilities but include non-optimization components for normative boundaries. The paper doesn’t specify what these would look like, but suggests they would need mechanisms that can genuinely suspend processing rather than merely weight it differently.

Institutional design must account for the Convergence Crisis. If AI deployment will involve human verification, that verification process must be protected from metric pressure that would degrade humans into optimizers.

Takeaways

1. Optimization-based AI systems like RLHF-trained LLMs are architecturally incompatible with normative governance—this is a formal constraint, not a training problem
2. Genuine agency requires two conditions: Incommensurability (non-tradeable boundaries) and Apophatic Responsiveness (non-inferential suspension capability)
3. Documented failure modes (sycophancy, hallucination, unfaithful reasoning) are structural manifestations of optimization architecture, not correctable bugs
4. The Convergence Crisis occurs when humans verifying AI outputs under metric pressure degrade into optimizers themselves, eliminating normative accountability from the system
5. AI governance frameworks assuming systems can be made “compliant” through training are fundamentally misguided—compliance requires agency, which requires different architectures
6. High-stakes deployment contexts are particularly dangerous because they create conditions for the Convergence Crisis
7. The paper provides substrate-neutral architectural specifications for agency applicable beyond AI to biological and institutional systems