Softmax Attention Was Always a Born-Rule Object (If You Live on the Simplex)

Paper: 2608.11173 Authors: Eric A. F. Reinhardt, Adam J. Hauser Categories: quant-ph, cs.LG

The Gap

Every attempt to put a Transformer on a quantum computer runs into the same wall: softmax needs two things a unitary circuit does not natively give you. It needs an elementwise exp, which is not linear, and it needs a global normalizer, a sum over all keys that couples every score to every other score.

The prior art works around this rather than through it. Variational quantum self-attention networks compute scores as expectation values and then feed them to a classical softmax on the CPU. Quantum vision transformer designs based on data loaders and orthogonal/compound-matrix layers keep the linear algebra quantum but hand normalization back to the host. Quantum-kernel attention swaps exp(q·k) for a Gaussian or linear-attention surrogate and argues it is close enough. The polynomial-approximation route (quantum singular value transformation on a block-encoding) can approximate exp to precision epsilon, but you pay in circuit depth and you never get the real thing.

So the honest state of the field was: nobody had exhibited a regime where softmax attention is exactly what a quantum measurement produces, rather than something a quantum device approximates. This paper picks a restriction that makes that possible: assume the data is simplex-bound. Inputs are probability vectors, outputs are probability vectors, everything sums to one. That is the natural habitat of amplitude encoding, and once you are there the authors claim the entire attention layer, scores, temperature, value mixing, and residual gate, becomes a dictionary of rotation angles and Born-rule statistics with no approximation step in the algebra.

[ Problem ] softmax needs exp() and a global normalizer;
            a unitary circuit provides neither natively
     |
     v
[ Prior art ] QSANN / quantum ViT / kernel attention / QSVT
     |    . swap softmax for a Gaussian or linear surrogate
     |    . or approximate exp() to eps, paying in depth
     |    . or do the normalization on a classical CPU
     v
[ Assumption ] restrict to the subclass where inputs AND outputs
               live on the probability simplex (sum == 1)
     |
     v
[ Method ] amplitude-encode x as sqrt(x)
           scores == Hadamard-test statistics on a
                     block-encoded projection
           weights == normalized cos^2 family, in exact
                      bijection with the exp family
           temperature == number of post-selected repetitions
           values == column-loading channel (dilated V)
           residual == one ancilla preparation angle
     |
     v
[ Evidence ] algebraic core machine-checked in Lean 4;
             layer exact in the infinite-shot limit
             (one measure-and-reload per score)
     |
     v
[ Conclusion ] on the simplex, softmax attention IS a Born-rule
               object, not an approximation target; and its
               boundary is a sparse-attention family with
               exact zeros at finite parameter values

The Increment

One sentence: Before, quantum attention meant “find a circuit-friendly thing that behaves like softmax”; after, on simplex-bound data, softmax is revealed as the interior chart of a cos-squared measurement family, with the extra territory outside that chart being exact sparse attention.

Core Mechanism

Start with amplitude encoding. A probability vector x with entries summing to one becomes the state whose amplitudes are sqrt(x_i). This is not a lossy embedding trick, it is an isomorphism between the simplex and the positive orthant of the unit sphere. Because outputs are also required to be simplex-bound, the layer never has to leave this manifold, which is what kills the need for an external normalizer: the Born rule normalizes for you, by construction.

Scores come next. Instead of a raw inner product, the query/key interaction is a block-encoded projection U(theta) applied inside a Hadamard test. The Hadamard test’s ancilla measurement statistic gives you the real part of an overlap, which after the standard rescaling is a cos^2 of some effective angle phi_ij. Here is the load-bearing algebraic claim: any point in the open simplex of attention weights can be written as normalized cos^2(phi) values, and this correspondence with the exponential softmax family is a bijection. Softmax is not approximated; it is re-coordinatized. And the moment you allow phi = pi/2, you get a weight that is exactly zero at a finite parameter, something softmax can only reach in the limit of minus-infinity logits. Exact sparsity falls out of the boundary of the chart.

Temperature is the cutest piece. If a single post-selected round yields probability p = cos^2(phi), then insisting on k successful rounds yields p^k. Raising every weight to the k-th power and renormalizing is exactly an inverse-temperature knob, so the softmax temperature becomes a repetition count, discretized to integers. Value aggregation then exploits the same simplex constraint: if the value matrix V is column-stochastic, each column is itself a probability vector, so V can be dilated into a deterministic “column-loading” channel rather than being approximated by some non-unitary operator. Finally the gated residual, normally alpha * attn + (1 - alpha) * x, becomes a single ancilla preparation angle whose cos^2 and sin^2 are automatically a convex pair; at mixing angle pi/2 you recover plain addition. Every learnable parameter in the layer is a rotation angle. The seam is that composing all this exactly requires one measure-and-reload step per attention score, so the exact version is hybrid; a fully coherent version exists but is only epsilon-approximate via QSVT, in the infinite-depth limit.

classical simplex input x  (sum_i x_i == 1)
        |
        v
+--------------------------+
| amplitude encoding       |  x_i -> sqrt(x_i) |i>
+--------------------------+
        |
        v
+---------------------------+      +--------------------------+
| block-encoded projection  | <--- | learnable params:        |
| U(theta) : Q / K role     |      | rotation angles ONLY     |
+---------------------------+      +--------------------------+
        |
        v
+--------------------------+
| Hadamard test on ancilla |  stat -> Re <x| U(theta) |x'>
+--------------------------+
        |
        v   effective score angle phi_ij
+-------------------------------------------+
| Born measurement, k post-selected rounds  |
|    p == cos^2(phi)   ==>   p^k            |  k == inverse temperature
+-------------------------------------------+
        |
        v   weights a_ij
        |     interior of cos^2 family  ==  softmax image (bijection)
        |     boundary (phi == pi/2)    ==  EXACT zeros, finite params
        v
+-----------------------------+
| column-loading channel V    |  column-stochastic V dilated
| deterministic, no approx    |  into a CPTP map on the simplex
+-----------------------------+
        |
        v
+-----------------------------+
| ancilla gate angle g        |  cos^2(g) * attn + sin^2(g) * x
| g == pi/2 -> additive id.   |  == gated residual, convex for free
+-----------------------------+
        |
        v
   output, already on the simplex

 seam:  [ measure-and-reload, once per score ]  -> exact, hybrid
 alt:   [ fully coherent QSVT variant ]         -> eps-approx, deep

Now the metaphor, and this one is not decorative, it is the same physics: the layer is a rack of polarizing filters in a darkroom.

Malus’s law says light through two polarizers at relative angle phi comes out with intensity cos^2(phi). That is the Born rule for polarization. So: the amplitude-encoded input is the beam of light, already carrying a fixed total intensity of one. Each key is a filter, and the learnable parameter is the angle you twist that filter to. The attention weight for a key is simply how much light gets through its filter, and because you measure fractions of one beam, the weights normalize themselves; nobody has to run down the hall with a calculator to divide by a sum. Temperature is stacking k identical filter pairs in series, cos^(2k), which sharpens the contrast between the bright channels and the dim ones exactly the way lowering softmax temperature does. Crossing a filter to ninety degrees gives you true black at a finite twist, which is the exact-zero sparsity that softmax, forever needing an infinite logit, cannot reach. Each filter is also tinted: the light that passes gets dyed with a fixed recipe of pigments that always uses up the full palette, which is the column-stochastic value matrix as a loading channel. The residual connection is a half-silvered mirror whose reflectivity is set by one more angle, blending the filtered beam with the original beam, and at forty-five degrees you get the plain even blend. The catch, the measure-and-reload seam, is that for each score you have to develop the plate and re-shoot rather than keeping one continuous exposure. That is precisely why the exact scheme is hybrid and claims no speedup.

Key Concepts

• Amplitude encoding as an isomorphism, not a compression: usually people describe amplitude encoding as “cram 2^n numbers into n qubits”, which immediately raises the read-out problem. Here the framing is different. A probability distribution over d outcomes and a real non-negative unit vector in d dimensions are the same object under the map p_i <-> sqrt(p_i). If your problem *already speaks in distributions, say a mixture-of-experts gate, a topic proportion, a chemical composition, or a distributional-RL value histogram, then there is no encoding loss to argue about. That is the entire reason this paper can claim exactness where others claim approximation: the restriction to simplex-bound data removes the mismatch between the data’s geometry and the state space’s geometry.

• The cos-squared family, and why its boundary is a gift: softmax weights are exp(z_i) over their sum. Cos-squared weights are cos^2(phi_i) over their sum. Both are recipes for turning free parameters into a point on the simplex. The claim is that they cover the same *interior, so any softmax pattern you want is reachable by some choice of angles, and vice versa. The difference is at the edges. To make a softmax weight exactly zero you need a logit of minus infinity, which is why practitioners invented sparsemax and entmax to get honest zeros by Euclidean or Tsallis projection. The cos-squared family gets them for free: twist to pi/2 and the weight is zero, full stop, at a finite parameter that gradient descent can actually reach. Concretely, with three keys at angles (0, pi/3, pi/2) you get raw values (1, 0.25, 0) and weights (0.8, 0.2, 0), with the third key genuinely dropped, not merely down-weighted.

• Temperature as post-selection depth: post-selection means you only keep the experimental runs where an ancilla measurement came out the way you wanted. If a channel passes with probability p, then demanding k independent passes in a row keeps it with probability p^k. Since p^k = exp(k ** ln p), repeating the measurement is literally multiplying the log-weights by k, which is what inverse temperature does. Elegant, and also the paper’s main cost center: the probability that all your rounds succeed shrinks geometrically, so a sharper attention distribution directly buys you an exponentially larger shot budget. Also, k is an integer, so you get temperatures 1, 1/2, 1/3, and nothing in between unless you interpolate some other way.

Framework Shift

Before (mainstream quantum attention):     After (this paper):

   quantum circuit                            simplex is the native
        |                                     object; Born rule IS
        |  linear or Gaussian                 the normalizer
        |  surrogate score
        v                                     x  --(sqrt)-->  amplitudes
   classical CPU                                        |
        |  exp()                                        v
        |  sum over keys                        cos^2 family:
        |  divide                                interior == softmax
        v                                        boundary == exact 0
   weights                                               |
        |                                                v
        v                                       column-loading V
   quantum circuit                                       |
        |  value matmul                                  v
        v                                       ancilla angle gate
   classical CPU                                         |
        |  renormalize output                            v
        v                                       output sums to 1
   output                                        by construction

 "softmax is a nonlinearity                 "softmax is one chart on a
  we must approximate"                       Born-rule manifold, and
                                             the manifold is bigger"

From “approximate the nonlinearity” to “re-coordinatize the nonlinearity”, the core shift is that the exponential was never the primitive; the primitive is a normalized measurement probability, and exp is just one coordinate system on it.

Expert Assessment

(Caveat: I am reading this at abstract-and-claims level, so treat the resource commentary below as the questions I would ask in review rather than as findings from the full text.)

Problem choice: The gap is real, not manufactured. “Softmax is quantum-unfriendly” has been the standing excuse in quantum-machine-learning papers for years, and nobody had asked whether there is a data regime where the excuse dissolves. Restricting to simplex-bound attention is a legitimate move rather than a dodge, because such problems genuinely exist: routing/gating distributions, distributional RL, topic and composition data, Markov-chain aggregation, models that mix categorical distributions. The weakness is that the paper appears to *assert this subclass rather than sell it. If simplex-bound attention is the whole premise, the reader deserves two pages naming real workloads and showing the constraint is not a straitjacket. Positionally, this sits where quantum ML papers should sit right now: after the era of “we ran a 4-qubit variational circuit and it learned XOR”, and honestly labeled a roadmap.

Method maturity: The cos-squared/exp bijection is genuine cleverness and the best thing in the paper, partly because it is not even quantum-dependent. The pi/2 residual angle giving convexity for free and the column-stochastic-V-as-channel dilation are both the right kind of observation: the constraint you were fighting turns out to be the structure that makes the construction work. Temperature-as-repetition-count is charming but is the weak joint. Integer-only inverse temperature is a real modeling limitation, and post-selection acceptance decaying like the product of p_i^k means the sharper the attention, the worse the sampling. Meanwhile, “exact in the infinite-shot limit with one measure-and-reload per attention score” deserves a hard stare: n^2 scores, each estimated to precision epsilon at 1/epsilon^2 shots, with a state reload in between, is a lot of quantum machinery to reproduce something a CPU does in nanoseconds. The paper is careful not to claim a speedup, which is to its credit, but a reader skimming the abstract will not notice that.

Experimental integrity: There are effectively no experiments, and the paper does not pretend otherwise. The Lean 4 verification is real intellectual hygiene and I wish more theory papers did it, but be clear about what it buys: it certifies the algebraic identities, which are the part least likely to be wrong. It says nothing about shot cost, noise sensitivity, trainability of angle parameterizations (barren-plateau behavior for these block-encoded projections is an obvious open question), or whether a cos-squared-attention model actually trains as well as a softmax one. The two experiments I would most want, and both are cheap, are: (a) a classical simulator comparing cos^2 attention against softmax, sparsemax, and entmax on a genuinely simplex-bound task, and (b) an explicit shot-budget table versus classical FLOPs. Absent those, “exact” is a statement about a limit, not about anything you can run.

Writing quality: The abstract is a wall of claims stacked at maximum density, each individually interesting and collectively exhausting; it reads as if written to survive a skeptical skim rather than to be understood. The corner cut is cost accounting: the phrases “infinite-shot limit” and “infinite depth limit” are doing heavy lifting in a single sentence at the end. The section that would elevate the whole paper is a plain resource analysis, stating flatly what this costs, what it does not accelerate, and why the construction is still worth having (the answer being conceptual clarity plus the sparse boundary, which is a fine answer).

Verdict: weak accept — the exp-to-cos-squared bijection with an exactly-sparse boundary is a clean, reusable result that reframes a standing obstacle, but the paper is a dictionary rather than an algorithm, and it hides its costs behind limit language while shipping zero empirical support.

Takeaways

• Steal the cos-squared attention head, no quantum computer required. Weights proportional to cos^(2k)(phi_i) give you exact zeros at finite parameters, which is the whole selling point of sparsemax and entmax, but without a projection step or a Tsallis-entropy solver. It is one elementwise trig call plus a normalize. If you need honest sparsity in an attention or gating layer, this is worth ten minutes on a toy benchmark.
• Parameterize convex mixtures by angles rather than by a sigmoid. Writing a gate as cos^2(g) and sin^2(g) makes the pair sum to one identically, with no clamping, no 1 - alpha bookkeeping, and no way to drift off the simplex through numerical error. For any model whose outputs must remain distributions, this is a free invariant.
• Sharpening by powering and renormalizing is the same as lowering temperature. Obvious once stated, and useful when you have a positive score you cannot take logs of safely: raise to a power instead of dividing a logit.
• Column-stochastic weight matrices are probability-preserving maps, not just matrices. If your value or mixing matrix has columns that sum to one, it is a Markov kernel, and simplex-in/simplex-out is structural rather than something you have to enforce with a post-hoc softmax. That framing transfers directly to mixture models and to any layer stacking over distributions.
• The design pattern worth remembering: when a nonlinearity looks hostile to your substrate, look for a different chart on the same manifold instead of a polynomial approximation. The paper’s real contribution is methodological. exp was never the thing; a normalized non-negative weight was the thing.
• Machine-check your algebraic core. Whatever you think of the physics, a Lean 4 proof of the identities is cheap credibility and stops the tedious class of reviewer objection cold. Just do not let it stand in for empirical evidence, which is exactly the trap this paper falls into.