ORCA: Making SVM Decisions Transparent Through Orthogonal Polynomial Kernels

Paper: 2604.15285 Authors: Víctor Soto-Larrosa, Nuria Torrado, Edmundo J. Huertas Categories: stat.ML, cs.LG, math.ST

The Gap

SVMs are powerful but opaque. When you train an SVM with a kernel, you get a black box that makes predictions but doesn’t explain itself. Existing interpretability methods either fit surrogate models (LIME, SHAP) that approximate the SVM from outside, or require retraining with specialized architectures. Both approaches have problems: surrogates introduce approximation error and may misrepresent the true model, while retraining sacrifices the original model’s performance.

The specific limitation: for general kernels, the reproducing kernel Hilbert space (RKHS) is infinite-dimensional and lacks an explicit basis. You can’t directly decompose the decision function into interpretable components because you don’t have coordinates to work with.

Problem: SVM with general kernel
   |
   v
Limitation: Infinite-dimensional RKHS, no explicit basis
   |
   v
Assumption: Use truncated orthogonal polynomial kernels
   |                (finite-dimensional RKHS with known basis)
   v
Method: ORCA - expand decision function in RKHS coordinates
   |
   v
Evidence: OKC indices reveal interaction structure
   |        (synthetic spiral + real echocardiogram data)
   v
Conclusion: Post-training interpretability without surrogates

The Increment

One sentence: Before this paper, interpreting SVMs meant approximating them with external models; after, you can directly read the internal structure of the decision function if you use the right kernel.

Core Mechanism

ORCA works by exploiting a mathematical property of truncated orthogonal polynomial kernels: they generate finite-dimensional RKHS with explicit tensor-product bases. When you train an SVM with such a kernel, the decision function lives in this finite space and can be written as a finite sum of basis functions with computable coefficients.

The method has three components: (1) the kernel itself, constructed from univariate orthogonal polynomials up to degree d, (2) the RKHS expansion, which expresses the fitted decision function as a weighted sum of basis elements, and (3) the OKC indices, which aggregate the squared RKHS norm across different structural dimensions—interaction order, total degree, marginal effects, pairwise interactions.

Data flows like this: trained SVM → extract support vectors and dual coefficients → compute RKHS coordinates by projecting onto basis functions → calculate squared norms for each structural component → normalize to get OKC indices. No retraining, no external approximation—just direct decomposition of what’s already there.

Input: Trained SVM with truncated polynomial kernel
   |
   v
[Support Vectors] + [Dual Coefficients]
   |
   v
Project onto RKHS basis: f = sum_i c_i * phi_i
   |
   v
Compute ‖f_component‖^2 for each structural slice
   |
   v
Normalize: OKC_component = ‖f_component‖^2 / ‖f‖^2
   |
   v
Output: Interpretability indices (interaction order, degree, marginal, pairwise)

Think of the decision function as a musical chord. With a general kernel, you hear the chord but can’t identify the individual notes—the sound exists in an infinite-dimensional space of possible frequencies. ORCA’s truncated polynomial kernel is like restricting to a finite piano keyboard. Now the chord is a specific combination of known keys. The RKHS basis functions are the individual keys, and the coefficients tell you how hard each key is pressed. The OKC indices group keys by musical structure: how much power comes from single notes (marginal effects), two-note intervals (pairwise interactions), three-note triads (third-order interactions), and so on. You’re not approximating the chord with a different instrument—you’re reading the sheet music that was there all along.

Key Concepts

• Reproducing Kernel Hilbert Space (RKHS): Imagine a function space where every function has a “length” (norm) and an “angle” with other functions (inner product). The kernel k(x,y) is a recipe for computing inner products without explicitly writing down the functions. For most kernels (like RBF), this space is infinite-dimensional—you’d need infinitely many coordinates to describe a function. But for truncated polynomial kernels, the space is finite-dimensional, like R^n. You can list all the basis functions explicitly: {1, x, x^2, …, x^d} in 1D, and tensor products in higher dimensions. This makes the abstract space concrete and computable.

• Orthogonal Kernel Contribution (OKC) indices: These are normalized energy budgets. The decision function has a total squared RKHS norm ‖f‖^2, which measures its complexity. OKC indices answer: “What fraction of this complexity comes from interactions of order k?” or “What fraction comes from coordinate j alone?” For example, OKC_order=2 might be 0.3, meaning 30% of the model’s complexity is in pairwise interactions. Unlike feature importance scores that measure prediction impact, OKC indices measure structural composition—they tell you how the model is built, not just what it does.

• Post-training interpretability: Most interpretability methods either require special training procedures (attention mechanisms, sparse models) or fit external approximations after training (LIME, SHAP). Post-training interpretability means you take an already-trained model and extract explanations directly from its internal representation, without retraining or approximation. ORCA achieves this because the RKHS basis is known at training time—the model naturally lives in interpretable coordinates, you just need to read them out.

Framework Shift

Before (mainstream approach):        After (this paper):

Train SVM with RBF/polynomial        Train SVM with truncated
kernel (infinite-dim RKHS)           orthogonal polynomial kernel
         |                                    |
         v                                    v
Black box decision function          Decision function in finite
         |                           RKHS with explicit basis
         v                                    |
Fit surrogate model                          v
(LIME/SHAP) to approximate           Direct RKHS expansion:
         |                           f = sum c_i * phi_i
         v                                    |
Approximate feature                          v
importance scores                    Exact OKC indices from
                                     ‖f‖^2 decomposition

From external approximation to internal decomposition, the core shift is treating interpretability as a coordinate system problem rather than a model approximation problem.

Expert Assessment

Problem choice: Real gap. SVM interpretability has been stuck between two bad options: lose fidelity (surrogates) or lose performance (constrained training). The insight that kernel choice determines RKHS structure is mathematically obvious but practically underexploited. This sits at the intersection of classical kernel methods and modern interpretability demands—timely given renewed interest in non-neural approaches.

Method maturity: Elegant, not clever. The authors aren’t inventing new math—they’re applying known properties of orthogonal polynomials to a practical problem. The limitation is obvious: you must use their specific kernel family, which may underperform RBF or other kernels on some tasks. The paper doesn’t adequately address this trade-off. A practitioner needs to know: “How much accuracy do I sacrifice for interpretability?” The experiments show the method works but don’t quantify the performance cost.

Experimental integrity: Baselines are fair but minimal. The synthetic spiral problem is illustrative but too clean. The echocardiogram dataset (5D, 132 samples) is real but small. Missing: comparison with standard RBF kernel on the same data, showing accuracy vs interpretability trade-off. Also missing: larger-scale experiments and comparison with SHAP/LIME to show where exact decomposition differs from approximation. The OKC indices reveal structure, but the paper doesn’t validate whether this structure matches domain knowledge or improves decision-making.

Writing quality: The mathematical exposition is rigorous but front-loaded. Section 2 dumps RKHS theory before motivating why it matters. The paper would be stronger if it started with the interpretability problem, showed why existing methods fail, then introduced the math as a solution. The experimental section (Section 4) is where the value becomes clear, but it’s buried after dense theory. Rewriting Section 1 to include a motivating example and moving some theory to an appendix would make the contribution more accessible.

Verdict: Weak accept — solid theoretical contribution with practical value, but experiments are too limited to assess real-world viability and the accuracy trade-off is underexplored.

Takeaways

Practitioners can steal the core idea even without using this exact method: interpretability is often a coordinate system problem. If you can represent your model in a basis where components have semantic meaning, you get interpretability for free. This applies beyond SVMs—neural networks with structured architectures (tensor decompositions, polynomial layers) could use similar decomposition strategies.

Specific technique: the OKC index calculation pattern (decompose squared norm across structural dimensions, normalize) is a general recipe for interpretability metrics. If you have any model with an explicit expansion (linear models, GAMs, polynomial networks), you can compute analogous indices for interaction order, feature groups, or other structural slices.

The trade-off framework is worth adopting: don’t treat interpretability as a post-hoc add-on. Choose model architectures where interpretability is intrinsic, then quantify the performance cost. This paper shows one instance (truncated polynomial kernels for SVMs) but the principle generalizes.