Rational ANOVA Networks

TL;DR

Rational ANOVA Networks (RAN) introduce a stable, interpretable neural architecture based on functional ANOVA and rational approximations, improving extrapolation, stability, and data efficiency over traditional models.

Contribution

The paper presents RAN, a novel neural network architecture that combines functional ANOVA decomposition with rational function approximation for enhanced stability and interpretability.

Findings

RAN matches or exceeds baseline models on benchmarks like CIFAR-10.

RAN demonstrates improved stability and throughput compared to polynomial-based models.

RAN effectively captures sharp transitions and near-singular behaviors.

Abstract

Deep neural networks typically treat nonlinearities as fixed primitives (e.g., ReLU), limiting both interpretability and the granularity of control over the induced function class. While recent additive models (like KANs) attempt to address this using splines, they often suffer from computational inefficiency and boundary instability. We propose the Rational-ANOVA Network (RAN), a foundational architecture grounded in functional ANOVA decomposition and Pad\'e-style rational approximation. RAN models f(x) as a composition of main effects and sparse pairwise interactions, where each component is parameterized by a stable, learnable rational unit. Crucially, we enforce a strictly positive denominator, which avoids poles and numerical instability while capturing sharp transitions and near-singular behaviors more efficiently than polynomial bases. This ANOVA structure provides an explicit low-order interaction bias for data efficiency and interpretability, while the rational parameterization significantly improves extrapolation. Across controlled function benchmarks and vision classification tasks (e.g., CIFAR-10) under matched parameter and compute budgets, RAN matches or surpasses parameter-matched MLPs and learnable-activation baselines, with better stability and throughput. Code is available at https://github.com/jushengzhang/Rational-ANOVA-Networks.git.