Neural Additive Experts: Context-Gated Experts for Controllable Model Additivity

TL;DR

Neural Additive Experts (NAEs) introduce a flexible framework that combines the interpretability of additive models with the predictive power of feature interactions, using a mixture of experts and gating mechanisms.

Contribution

NAEs present a novel approach that balances interpretability and accuracy by integrating specialized experts and dynamic gating, with regularization to control feature interaction complexity.

Findings

NAEs outperform traditional GAMs in accuracy while maintaining interpretability.

The model effectively captures feature interactions without sacrificing transparency.

Experimental results on real-world datasets validate the approach's effectiveness.

Abstract

The trade-off between interpretability and accuracy remains a core challenge in machine learning. Standard Generalized Additive Models (GAMs) offer clear feature attributions but are often constrained by their strictly additive nature, which can limit predictive performance. Introducing feature interactions can boost accuracy yet may obscure individual feature contributions. To address these issues, we propose Neural Additive Experts (NAEs), a novel framework that seamlessly balances interpretability and accuracy. NAEs employ a mixture of experts framework, learning multiple specialized networks per feature, while a dynamic gating mechanism integrates information across features, thereby relaxing rigid additive constraints. Furthermore, we propose targeted regularization techniques to mitigate variance among expert predictions, facilitating a smooth transition from an exclusively additive model to one that captures intricate feature interactions while maintaining clarity in feature attributions. Our theoretical analysis and experiments on synthetic data illustrate the model's flexibility, and extensive evaluations on real-world datasets confirm that NAEs achieve an optimal balance between predictive accuracy and transparent, feature-level explanations. The code is available at https://github.com/Teddy-XiongGZ/NAE.