Efficient Neural Networks with Discrete Cosine Transform Activations

TL;DR

This paper introduces an efficient neural network architecture using DCT-based activations that enhances interpretability, allows effective pruning, and achieves state-of-the-art accuracy with fewer parameters.

Contribution

It presents a novel DCT-parameterized activation function framework that improves neural network efficiency, interpretability, and pruning capabilities.

Findings

Up to 40% of activation coefficients can be pruned without performance loss.

ENNs achieve state-of-the-art accuracy with fewer parameters.

DCT-based parameterization reveals neuron roles and supports pruning.

Abstract

In this paper, we extend our previous work on the Expressive Neural Network (ENN), a multilayer perceptron with adaptive activation functions parametrized using the Discrete Cosine Transform (DCT). Building upon previous work that demonstrated the strong expressiveness of ENNs with compact architectures, we now emphasize their efficiency, interpretability and pruning capabilities. The DCT-based parameterization provides a structured and decorrelated representation that reveals the functional role of each neuron and allows direct identification of redundant components. Leveraging this property, we propose an efficient pruning strategy that removes unnecessary DCT coefficients with negligible or no loss in performance. Experimental results across classification and implicit neural representation tasks confirm that ENNs achieve state-of-the-art accuracy while maintaining a low number of parameters. Furthermore, up to 40% of the activation coefficients can be safely pruned, thanks to the orthogonality and bounded nature of the DCT basis. Overall, these findings demonstrate that the ENN framework offers a principled integration of signal processing concepts into neural network design, achieving a balanced trade-off between expressiveness, compactness, and interpretability.