Local Control Networks (LCNs): Optimizing Flexibility in Neural Network Data Pattern Capture

TL;DR

This paper introduces Local Control Networks (LCNs), which use diverse activation functions at each node to improve neural network flexibility, performance, and efficiency over traditional uniform-activation MLPs and more complex architectures.

Contribution

We propose LCNs that leverage B-spline functions for node-specific activations, demonstrating theoretical advantages and empirical improvements over existing models like MLPs and KANs.

Findings

LCNs outperform MLPs and KANs in various benchmarks.

LCNs are more computationally efficient than KANs.

LCNs achieve comparable or better results in vision, machine learning, and symbolic tasks.

Abstract

The widespread use of Multi-layer perceptrons (MLPs) often relies on a fixed activation function (e.g., ReLU, Sigmoid, Tanh) for all nodes within the hidden layers. While effective in many scenarios, this uniformity may limit the networks ability to capture complex data patterns. We argue that employing the same activation function at every node is suboptimal and propose leveraging different activation functions at each node to increase flexibility and adaptability. To achieve this, we introduce Local Control Networks (LCNs), which leverage B-spline functions to enable distinct activation curves at each node. Our mathematical analysis demonstrates the properties and benefits of LCNs over conventional MLPs. In addition, we demonstrate that more complex architectures, such as Kolmogorov-Arnold Networks (KANs), are unnecessary in certain scenarios, and LCNs can be a more efficient alternative. Empirical experiments on various benchmarks and datasets validate our theoretical findings. In computer vision tasks, LCNs achieve marginal improvements over MLPs and outperform KANs by approximately 5\%, while also being more computationally efficient than KANs. In basic machine learning tasks, LCNs show a 1\% improvement over MLPs and a 0.6\% improvement over KANs. For symbolic formula representation tasks, LCNs perform on par with KANs, with both architectures outperforming MLPs. Our findings suggest that diverse activations at the node level can lead to improved performance and efficiency.