NeuronSeek: On Stability and Expressivity of Task-driven Neurons

TL;DR

NeuronSeek employs tensor decomposition to optimize task-driven neurons, enhancing stability and expressivity, with theoretical guarantees and competitive empirical performance across benchmarks.

Contribution

This work introduces tensor decomposition into NeuronSeek, replacing symbolic regression, to improve stability and convergence in discovering task-driven neurons, supported by theoretical guarantees.

Findings

Enhanced stability over previous methods

Competitive performance on diverse benchmarks

Theoretical proof of universal approximation capabilities

Abstract

Drawing inspiration from our human brain that designs different neurons for different tasks, recent advances in deep learning have explored modifying a network's neurons to develop so-called task-driven neurons. Prototyping task-driven neurons (referred to as NeuronSeek) employs symbolic regression (SR) to discover the optimal neuron formulation and construct a network from these optimized neurons. Along this direction, this work replaces symbolic regression with tensor decomposition (TD) to discover optimal neuronal formulations, offering enhanced stability and faster convergence. Furthermore, we establish theoretical guarantees that modifying the aggregation functions with common activation functions can empower a network with a fixed number of parameters to approximate any continuous function with an arbitrarily small error, providing a rigorous mathematical foundation for the NeuronSeek framework. Extensive empirical evaluations demonstrate that our NeuronSeek-TD framework not only achieves superior stability, but also is competitive relative to the state-of-the-art models across diverse benchmarks. The code is available at https://github.com/HanyuPei22/NeuronSeek.