# A simple and efficient architecture for trainable activation functions

## TL;DR

This paper introduces a straightforward and efficient method for learning activation functions in neural networks by adding small local subnetworks, achieving improved results without significantly increasing complexity.

## Contribution

It proposes a simple, parameter-efficient approach to trainable activation functions using local subnetworks, simplifying implementation and theoretical understanding.

## Key findings

- Improved performance over fixed activation functions
- Minimal additional parameters required
- Effective across different neural network architectures

## Abstract

Learning automatically the best activation function for the task is an active topic in neural network research. At the moment, despite promising results, it is still difficult to determine a method for learning an activation function that is at the same time theoretically simple and easy to implement. Moreover, most of the methods proposed so far introduce new parameters or adopt different learning techniques. In this work we propose a simple method to obtain trained activation function which adds to the neural network local subnetworks with a small amount of neurons. Experiments show that this approach could lead to better result with respect to using a pre-defined activation function, without introducing a large amount of extra parameters that need to be learned.