Increasing biases can be more efficient than increasing weights

TL;DR

This paper proposes a new neural network unit with multiple biases, demonstrating that increasing biases can be more effective than increasing weights for improving model performance, supported by empirical and theoretical analysis.

Contribution

Introduces a novel neural network unit with multiple biases, emphasizing bias increases over weight increases for better information flow and performance.

Findings

Bias-focused units can outperform weight-focused ones.

Empirical results show improved accuracy with bias enhancements.

Theoretical analysis supports bias-based optimization benefits.

Abstract

We introduce a novel computational unit for neural networks that features multiple biases, challenging the traditional perceptron structure. This unit emphasizes the importance of preserving uncorrupted information as it is passed from one unit to the next, applying activation functions later in the process with specialized biases for each unit. Through both empirical and theoretical analyses, we show that by focusing on increasing biases rather than weights, there is potential for significant enhancement in a neural network model's performance. This approach offers an alternative perspective on optimizing information flow within neural networks. See source code at https://github.com/CuriosAI/dac-dev.