Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective

TL;DR

This paper analyzes the failure modes of Rectified Power Unit networks using effective field theory and introduces MRePU, a new activation function that improves training stability and performance in deep neural networks.

Contribution

It provides a theoretical understanding of RePU failures and proposes MRePU, a novel activation function that enhances stability and universality in deep networks.

Findings

MRePU improves training stability across tasks

Theoretical analysis confirms MRePU's criticality conditions

Experiments show enhanced performance in vision and physics tasks

Abstract

The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies criticality conditions necessary for stable training, placing it in a distinct universality class. Extensive experiments validate the effectiveness of MRePU, showing significant improvements in training stability and performance across various tasks, including polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks. Our findings highlight the potential of MRePU as a robust alternative for building deep neural networks.