Maximum and Leaky Maximum Propagation

TL;DR

This paper introduces maximum and leaky maximum propagation methods as alternatives to residual connections, demonstrating comparable performance with benefits like faster learning and better generalization, especially when combined with residual networks.

Contribution

It proposes maximum and leaky maximum propagation techniques, offering a novel alternative to residual connections with improved properties and compatibility with ensemble methods.

Findings

Comparable performance to residual connections

Better generalization with constant batch normalization

Faster learning and effective ensemble integration

Abstract

In this work, we present an alternative to conventional residual connections, which is inspired by maxout nets. This means that instead of the addition in residual connections, our approach only propagates the maximum value or, in the leaky formulation, propagates a percentage of both. In our evaluation, we show on different public data sets that the presented approaches are comparable to the residual connections and have other interesting properties, such as better generalization with a constant batch normalization, faster learning, and also the possibility to generalize without additional activation functions. In addition, the proposed approaches work very well if ensembles together with residual networks are formed. https://atreus.informatik.uni-tuebingen.de/seafile/d/8e2ab8c3fdd444e1a135/?p=%2FMaximumPropagation&mode=list