Channel Equilibrium Networks for Learning Deep Representation

TL;DR

This paper introduces Channel Equilibrium (CE) blocks that address channel inhibition in CNNs, leading to improved generalization and state-of-the-art results on benchmarks like ImageNet and COCO.

Contribution

The paper proposes a novel CE block that 'wakes up' inhibited channels, enhancing CNN performance and generalization, and connects it with Nash Equilibrium theory.

Findings

CE blocks improve CNN accuracy on ImageNet and COCO

CE can be integrated into architectures like ResNet and MobileNet

CE achieves state-of-the-art performance on multiple benchmarks

Abstract

Convolutional Neural Networks (CNNs) are typically constructed by stacking multiple building blocks, each of which contains a normalization layer such as batch normalization (BN) and a rectified linear function such as ReLU. However, this work shows that the combination of normalization and rectified linear function leads to inhibited channels, which have small magnitude and contribute little to the learned feature representation, impeding the generalization ability of CNNs. Unlike prior arts that simply removed the inhibited channels, we propose to "wake them up" during training by designing a novel neural building block, termed Channel Equilibrium (CE) block, which enables channels at the same layer to contribute equally to the learned representation. We show that CE is able to prevent inhibited channels both empirically and theoretically. CE has several appealing benefits. (1) It can be integrated into many advanced CNN architectures such as ResNet and MobileNet, outperforming their original networks. (2) CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. (3) Extensive experiments show that CE achieves state-of-the-art performance on various challenging benchmarks such as ImageNet and COCO.