Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization

TL;DR

This paper introduces a biologically inspired method for self-reorganizing and rejuvenating CNNs, which enhances model capacity utilization and performance without altering network architecture.

Contribution

The proposed method reorganizes and rejuvenates CNN parameters based on channel activations, increasing capacity utilization and improving performance in a model-agnostic manner.

Findings

Increases network performance without structural changes.

Applicable during training or on pre-trained models.

Effective across various backbone architectures.

Abstract

In this paper, we propose self-reorganizing and rejuvenating convolutional neural networks; a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The reorganized parameters are clustered to avoid parameter redundancies. As such, redundant neurons with similar activations are merged leaving room for the remaining parameters to rejuvenate. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters. As a result, the network capacity utilization increases improving the baseline network performance without any changes to the network structure. The proposed method can be applied to various network architectures during the training stage, or applied to a pre-trained model improving its performance. Experimental results showed that the proposed method is model-agnostic and can be applied to any backbone architecture increasing its performance due to the elevated utilization of the network capacity.