Triangular Dropout: Variable Network Width without Retraining

TL;DR

This paper introduces Triangular Dropout, a novel layer design allowing neural networks to be compressed post-training without retraining, enabling flexible model size and performance trade-offs across various applications.

Contribution

The paper proposes Triangular Dropout, a new layer that permits post-training width reduction, facilitating adaptable model compression without retraining.

Findings

Effective in autoencoders for post-training compression.

Reduces parameters in VGG19 on ImageNet without retraining.

Applicable to reinforcement learning policies for flexible model sizing.

Abstract

One of the most fundamental design choices in neural networks is layer width: it affects the capacity of what a network can learn and determines the complexity of the solution. This latter property is often exploited when introducing information bottlenecks, forcing a network to learn compressed representations. However, such an architecture decision is typically immutable once training begins; switching to a more compressed architecture requires retraining. In this paper we present a new layer design, called Triangular Dropout, which does not have this limitation. After training, the layer can be arbitrarily reduced in width to exchange performance for narrowness. We demonstrate the construction and potential use cases of such a mechanism in three areas. Firstly, we describe the formulation of Triangular Dropout in autoencoders, creating models with selectable compression after training. Secondly, we add Triangular Dropout to VGG19 on ImageNet, creating a powerful network which, without retraining, can be significantly reduced in parameters. Lastly, we explore the application of Triangular Dropout to reinforcement learning (RL) policies on selected control problems.