Training CNNs faster with Dynamic Input and Kernel Downsampling

TL;DR

This paper introduces a method to accelerate CNN training by intermittently downsampling inputs and filters, reducing computation and memory usage while maintaining accuracy.

Contribution

The authors propose a novel training approach combining input and kernel downsampling with interleaved passes, significantly reducing training time with minimal accuracy loss.

Findings

Achieved up to 23% reduction in training time.

Minimal loss in validation accuracy.

Effective for residual architectures.

Abstract

We reduce training time in convolutional networks (CNNs) with a method that, for some of the mini-batches: a) scales down the resolution of input images via downsampling, and b) reduces the forward pass operations via pooling on the convolution filters. Training is performed in an interleaved fashion; some batches undergo the regular forward and backpropagation passes with original network parameters, whereas others undergo a forward pass with pooled filters and downsampled inputs. Since pooling is differentiable, the gradients of the pooled filters propagate to the original network parameters for a standard parameter update. The latter phase requires fewer floating point operations and less storage due to the reduced spatial dimensions in feature maps and filters. The key idea is that this phase leads to smaller and approximate updates and thus slower learning, but at significantly reduced cost, followed by passes that use the original network parameters as a refinement stage. Deciding how often and for which batches the downsmapling occurs can be done either stochastically or deterministically, and can be defined as a training hyperparameter itself. Experiments on residual architectures show that we can achieve up to 23% reduction in training time with minimal loss in validation accuracy.