Reconstructing Pruned Filters using Cheap Spatial Transformations

TL;DR

This paper introduces a parameter-efficient convolutional layer replacement using spatial transformations, leveraging filter redundancy to maintain high accuracy while reducing model complexity.

Contribution

It proposes a novel method modeling pruning as replacing filters with cheap spatial transformations, enhancing efficiency without sacrificing performance.

Findings

Achieves comparable or better accuracy than state-of-the-art pruning methods.

Effective on CIFAR-10 and ImageNet-1K datasets.

Provides efficient implementation and extensions for improved expressivity.

Abstract

We present an efficient alternative to the convolutional layer using cheap spatial transformations. This construction exploits an inherent spatial redundancy of the learned convolutional filters to enable a much greater parameter efficiency, while maintaining the top-end accuracy of their dense counter-parts. Training these networks is modelled as a generalised pruning problem, whereby the pruned filters are replaced with cheap transformations from the set of non-pruned filters. We provide an efficient implementation of the proposed layer, followed by two natural extensions to avoid excessive feature compression and to improve the expressivity of the transformed features. We show that these networks can achieve comparable or improved performance to state-of-the-art pruning models across both the CIFAR-10 and ImageNet-1K datasets.