Rank-1 Convolutional Neural Network

TL;DR

This paper introduces a CNN with 3-D rank-1 filters constructed from outer products of 1-D filters, enabling efficient training and inference by maintaining low-rank constraints and improving gradient flow.

Contribution

The paper proposes a novel CNN architecture using 3-D rank-1 filters trained via gradient flow and outer product constraints, enhancing training stability and inference speed.

Findings

Training with 3-D rank-1 filters improves gradient flow.

Decomposed filters enable fast inference.

Outputs are constrained to low-rank subspace.

Abstract

In this paper, we propose a convolutional neural network(CNN) with 3-D rank-1 filters which are composed by the outer product of 1-D filters. After being trained, the 3-D rank-1 filters can be decomposed into 1-D filters in the test time for fast inference. The reason that we train 3-D rank-1 filters in the training stage instead of consecutive 1-D filters is that a better gradient flow can be obtained with this setting, which makes the training possible even in the case where the network with consecutive 1-D filters cannot be trained. The 3-D rank-1 filters are updated by both the gradient flow and the outer product of the 1-D filters in every epoch, where the gradient flow tries to obtain a solution which minimizes the loss function, while the outer product operation tries to make the parameters of the filter to live on a rank-1 sub-space. Furthermore, we show that the convolution with the rank-1 filters results in low rank outputs, constraining the final output of the CNN also to live on a low dimensional subspace.