Video Classification with Channel-Separated Convolutional Networks

TL;DR

This paper investigates the use of channel-separated 3D convolutions for video classification, demonstrating improved accuracy and efficiency by separating channel and spatiotemporal interactions, leading to a novel architecture called CSN.

Contribution

It introduces the Channel-Separated Convolutional Network (CSN), a simple and efficient architecture that leverages channel separation in 3D convolutions for better video classification performance.

Findings

Channel separation improves accuracy and reduces computation.

Factorizing 3D convolutions enhances regularization and test accuracy.

CSN achieves comparable or better results than state-of-the-art methods, with 2-3x efficiency.

Abstract

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.