CT-Net: Channel Tensorization Network for Video Classification

TL;DR

This paper introduces CT-Net, a novel channel tensorization network for video classification that balances efficiency and feature interaction by tensorizing channels and integrating a tensor excitation mechanism, achieving state-of-the-art results.

Contribution

Proposes a new Channel Tensorization Network (CT-Net) that factorizes channels into multiple sub-dimensions and incorporates a Tensor Excitation mechanism for improved video classification.

Findings

Outperforms recent SOTA methods on Kinetics-400 and Something-Something benchmarks.

Achieves better accuracy and efficiency compared to existing approaches.

Effectively enlarges the 3D receptive field through channel tensorization.

Abstract

3D convolution is powerful for video classification but often computationally expensive, recent studies mainly focus on decomposing it on spatial-temporal and/or channel dimensions. Unfortunately, most approaches fail to achieve a preferable balance between convolutional efficiency and feature-interaction sufficiency. For this reason, we propose a concise and novel Channel Tensorization Network (CT-Net), by treating the channel dimension of input feature as a multiplication of K sub-dimensions. On one hand, it naturally factorizes convolution in a multiple dimension way, leading to a light computation burden. On the other hand, it can effectively enhance feature interaction from different channels, and progressively enlarge the 3D receptive field of such interaction to boost classification accuracy. Furthermore, we equip our CT-Module with a Tensor Excitation (TE) mechanism. It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module. Finally, we flexibly adapt ResNet as our CT-Net. Extensive experiments are conducted on several challenging video benchmarks, e.g., Kinetics-400, Something-Something V1 and V2. Our CT-Net outperforms a number of recent SOTA approaches, in terms of accuracy and/or efficiency. The codes and models will be available on https://github.com/Andy1621/CT-Net.