Searching for Two-Stream Models in Multivariate Space for Video Recognition

TL;DR

This paper introduces an efficient neural architecture search method to automatically discover high-performing two-stream video recognition models, significantly reducing manual design effort and computational costs.

Contribution

The paper proposes a multivariate search space and a progressive search procedure for automatic design of two-stream video models, outperforming manually designed architectures.

Findings

Auto-TSNet models outperform existing models on benchmarks.

Auto-TSNet-L reduces FLOPS by 11 times while maintaining accuracy.

Auto-TSNet-M improves accuracy on Something-Something-V2 with less than 50 GFLOPS.

Abstract

Conventional video models rely on a single stream to capture the complex spatial-temporal features. Recent work on two-stream video models, such as SlowFast network and AssembleNet, prescribe separate streams to learn complementary features, and achieve stronger performance. However, manually designing both streams as well as the in-between fusion blocks is a daunting task, requiring to explore a tremendously large design space. Such manual exploration is time-consuming and often ends up with sub-optimal architectures when computational resources are limited and the exploration is insufficient. In this work, we present a pragmatic neural architecture search approach, which is able to search for two-stream video models in giant spaces efficiently. We design a multivariate search space, including 6 search variables to capture a wide variety of choices in designing two-stream models. Furthermore, we propose a progressive search procedure, by searching for the architecture of individual streams, fusion blocks, and attention blocks one after the other. We demonstrate two-stream models with significantly better performance can be automatically discovered in our design space. Our searched two-stream models, namely Auto-TSNet, consistently outperform other models on standard benchmarks. On Kinetics, compared with the SlowFast model, our Auto-TSNet-L model reduces FLOPS by nearly 11 times while achieving the same accuracy 78.9%. On Something-Something-V2, Auto-TSNet-M improves the accuracy by at least 2% over other methods which use less than 50 GFLOPS per video.