NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification

TL;DR

NeXtVLAD is a novel, efficient neural network architecture designed to aggregate frame-level features into compact vectors, significantly improving large-scale video classification performance with fewer parameters.

Contribution

The paper proposes NeXtVLAD, a new aggregation method that decomposes high-dimensional features into low-dimensional groups with attention, enhancing efficiency and effectiveness over existing methods.

Findings

Achieved 0.87846 GAP score with a single NeXtVLAD model on YouTube-8M.

Ensemble of three NeXtVLAD models reached 0.88722 GAP score.

Secured 3rd place in the YouTube-8M challenge among 394 teams.

Abstract

This paper introduces a fast and efficient network architecture, NeXtVLAD, to aggregate frame-level features into a compact feature vector for large-scale video classification. Briefly speaking, the basic idea is to decompose a high-dimensional feature into a group of relatively low-dimensional vectors with attention before applying NetVLAD aggregation over time. This NeXtVLAD approach turns out to be both effective and parameter efficient in aggregating temporal information. In the 2nd Youtube-8M video understanding challenge, a single NeXtVLAD model with less than 80M parameters achieves a GAP score of 0.87846 in private leaderboard. A mixture of 3 NeXtVLAD models results in 0.88722, which is ranked 3rd over 394 teams. The code is publicly available at https://github.com/linrongc/youtube-8m.