Differentiable Resolution Compression and Alignment for Efficient Video Classification and Retrieval

TL;DR

This paper introduces a differentiable resolution compression and alignment method for efficient video classification and retrieval, reducing computation while maintaining accuracy through a novel transformer-based approach.

Contribution

It proposes a differentiable compression and alignment mechanism that adaptively reduces resolution and captures global temporal correlations, improving efficiency without sacrificing performance.

Findings

Achieves the best efficiency-performance trade-off on near-duplicate video retrieval.

Provides competitive results on dynamic video classification.

Enables end-to-end optimization of the video representation network.

Abstract

Optimizing video inference efficiency has become increasingly important with the growing demand for video analysis in various fields. Some existing methods achieve high efficiency by explicit discard of spatial or temporal information, which poses challenges in fast-changing and fine-grained scenarios. To address these issues, we propose an efficient video representation network with Differentiable Resolution Compression and Alignment mechanism, which compresses non-essential information in the early stage of the network to reduce computational costs while maintaining consistent temporal correlations. Specifically, we leverage a Differentiable Context-aware Compression Module to encode the saliency and non-saliency frame features, refining and updating the features into a high-low resolution video sequence. To process the new sequence, we introduce a new Resolution-Align Transformer Layer to capture global temporal correlations among frame features with different resolutions, while reducing spatial computation costs quadratically by utilizing fewer spatial tokens in low-resolution non-saliency frames. The entire network can be end-to-end optimized via the integration of the differentiable compression module. Experimental results show that our method achieves the best trade-off between efficiency and performance on near-duplicate video retrieval and competitive results on dynamic video classification compared to state-of-the-art methods. Code:https://github.com/dun-research/DRCA