Unsupervised Motion Representation Enhanced Network for Action Recognition

TL;DR

This paper introduces UF-TSN, an end-to-end action recognition network that incorporates a lightweight unsupervised optical flow estimator, improving motion representation efficiency and accuracy without relying on costly pre-computed optical flow.

Contribution

The paper proposes a novel integrated unsupervised optical flow estimation module within an action recognition network, enhancing efficiency and accuracy over existing methods.

Findings

Achieves better accuracy than state-of-the-art unsupervised methods.

Maintains efficiency comparable to supervised approaches.

Effectively captures motion cues with a coarse-to-fine estimation process.

Abstract

Learning reliable motion representation between consecutive frames, such as optical flow, has proven to have great promotion to video understanding. However, the TV-L1 method, an effective optical flow solver, is time-consuming and expensive in storage for caching the extracted optical flow. To fill the gap, we propose UF-TSN, a novel end-to-end action recognition approach enhanced with an embedded lightweight unsupervised optical flow estimator. UF-TSN estimates motion cues from adjacent frames in a coarse-to-fine manner and focuses on small displacement for each level by extracting pyramid of feature and warping one to the other according to the estimated flow of the last level. Due to the lack of labeled motion for action datasets, we constrain the flow prediction with multi-scale photometric consistency and edge-aware smoothness. Compared with state-of-the-art unsupervised motion representation learning methods, our model achieves better accuracy while maintaining efficiency, which is competitive with some supervised or more complicated approaches.