Unsupervised Action Localization Crop in Video Retargeting for 3D ConvNets

TL;DR

This paper introduces an unsupervised method for cropping videos to focus on subjects for 3D CNNs, improving classification accuracy by maintaining subject visibility and reducing flickering through a novel shape fitting approach.

Contribution

It presents a new unsupervised video cropping technique using action localization and 3D shape fitting to enhance 3D CNN performance on untrimmed videos.

Findings

Outperforms random cropping in classification accuracy on benchmark datasets.

Maintains subject focus and reduces flickering in cropped videos.

Achieves higher accuracy with smaller or same-sized crops compared to larger random crops.

Abstract

Untrimmed videos on social media or those captured by robots and surveillance cameras are of varied aspect ratios. However, 3D CNNs usually require as input a square-shaped video, whose spatial dimension is smaller than the original. Random- or center-cropping may leave out the video's subject altogether. To address this, we propose an unsupervised video cropping approach by shaping this as a retargeting and video-to-video synthesis problem. The synthesized video maintains a 1:1 aspect ratio, is smaller in size and is targeted at video-subject(s) throughout the entire duration. First, action localization is performed on each frame by identifying patches with homogeneous motion patterns. Thus, a single salient patch is pinpointed per frame. But to avoid viewpoint jitters and flickering, any inter-frame scale or position changes among the patches should be performed gradually over time. This issue is addressed with a polyBezier fitting in 3D space that passes through some chosen pivot timestamps and whose shape is influenced by the in-between control timestamps. To corroborate the effectiveness of the proposed method, we evaluate the video classification task by comparing our dynamic cropping technique with random cropping on three benchmark datasets, viz. UCF-101, HMDB-51 and ActivityNet v1.3. The clip and top-1 accuracy for video classification after our cropping, outperform 3D CNN performances for same-sized random-crop inputs, also surpassing some larger random-crop sizes.