PAMI-AD: An Activity Detector Exploiting Part-attention and Motion Information in Surveillance Videos

TL;DR

This paper introduces a comprehensive activity detection system for surveillance videos that leverages part-attention and motion information, achieving state-of-the-art results on large-scale datasets and winning a major challenge.

Contribution

It presents a novel activity detection framework with part-attention and localization masking modules tailored for person and vehicle activities in untrimmed videos.

Findings

Achieved top performance on VIRAT dataset

Won 1st place in TRECVID 2021 ActEV challenge

Outperformed existing methods in activity detection accuracy

Abstract

Activity detection in surveillance videos is a challenging task caused by small objects, complex activity categories, its untrimmed nature, etc. Existing methods are generally limited in performance due to inaccurate proposals, poor classifiers or inadequate post-processing method. In this work, we propose a comprehensive and effective activity detection system in untrimmed surveillance videos for person-centered and vehicle-centered activities. It consists of four modules, i.e., object localizer, proposal filter, activity classifier and activity refiner. For person-centered activities, a novel part-attention mechanism is proposed to explore detailed features in different body parts. As for vehicle-centered activities, we propose a localization masking method to jointly encode motion and foreground attention features. We conduct experiments on the large-scale activity detection datasets VIRAT, and achieve the best results for both groups of activities. Furthermore, our team won the 1st place in the TRECVID 2021 ActEV challenge.