Spatio-temporal Graph Learning on Adaptive Mined Key Frames for High-performance Multi-Object Tracking

TL;DR

This paper introduces a novel multi-object tracking method that uses adaptive key frame mining and intra-frame graph-based feature fusion to improve tracking accuracy, especially under occlusion conditions.

Contribution

It proposes a reinforcement learning-based key frame extraction and a graph convolutional network for intra-frame feature fusion, enhancing object association and occlusion handling in tracking.

Findings

Achieves 68.6 HOTA on MOT17 dataset

Improves IDF1 to 81.0, reducing ID switches

Outperforms existing methods in occlusion scenarios

Abstract

In the realm of multi-object tracking, the challenge of accurately capturing the spatial and temporal relationships between objects in video sequences remains a significant hurdle. This is further complicated by frequent occurrences of mutual occlusions among objects, which can lead to tracking errors and reduced performance in existing methods. Motivated by these challenges, we propose a novel adaptive key frame mining strategy that addresses the limitations of current tracking approaches. Specifically, we introduce a Key Frame Extraction (KFE) module that leverages reinforcement learning to adaptively segment videos, thereby guiding the tracker to exploit the intrinsic logic of the video content. This approach allows us to capture structured spatial relationships between different objects as well as the temporal relationships of objects across frames. To tackle the issue of object occlusions, we have developed an Intra-Frame Feature Fusion (IFF) module. Unlike traditional graph-based methods that primarily focus on inter-frame feature fusion, our IFF module uses a Graph Convolutional Network (GCN) to facilitate information exchange between the target and surrounding objects within a frame. This innovation significantly enhances target distinguishability and mitigates tracking loss and appearance similarity due to occlusions. By combining the strengths of both long and short trajectories and considering the spatial relationships between objects, our proposed tracker achieves impressive results on the MOT17 dataset, i.e., 68.6 HOTA, 81.0 IDF1, 66.6 AssA, and 893 IDS, proving its effectiveness and accuracy.