MeMoSORT: Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking

TL;DR

MeMoSORT introduces a real-time multi-object tracking method that uses memory-augmented neural networks and adaptive IoU to improve accuracy in complex scenarios with occlusions and motion mismatches.

Contribution

The paper presents MeMoSORT, a novel online MOT tracker combining memory-assisted Kalman filtering and motion-adaptive IoU for enhanced tracking performance.

Findings

Achieves state-of-the-art HOTA scores on DanceTrack and SportsMOT datasets.

Effectively reduces identity switches and target loss in occlusion scenarios.

Outperforms conventional methods in complex human-dominant tracking environments.

Abstract

Multi-object tracking (MOT) in human-dominant scenarios, which involves continuously tracking multiple people within video sequences, remains a significant challenge in computer vision due to targets' complex motion and severe occlusions. Conventional tracking-by-detection methods are fundamentally limited by their reliance on Kalman filter (KF) and rigid Intersection over Union (IoU)-based association. The motion model in KF often mismatches real-world object dynamics, causing filtering errors, while rigid association struggles under occlusions, leading to identity switches or target loss. To address these issues, we propose MeMoSORT, a simple, online, and real-time MOT tracker with two key innovations. First, the Memory-assisted Kalman filter (MeKF) uses memory-augmented neural networks to compensate for mismatches between assumed and actual object motion. Second, the Motion-adaptive IoU (Mo-IoU) adaptively expands the matching space and incorporates height similarity to reduce the influence of detection errors and association failures, while remaining lightweight. Experiments on DanceTrack and SportsMOT show that MeMoSORT achieves state-of-the-art performance, with HOTA scores of 67.9\% and 82.1\%, respectively.