End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning

TL;DR

This paper introduces MOHART, an end-to-end, class-agnostic multi-object tracking system with relational reasoning that models object interactions, improving tracking and trajectory prediction in complex scenes.

Contribution

It presents a novel multi-object tracking method that incorporates relational reasoning using permutation-invariant architectures and multi-headed attention, outperforming simpler models.

Findings

Relational reasoning improves tracking accuracy.

Multi-headed attention architecture outperforms DeepSets.

Modeling interactions enhances trajectory prediction in real-world datasets.

Abstract

The majority of contemporary object-tracking approaches do not model interactions between objects. This contrasts with the fact that objects' paths are not independent: a cyclist might abruptly deviate from a previously planned trajectory in order to avoid colliding with a car. Building upon HART, a neural class-agnostic single-object tracker, we introduce a multi-object tracking method MOHART capable of relational reasoning. Importantly, the entire system, including the understanding of interactions and relations between objects, is class-agnostic and learned simultaneously in an end-to-end fashion. We explore a number of relational reasoning architectures and show that permutation-invariant models outperform non-permutation-invariant alternatives. We also find that architectures using a single permutation invariant operation like DeepSets, despite, in theory, being universal function approximators, are nonetheless outperformed by a more complex architecture based on multi-headed attention. The latter better accounts for complex physical interactions in a challenging toy experiment. Further, we find that modelling interactions leads to consistent performance gains in tracking as well as future trajectory prediction on three real-world datasets (MOTChallenge, UA-DETRAC, and Stanford Drone dataset), particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs.