Watching Swarm Dynamics from Above: A Framework for Advanced Object Tracking in Drone Videos

TL;DR

This paper introduces a novel drone-based framework that combines probabilistic particle filtering and semantic segmentation to track and analyze the collective behavior of fish schools in their natural environment from long-duration videos.

Contribution

It presents a new method for long-term, 2D object tracking in drone videos that fuses video data with drone sensor information to study animal social behavior.

Findings

Effective tracking of fish schools in open ocean environments.

Integration of semantic segmentation with particle filtering enhances tracking accuracy.

First framework enabling non-invasive, scalable study of collective animal behavior in natural habitats.

Abstract

Easily accessible sensors, like drones with diverse onboard sensors, have greatly expanded studying animal behavior in natural environments. Yet, analyzing vast, unlabeled video data, often spanning hours, remains a challenge for machine learning, especially in computer vision. Existing approaches often analyze only a few frames. Our focus is on long-term animal behavior analysis. To address this challenge, we utilize classical probabilistic methods for state estimation, such as particle filtering. By incorporating recent advancements in semantic object segmentation, we enable continuous tracking of rapidly evolving object formations, even in scenarios with limited data availability. Particle filters offer a provably optimal algorithmic structure for recursively adding new incoming information. We propose a novel approach for tracking schools of fish in the open ocean from drone videos. Our framework not only performs classical object tracking in 2D, instead it tracks the position and spatial expansion of the fish school in world coordinates by fusing video data and the drone's on board sensor information (GPS and IMU). The presented framework for the first time allows researchers to study collective behavior of fish schools in its natural social and environmental context in a non-invasive and scalable way.