Learning Behavior Representations Through Multi-Timescale Bootstrapping

TL;DR

This paper introduces BAMS, a multi-scale behavior representation learning model that captures complex temporal dynamics in naturalistic settings, outperforming prior single-scale models.

Contribution

The paper proposes a novel multi-timescale bootstrap learning framework for behavior representation, enabling disentanglement across different temporal scales.

Findings

Captures temporal complexity in quadruped navigation data.

Ranks 3rd overall and 1st on two subtasks in MABe 2022 challenge.

Demonstrates the importance of multi-timescale analysis in behavior modeling.

Abstract

Natural behavior consists of dynamics that are both unpredictable, can switch suddenly, and unfold over many different timescales. While some success has been found in building representations of behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings due to the fact that they assume a single scale of temporal dynamics. In this work, we introduce Bootstrap Across Multiple Scales (BAMS), a multi-scale representation learning model for behavior: we combine a pooling module that aggregates features extracted over encoders with different temporal receptive fields, and design a set of latent objectives to bootstrap the representations in each respective space to encourage disentanglement across different timescales. We first apply our method on a dataset of quadrupeds navigating in different terrain types, and show that our model captures the temporal complexity of behavior. We then apply our method to the MABe 2022 Multi-agent behavior challenge, where our model ranks 3rd overall and 1st on two subtasks, and show the importance of incorporating multi-timescales when analyzing behavior.