Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression

TL;DR

This study introduces a vision-language model that classifies mouse behaviors from videos with high accuracy, producing detailed behavioral datasets without fine-tuning, aiding interdisciplinary research.

Contribution

It demonstrates the effective use of a vision-language model with prompts and in-context learning for behavior classification without model fine-tuning.

Findings

High F1 scores across behaviors, including rare ones

Enhanced performance through prompts, ICL, and preprocessing

Minimal user input needed for accurate classification

Abstract

Integration of diverse data will be a pivotal step towards improving scientific explorations in many disciplines. This work establishes a vision-language model (VLM) that encodes videos with text input in order to classify various behaviors of a mouse existing in and engaging with their environment. Importantly, this model produces a behavioral vector over time for each subject and for each session the subject undergoes. The output is a valuable dataset that few programs are able to produce with as high accuracy and with minimal user input. Specifically, we use the open-source Qwen2.5-VL model and enhance its performance through prompts, in-context learning (ICL) with labeled examples, and frame-level preprocessing. We found that each of these methods contributes to improved classification, and that combining them results in strong F1 scores across all behaviors, including rare classes like freezing and fleeing, without any model fine-tuning. Overall, this model will support interdisciplinary researchers studying mouse behavior by enabling them to integrate diverse behavioral features, measured across multiple time points and environments, into a comprehensive dataset that can address complex research questions.