# BehaveAI enables rapid detection and classification of objects and behavior from motion

## TL;DR

BehaveAI is a video analysis tool that uses motion as color to detect and classify objects and behaviors efficiently, even in complex scenes.

## Contribution

BehaveAI introduces a novel color-from-motion encoding strategy for video analysis that improves detection and classification of motion patterns.

## Key findings

- BehaveAI enables robust detection of objects in complex scenes using motion information.
- The framework supports real-time processing on low-end devices like a Raspberry Pi.
- It reduces annotation effort and allows model training on conventional computers.

## Abstract

Here we introduce BehaveAI, a biologically inspired video analysis framework that integrates static and motion information through a novel color-from-motion encoding strategy. This method translates object movement—direction, speed, and acceleration—into color gradients, enabling both human annotators and pre-trained convolutional neural networks (CNNs) to infer motion patterns while retaining high-resolution spatial detail. Using a range of case studies, we demonstrate how the increased salience of motion information allows for the robust detection of objects that are challenging or impossible to identify reliably from static frames alone, particularly in complex natural scenes. We further demonstrate the reliable classification of different behaviors in animals and single-celled organisms. Additionally, the framework supports flexible hierarchical model structures that can separate the tasks of detection and classification for optimal efficiency, and provide individual tracking data that specifies what is present where and what it is doing in each frame. The framework makes use of the latest deep learning architecture (YOLO11), combined with a semi-supervised annotation workflow. Together with salient motion information, these features can dramatically reduce the effort required for dataset annotation such that reliable models can often be made within an hour. Moreover, smaller annotation datasets mean that model training can be achieved on conventional computers without dedicated hardware, thereby improving accessibility. The motion encoding approach is also computationally lightweight, and can run in real-time on low-end edge devices such as a Raspberry Pi. We release the framework as a free, open source, and user-friendly package.

Despite decades of advancement in video processing, efficient and accurate quantification of complex motion information remains a computational challenge. This study presents BehaveAI, a biologically inspired video analysis tool that sees motion as color, able to track animals or objects, classify their behavior, and handle complex natural scenes.