The ADVANCE toolkit: Automated descriptive video annotation in naturalistic child environments

TL;DR

The ADVANCE toolkit automates video annotation for naturalistic child environments, reducing manual work and improving accuracy in behavior analysis.

Contribution

ADVANCE introduces an automated pipeline for video annotation in unscripted, naturalistic settings with multiple individuals.

Findings

ADVANCE accurately detects and tracks individuals in dynamic, unscripted environments.

The toolkit estimates skeletal joint positions and labels poses with high precision.

It reduces clinical workload and enhances ethological validity in video-based assessments.

Abstract

Video recordings are commonplace for observing human and animal behaviours, including interindividual interactions. In studies of humans, analyses for clinical applications remain particularly cumbersome, requiring human-based annotation that is time-consuming, bias-prone, and cost-ineffective. Attempts to use machine learning to address these limitations still oftentimes require highly standardised environments, scripted scenarios, and forward-facing individuals. Here, we provide the ADVANCE toolkit, an automated video annotation pipeline. The versatility of ADVANCE is demonstrated with schoolchildren and adults in an unscripted clinical setting within an art classroom environment that included 2–5 individuals, dynamic occlusions, and large variations in actions. We accurately detected each individual, tracked them simultaneously throughout the duration of the recording (including when an individual left and re-entered the field of view), estimated the position of their skeletal joints, and labelled their poses. By resolving challenges of manual annotation, we radically enhance the ability to extract information from video recordings across different scenarios and settings. This toolkit reduces clinical workload and enhances the ethological validity of video-based assessments, offering scalable solutions for behaviour analyses in naturalistic contexts.

Figures