BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change

TL;DR

This paper introduces the BAH dataset, a multimodal video dataset designed to recognize ambivalence and hesitancy in digital health interventions, and provides baseline benchmarks highlighting the need for advanced models.

Contribution

The paper presents the first dataset for automatic A/H recognition in videos, including annotations and baseline results, facilitating research in digital health behavior change.

Findings

Baseline models show limited performance, indicating the need for improved multimodal and spatio-temporal approaches.

The dataset contains 1,427 videos from 300 participants, annotated for A/H cues.

Publicly available data and code support further research in this area.

Abstract

Ambivalence and hesitancy (A/H), closely related constructs, are the primary reasons why individuals delay, avoid, or abandon health behaviour changes. They are subtle and conflicting emotions that sets a person in a state between positive and negative orientations, or between acceptance and refusal to do something. They manifest as a discord in affect between multiple modalities or within a modality, such as facial and vocal expressions, and body language. Although experts can be trained to recognize A/H as done for in-person interactions, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition is therefore critical for the personalization and cost-effectiveness of digital behaviour change interventions. However, no datasets currently exist for the design of machine learning models to recognize A/H. This paper introduces the Behavioural Ambivalence/Hesitancy (BAH) dataset collected for multimodal recognition of A/H in videos. It contains 1,427 videos with a total duration of 10.60 hours, captured from 300 participants across Canada, answering predefined questions to elicit A/H. It is intended to mirror real-world digital behaviour change interventions delivered online. BAH is annotated by three experts to provide timestamps that indicate where A/H occurs, and frame- and video-level annotations with A/H cues. Video transcripts, cropped and aligned faces, and participant metadata are also provided. Since A and H manifest similarly in practice, we provide a binary annotation indicating the presence or absence of A/H. Additionally, this paper includes benchmarking results using baseline models on BAH for frame- and video-level recognition, and different learning setups. The limited performance highlights the need for adapted multimodal and spatio-temporal models for A/H recognition. The data and code are publicly available.