Vision Large Language Models Are Good Noise Handlers in Engagement Analysis

TL;DR

This paper introduces a framework using Vision Large Language Models to refine noisy engagement annotations in videos, improving model training and surpassing state-of-the-art benchmarks.

Contribution

It proposes a novel annotation refinement and training strategy leveraging VLMs, curriculum learning, and soft labels to handle subjective noise in engagement datasets.

Findings

Improved engagement recognition performance on benchmarks.

Enhanced model robustness with refined annotations.

Surpassed previous state-of-the-art results.

Abstract

Engagement recognition in video datasets, unlike traditional image classification tasks, is particularly challenged by subjective labels and noise limiting model performance. To overcome the challenges of subjective and noisy engagement labels, we propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process. Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets. We also introduce a training strategy combining curriculum learning with soft label refinement, gradually incorporating ambiguous samples while adjusting supervision to reflect uncertainty. We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements, highlighting benefits of addressing label subjectivity with VLMs. This method surpasses prior state of the art across engagement benchmarks such as EngageNet (three of six feature settings, maximum improvement of +1.21%), and DREAMS / PAFE with F1 gains of +0.22 / +0.06.