Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

TL;DR

This paper systematically evaluates the uncertainty calibration of vision-language models in early action recognition for human-robot interaction, addressing safety and reliability in ambiguous, partial observations.

Contribution

It introduces a temporal-prefix evaluation protocol and metrics for assessing uncertainty calibration in vision-language models for action recognition.

Findings

Identifies miscalibration patterns under partial observations.

Provides reliability evidence for confidence-based human-robot interaction.

Characterizes failure modes in early action prediction.

Abstract

Robots in shared workspaces must interpret human actions from partial, ambiguous observations, where overconfident early predictions can lead to unsafe or disruptive interaction. This challenge is amplified in egocentric views, where viewpoint changes and occlusions increase perceptual noise and ambiguity. As a result, downstream human-robot interaction modules require not only an action hypothesis but also a trustworthy estimate of confidence under partial observation. Recent vision-language model-based approaches have been proposed for short-term action recognition due to their open-vocabulary and context-aware reasoning, but their uncertainty reliability in the temporal-prefix regime is largely uncharacterized. We present the first systematic evaluation of uncertainty in vision-language model-based short-term action recognition for human-robot interaction. We introduce a temporal-prefix evaluation protocol and metrics for calibration and selective prediction. We also characterize miscalibration patterns and failure modes under partial observations. Our study provides the missing reliability evidence needed to use vision-language model predictions in confidence-gated human-robot interaction modules.