Conformal Predictions for Human Action Recognition with Vision-Language Models

TL;DR

This paper explores how Conformal Prediction techniques can improve the reliability of human action recognition systems based on Vision-Language Models by reducing candidate classes and tuning prediction confidence.

Contribution

It introduces a method to apply Conformal Prediction to VLM-based HAR systems, reducing candidate classes without extra data and tuning softmax temperature to handle distribution tails.

Findings

CP reduces average candidate classes significantly

Tuning softmax temperature mitigates long-tail distribution issues

Method enhances reliability of human-AI interaction in real-world scenarios

Abstract

Human-in-the-Loop (HITL) systems are essential in high-stakes, real-world applications where AI must collaborate with human decision-makers. This work investigates how Conformal Prediction (CP) techniques, which provide rigorous coverage guarantees, can enhance the reliability of state-of-the-art human action recognition (HAR) systems built upon Vision-Language Models (VLMs). We demonstrate that CP can significantly reduce the average number of candidate classes without modifying the underlying VLM. However, these reductions often result in distributions with long tails which can hinder their practical utility. To mitigate this, we propose tuning the temperature of the softmax prediction, without using additional calibration data. This work contributes to ongoing efforts for multi-modal human-AI interaction in dynamic real-world environments.