OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models

TL;DR

This paper introduces OV-HHIR, an open vocabulary human interaction recognition framework that uses large language models to describe interactions in open-world scenarios, overcoming fixed-vocabulary limitations.

Contribution

It presents a novel open vocabulary recognition method leveraging large language models and creates a comprehensive dataset for human interaction understanding.

Findings

Outperforms traditional fixed-vocabulary systems

Effective in recognizing unseen interactions

Sets new benchmarks in open-world interaction recognition

Abstract

Understanding human-to-human interactions, especially in contexts like public security surveillance, is critical for monitoring and maintaining safety. Traditional activity recognition systems are limited by fixed vocabularies, predefined labels, and rigid interaction categories that often rely on choreographed videos and overlook concurrent interactive groups. These limitations make such systems less adaptable to real-world scenarios, where interactions are diverse and unpredictable. In this paper, we propose an open vocabulary human-to-human interaction recognition (OV-HHIR) framework that leverages large language models to generate open-ended textual descriptions of both seen and unseen human interactions in open-world settings without being confined to a fixed vocabulary. Additionally, we create a comprehensive, large-scale human-to-human interaction dataset by standardizing and combining existing public human interaction datasets into a unified benchmark. Extensive experiments demonstrate that our method outperforms traditional fixed-vocabulary classification systems and existing cross-modal language models for video understanding, setting the stage for more intelligent and adaptable visual understanding systems in surveillance and beyond.