HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

TL;DR

HumanOmni is a large, specialized vision-speech language model designed for comprehensive understanding of human-centric videos, integrating visual and audio data with adaptive scene-specific processing.

Contribution

It introduces the first large-scale human-centric dataset and a multimodal model with specialized branches for improved scene understanding.

Findings

Outperforms existing models in emotion recognition and action understanding.

Effectively fuses visual and audio features for comprehensive scene analysis.

Demonstrates significant improvements in human-centric video comprehension tasks.

Abstract

In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.