RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios

TL;DR

RefHCM introduces a unified transformer-based framework for diverse human-centric referring tasks, integrating multimodal data and enabling complex reasoning, thus advancing applicability in real-world scenarios like chatbots and sports analysis.

Contribution

This work is the first to propose a general-purpose model for referring human perceptions, unifying multiple tasks and establishing a new benchmark in the field.

Findings

RefHCM achieves state-of-the-art performance across multiple tasks.

The model demonstrates strong generalization and reasoning capabilities.

Extensive experiments validate its effectiveness and versatility.

Abstract

Human-centric perceptions play a crucial role in real-world applications. While recent human-centric works have achieved impressive progress, these efforts are often constrained to the visual domain and lack interaction with human instructions, limiting their applicability in broader scenarios such as chatbots and sports analysis. This paper introduces Referring Human Perceptions, where a referring prompt specifies the person of interest in an image. To tackle the new task, we propose RefHCM (Referring Human-Centric Model), a unified framework to integrate a wide range of human-centric referring tasks. Specifically, RefHCM employs sequence mergers to convert raw multimodal data -- including images, text, coordinates, and parsing maps -- into semantic tokens. This standardized representation enables RefHCM to reformulate diverse human-centric referring tasks into a sequence-to-sequence paradigm, solved using a plain encoder-decoder transformer architecture. Benefiting from a unified learning strategy, RefHCM effectively facilitates knowledge transfer across tasks and exhibits unforeseen capabilities in handling complex reasoning. This work represents the first attempt to address referring human perceptions with a general-purpose framework, while simultaneously establishing a corresponding benchmark that sets new standards for the field. Extensive experiments showcase RefHCM's competitive and even superior performance across multiple human-centric referring tasks. The code and data are publicly at https://github.com/JJJYmmm/RefHCM.