HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context

TL;DR

This paper introduces HumanOmniV2, a multimodal reasoning model enhanced with global context understanding and reward mechanisms, achieving superior performance on complex intent and emotion comprehension tasks.

Contribution

The paper presents a novel approach integrating global context reasoning, reward-based training, and a new benchmark for omni-modal understanding, advancing multimodal LLM capabilities.

Findings

Outperforms existing open-source models on omni-modal benchmarks.

Effectively addresses context understanding and shortcut problems.

Enhances reasoning accuracy with reward-based training mechanisms.

Abstract

With the rapid evolution of multimodal large language models, the capacity to deeply understand and interpret human intentions has emerged as a critical capability, which demands detailed and thoughtful reasoning. In recent studies, Reinforcement Learning (RL) has demonstrated potential in enhancing the reasoning capabilities of Large Language Models (LLMs). Nonetheless, the challenges associated with adapting RL to multimodal data and formats remain largely unaddressed. In this paper, we identify two issues in existing multimodal reasoning models: insufficient global context understanding and shortcut problems. Insufficient context understanding can happen when a model misinterprets multimodal context, resulting in incorrect answers. The shortcut problem occurs when the model overlooks crucial clues in multimodal inputs, directly addressing the query without considering the multimodal information. To tackle these issues, we emphasize the necessity for the model to reason with a clear understanding of the global context within multimodal inputs. This global context understanding can effectively prevent the model from overlooking key multimodal cues and ensure a thorough reasoning process. To ensure the accurate interpretation of multimodal context information, we implement a context reward judged by a large language model, alongside format and accuracy rewards. Additionally, to improve complex reasoning capability, we employ the LLM to assess the logical reward, determining whether the reasoning process successfully integrates multimodal information with logical methods. We also introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating models in understanding complex human intentions and emotions. Our proposed method demonstrates advanced performance across multiple omni-modal benchmarks compared to other open-source omni-modal models.