Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning

TL;DR

This paper introduces egocentric instruction tuning and a new benchmark to improve and evaluate multimodal large language models' ability to understand object orientation from a user's perspective, addressing annotation inconsistencies.

Contribution

It proposes egocentric instruction tuning for better orientation understanding and introduces EgoOrientBench, a benchmark for evaluating this capability in MLLMs.

Findings

Egocentric instruction tuning significantly improves orientation understanding.

The approach maintains overall MLLM performance.

EgoOrientBench effectively evaluates orientation comprehension.

Abstract

Multimodal large language models (MLLMs) act as essential interfaces, connecting humans with AI technologies in multimodal applications. However, current MLLMs face challenges in accurately interpreting object orientation in images due to inconsistent orientation annotations in training data, hindering the development of a coherent orientation understanding. To overcome this, we propose egocentric instruction tuning, which aligns MLLMs' orientation understanding with the user's perspective, based on a consistent annotation standard derived from the user's egocentric viewpoint. We first generate egocentric instruction data that leverages MLLMs' ability to recognize object details and applies prior knowledge for orientation understanding. Using this data, we perform instruction tuning to enhance the model's capability for accurate orientation interpretation. In addition, we introduce EgoOrientBench, a benchmark that evaluates MLLMs' orientation understanding across three tasks using images collected from diverse domains. Experimental results on this benchmark show that egocentric instruction tuning significantly improves orientation understanding without compromising overall MLLM performance. The instruction data and benchmark dataset are available on our project page at https://github.com/jhCOR/EgoOrientBench.