Detailed Object Description with Controllable Dimensions

TL;DR

This paper introduces Dimension Tailor, a training-free pipeline that refines object descriptions by focusing on user-specified dimensions, improving relevance and detail control in multimodal large language models for visually impaired assistance.

Contribution

The paper presents a novel, training-free method for refining object descriptions to emphasize user-specified dimensions, enhancing controllability and relevance in multimodal models.

Findings

Dimension Tailor improves description quality for user-specified details.

The pipeline enhances the performance of recent multimodal large language models.

It offers flexible inclusion or exclusion of object dimensions based on user needs.

Abstract

Object description plays an important role for visually impaired individuals to understand and compare the differences between objects. Recent multimodal large language models(MLLMs) exhibit powerful perceptual abilities and demonstrate impressive potential for generating object-centric descriptions. However, the descriptions generated by such models may still usually contain a lot of content that is not relevant to the user intent or miss some important object dimension details. Under special scenarios, users may only need the details of certain dimensions of an object. In this paper, we propose a training-free object description refinement pipeline, Dimension Tailor, designed to enhance user-specified details in object descriptions. This pipeline includes three steps: dimension extracting, erasing, and supplementing, which decompose the description into user-specified dimensions. Dimension Tailor can not only improve the quality of object details but also offer flexibility in including or excluding specific dimensions based on user preferences. We conducted extensive experiments to demonstrate the effectiveness of Dimension Tailor on controllable object descriptions. Notably, the proposed pipeline can consistently improve the performance of the recent MLLMs. The code is currently accessible at https://github.com/xin-ran-w/ControllableObjectDescription.