DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution

TL;DR

DynRefer introduces a resolution-adaptive approach for multimodal models, mimicking human visual cognition to improve accuracy in region-level tasks like captioning and recognition, achieving state-of-the-art results.

Contribution

The paper presents DynRefer, a novel method that dynamically adapts resolution during training and inference for improved region-level multimodal task performance.

Findings

Improves accuracy in region-level captioning and recognition.

Achieves state-of-the-art results on multiple tasks.

Enhances model adaptability to human visual cognition.

Abstract

One fundamental task of multimodal models is to translate referred image regions to human preferred language descriptions. Existing methods, however, ignore the resolution adaptability needs of different tasks, which hinders them to find out precise language descriptions. In this study, we propose a DynRefer approach, to pursue high-accuracy region-level referring through mimicking the resolution adaptability of human visual cognition. During training, DynRefer stochastically aligns language descriptions of multimodal tasks with images of multiple resolutions, which are constructed by nesting a set of random views around the referred region. During inference, DynRefer performs selectively multimodal referring by sampling proper region representations for tasks from the nested views based on image and task priors. This allows the visual information for referring to better match human preferences, thereby improving the representational adaptability of region-level multimodal models. Experiments show that DynRefer brings mutual improvement upon broad tasks including region-level captioning, open-vocabulary region recognition and attribute detection. Furthermore, DynRefer achieves state-of-the-art results on multiple region-level multimodal tasks using a single model. Code is available at https://github.com/callsys/DynRefer.