Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers

TL;DR

This paper investigates the spatial understanding capabilities of visual prompts generated by resamplers in vision-language models, revealing that joint training enhances spatial encoding and suggesting the need for object-aware pretraining objectives.

Contribution

The study introduces diagnostic classifiers to evaluate spatial information in visual prompts and demonstrates that joint training improves spatial encoding in resamplers.

Findings

Visual prompts lack spatial info when frozen during training.

Joint training significantly improves spatial encoding.

Object-aware pretraining could enhance spatial understanding.

Abstract

An effective method for combining frozen large language models (LLM) and visual encoders involves a resampler module that creates a `visual prompt' which is provided to the LLM, along with the textual prompt. While this approach has enabled impressive performance across many coarse-grained tasks like image captioning and visual question answering, more fine-grained tasks that require spatial understanding have not been thoroughly examined. In this paper, we use \textit{diagnostic classifiers} to measure the extent to which the visual prompt produced by the resampler encodes spatial information. Our results show that this information is largely absent from the resampler output when kept frozen during training of the classifiers. However, when the resampler and classifier are trained jointly, we observe a significant performance boost. This shows that the compression achieved by the resamplers can in principle encode the requisite spatial information, but that more object-aware objectives are needed at the pretraining stage to facilitate this capability