Teaching Structured Vision&Language Concepts to Vision&Language Models

TL;DR

This paper enhances vision-language models' understanding of complex concepts like object attributes and relations by leveraging language structure, leading to significant improvements without additional data collection.

Contribution

It introduces a data-driven method that uses language structure understanding to improve VL models' grasp of structured vision&language concepts without extra datasets.

Findings

Up to 15% improvement in SVLC understanding.

Minimal impact on zero-shot performance.

Effective use of existing datasets for training.

Abstract

Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks. However, some aspects of complex language understanding still remain a challenge. We introduce the collective notion of Structured Vision&Language Concepts (SVLC) which includes object attributes, relations, and states which are present in the text and visible in the image. Recent studies have shown that even the best VL models struggle with SVLC. A possible way of fixing this issue is by collecting dedicated datasets for teaching each SVLC type, yet this might be expensive and time-consuming. Instead, we propose a more elegant data-driven approach for enhancing VL models' understanding of SVLCs that makes more effective use of existing VL pre-training datasets and does not require any additional data. While automatic understanding of image structure still remains largely unsolved, language structure is much better modeled and understood, allowing for its effective utilization in teaching VL models. In this paper, we propose various techniques based on language structure understanding that can be used to manipulate the textual part of off-the-shelf paired VL datasets. VL models trained with the updated data exhibit a significant improvement of up to 15% in their SVLC understanding with only a mild degradation in their zero-shot capabilities both when training from scratch or fine-tuning a pre-trained model.