Causal Graphical Models for Vision-Language Compositional Understanding

TL;DR

This paper introduces a causal graphical model approach to improve vision-language models' understanding of compositional language, significantly enhancing performance on benchmark tasks by focusing on causal dependencies.

Contribution

The paper proposes a novel causal graphical model framework that captures dependency relations in vision-language tasks, outperforming existing methods on multiple benchmarks.

Findings

Significant performance improvements on five compositional benchmarks.

Outperforms state-of-the-art approaches by a large margin.

Effective in learning causal dependencies, reducing spurious correlations.

Abstract

Recent work has empirically shown that Vision-Language Models (VLMs) struggle to fully understand the compositional properties of the human language, usually modeling an image caption as a "bag of words". As a result, they perform poorly on compositional tasks, which require a deeper understanding of the different entities of a sentence (subject, verb, etc.) jointly with their mutual relationships in order to be solved. In this paper, we model the dependency relations among textual and visual tokens using a Causal Graphical Model (CGM), built using a dependency parser, and we train a decoder conditioned by the VLM visual encoder. Differently from standard autoregressive or parallel predictions, our decoder's generative process is partially-ordered following the CGM structure. This structure encourages the decoder to learn only the main causal dependencies in a sentence discarding spurious correlations. Using extensive experiments on five compositional benchmarks, we show that our method significantly outperforms all the state-of-the-art compositional approaches by a large margin, and it also improves over methods trained using much larger datasets.