Object-centric Binding in Contrastive Language-Image Pretraining

TL;DR

This paper introduces a novel binding module that enhances contrastive vision-language models' ability to understand complex scenes with multiple objects and relationships by integrating scene graphs and structured representations without hard-negative training.

Contribution

The work presents a new binding module that connects scene graphs with slot-structured image representations, improving compositional understanding in CLIP-like models without additional hard negatives.

Findings

Improved multi-object scene understanding performance.

Enhanced image-text matching accuracy for complex scenes.

Sample-efficient learning with better relational reasoning.

Abstract

Recent advances in vision language models (VLM) have been driven by contrastive models such as CLIP, which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from commonly used strategies, which rely on the design of hard-negative augmentations. Instead, our work focuses on integrating inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using any additional hard-negatives. To that end, we introduce a binding module that connects a scene graph, derived from a text description, with a slot-structured image representation, facilitating a structured similarity assessment between the two modalities. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model not only enhances the performance of CLIP-based models in multi-object compositional understanding but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.