Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association

TL;DR

This paper introduces BiAug, a bimodal augmentation method that uses object-attribute decoupling and synthetic data generation to reduce reporting bias in visual-language datasets, enhancing model understanding and zero-shot retrieval performance.

Contribution

The paper presents a novel bimodal augmentation approach leveraging LLMs and inpainting to explicitly address reporting bias in object-attribute associations within visual-language datasets.

Findings

BiAug improves object-attribute understanding.

BiAug enhances zero-shot retrieval on MSCOCO and Flickr30K.

Mitigating reporting bias leads to richer visual-language models.

Abstract

Reporting bias arises when people assume that some knowledge is universally understood and hence, do not necessitate explicit elaboration. In this paper, we focus on the wide existence of reporting bias in visual-language datasets, embodied as the object-attribute association, which can subsequentially degrade models trained on them. To mitigate this bias, we propose a bimodal augmentation (BiAug) approach through object-attribute decoupling to flexibly synthesize visual-language examples with a rich array of object-attribute pairing and construct cross-modal hard negatives. We employ large language models (LLMs) in conjunction with a grounding object detector to extract target objects. Subsequently, the LLM generates a detailed attribute description for each object and produces a corresponding hard negative counterpart. An inpainting model is then used to create images based on these detailed object descriptions. By doing so, the synthesized examples explicitly complement omitted objects and attributes to learn, and the hard negative pairs steer the model to distinguish object attributes. Our experiments demonstrated that BiAug is superior in object-attribute understanding. In addition, BiAug also improves the performance on zero-shot retrieval tasks on general benchmarks like MSCOCO and Flickr30K. BiAug refines the way of collecting text-image datasets. Mitigating the reporting bias helps models achieve a deeper understanding of visual-language phenomena, expanding beyond mere frequent patterns to encompass the richness and diversity of real-world scenarios.