GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods

TL;DR

GIVL is a new vision-language pre-trained model designed to enhance geographical inclusivity by capturing regional visual concepts, leading to more balanced performance across diverse communities.

Contribution

The paper introduces GIVL with novel pre-training objectives IKM and IEC to improve regional knowledge understanding in vision-language models.

Findings

GIVL achieves state-of-the-art results on geo-diverse V&L tasks.

GIVL demonstrates more balanced regional performance.

Pre-training with IKM and IEC enhances geographical inclusivity.

Abstract

A key goal for the advancement of AI is to develop technologies that serve the needs not just of one group but of all communities regardless of their geographical region. In fact, a significant proportion of knowledge is locally shared by people from certain regions but may not apply equally in other regions because of cultural differences. If a model is unaware of regional characteristics, it may lead to performance disparity across regions and result in bias against underrepresented groups. We propose GIVL, a Geographically Inclusive Vision-and-Language Pre-trained model. There are two attributes of geo-diverse visual concepts which can help to learn geo-diverse knowledge: 1) concepts under similar categories have unique knowledge and visual characteristics, 2) concepts with similar visual features may fall in completely different categories. Motivated by the attributes, we design new pre-training objectives Image Knowledge Matching (IKM) and Image Edit Checking (IEC) to pre-train GIVL. Compared with similar-size models pre-trained with similar scale of data, GIVL achieves state-of-the-art (SOTA) and more balanced performance on geo-diverse V&L tasks.