Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

TL;DR

This paper proposes prompting strategies that incorporate geographic and socioeconomic attributes to enhance Large Multi-modal models' performance on underrepresented, lower-income data, addressing biases in training data.

Contribution

It introduces novel prompting techniques that leverage socioeconomic and geographic information to improve model fairness and performance on marginalized data groups.

Findings

Improved model performance on lower-income data.

Prompting strategies favoring low-income household topics.

Identified contexts with significant performance gains.

Abstract

Recent work has demonstrated that the unequal representation of cultures and socioeconomic groups in training data leads to biased Large Multi-modal (LMM) models. To improve LMM model performance on underrepresented data, we propose and evaluate several prompting strategies using non-English, geographic, and socioeconomic attributes. We show that these geographic and socioeconomic integrated prompts favor retrieving topic appearances commonly found in data from low-income households across different countries leading to improved LMM model performance on lower-income data. Our analyses identify and highlight contexts where these strategies yield the most improvements.