The World Wide recipe: A community-centred framework for fine-grained data collection and regional bias operationalisation

TL;DR

This paper presents a culturally aware framework and dataset for evaluating regional biases in AI-generated dishes, revealing significant inaccuracies and cultural insensitivity, especially in non-US regions.

Contribution

It introduces the World Wide recipe framework and dataset for assessing regional bias in AI, highlighting current system shortcomings and promoting culturally sensitive data collection.

Findings

AI models underperform in generating region-specific dishes

Models produce culturally insensitive and inaccurate outputs

US dishes are better represented than African regions

Abstract

We introduce the World Wide recipe, which sets forth a framework for culturally aware and participatory data collection, and the resultant regionally diverse World Wide Dishes evaluation dataset. We also analyse bias operationalisation to highlight how current systems underperform across several dimensions: (in-)accuracy, (mis-)representation, and cultural (in-)sensitivity, with evidence from qualitative community-based observations and quantitative automated tools. We find that these T2I models generally do not produce quality outputs of dishes specific to various regions. This is true even for the US, which is typically considered more well-resourced in training data -- although the generation of US dishes does outperform that of the investigated African countries. The models demonstrate the propensity to produce inaccurate and culturally misrepresentative, flattening, and insensitive outputs. These representational biases have the potential to further reinforce stereotypes and disproportionately contribute to erasure based on region. The dataset and code are available at https://github.com/oxai/world-wide-dishes.