What are Foundation Models Cooking in the Post-Soviet World?

TL;DR

This paper examines how well foundation models understand Post-Soviet culinary culture, revealing significant limitations in identifying dish origins and proposing new multimodal datasets and evaluation methods.

Contribution

Introduces BORSch, a new multimodal dataset for Post-Soviet dishes, and analyzes foundation models' cultural understanding through QA and visual description tasks.

Findings

Models struggle to identify dish origins accurately.

QA performance is limited and influenced by language biases.

Visual description tasks reveal additional cultural understanding gaps.

Abstract

The culture of the Post-Soviet states is complex, shaped by a turbulent history that continues to influence current events. In this study, we investigate the Post-Soviet cultural food knowledge of foundation models by constructing BORSch, a multimodal dataset encompassing 1147 and 823 dishes in the Russian and Ukrainian languages, centered around the Post-Soviet region. We demonstrate that leading models struggle to correctly identify the origins of dishes from Post-Soviet nations in both text-only and multimodal Question Answering (QA), instead over-predicting countries linked to the language the question is asked in. Through analysis of pretraining data, we show that these results can be explained by misleading dish-origin co-occurrences, along with linguistic phenomena such as Russian-Ukrainian code mixing. Finally, to move beyond QA-based assessments, we test models' abilities to produce accurate visual descriptions of dishes. The weak correlation between this task and QA suggests that QA alone may be insufficient as an evaluation of cultural understanding. To foster further research, we will make BORSch publicly available at https://github.com/alavrouk/BORSch.