Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval

TL;DR

This paper introduces a causal approach to improve image-to-recipe retrieval by addressing biases in cross-modal representations, especially for multicultural cuisines, leading to better retrieval of subtle culinary details.

Contribution

It proposes a novel causal method that predicts overlooked culinary elements and explicitly incorporates them into cross-modal learning, enhancing retrieval accuracy across diverse cuisines.

Findings

Improved retrieval performance on Recipe1M and multicultural datasets.

Effective uncovering of subtle ingredients and cooking actions.

Mitigation of representation bias in multicultural cuisine retrieval.

Abstract

Existing approaches for image-to-recipe retrieval have the implicit assumption that a food image can fully capture the details textually documented in its recipe. However, a food image only reflects the visual outcome of a cooked dish and not the underlying cooking process. Consequently, learning cross-modal representations to bridge the modality gap between images and recipes tends to ignore subtle, recipe-specific details that are not visually apparent but are crucial for recipe retrieval. Specifically, the representations are biased to capture the dominant visual elements, resulting in difficulty in ranking similar recipes with subtle differences in use of ingredients and cooking methods. The bias in representation learning is expected to be more severe when the training data is mixed of images and recipes sourced from different cuisines. This paper proposes a novel causal approach that predicts the culinary elements potentially overlooked in images, while explicitly injecting these elements into cross-modal representation learning to mitigate biases. Experiments are conducted on the standard monolingual Recipe1M dataset and a newly curated multilingual multicultural cuisine dataset. The results indicate that the proposed causal representation learning is capable of uncovering subtle ingredients and cooking actions and achieves impressive retrieval performance on both monolingual and multilingual multicultural datasets.