Retrieval Augmented Recipe Generation

TL;DR

This paper introduces a retrieval-augmented large multimodal model with stochastic retrieval and self-consistency voting to improve recipe generation from food images, achieving state-of-the-art results on Recipe1M.

Contribution

The paper proposes a novel retrieval-augmented approach with SDRA and self-consistency voting to enhance recipe generation accuracy and diversity.

Findings

Achieves state-of-the-art performance on Recipe1M dataset.

Effectively reduces hallucinations in recipe generation.

Enhances diversity and relevance of generated recipes.

Abstract

Given the potential applications of generating recipes from food images, this area has garnered significant attention from researchers in recent years. Existing works for recipe generation primarily utilize a two-stage training method, first generating ingredients and then obtaining instructions from both the image and ingredients. Large Multi-modal Models (LMMs), which have achieved notable success across a variety of vision and language tasks, shed light to generating both ingredients and instructions directly from images. Nevertheless, LMMs still face the common issue of hallucinations during recipe generation, leading to suboptimal performance. To tackle this, we propose a retrieval augmented large multimodal model for recipe generation. We first introduce Stochastic Diversified Retrieval Augmentation (SDRA) to retrieve recipes semantically related to the image from an existing datastore as a supplement, integrating them into the prompt to add diverse and rich context to the input image. Additionally, Self-Consistency Ensemble Voting mechanism is proposed to determine the most confident prediction recipes as the final output. It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation. Extensive experiments validate the effectiveness of our proposed method, which demonstrates state-of-the-art (SOTA) performance in recipe generation tasks on the Recipe1M dataset.