LLaVA-Chef: A Multi-modal Generative Model for Food Recipes

TL;DR

LLaVA-Chef is a multi-modal generative model trained specifically on food recipes, combining visual and textual data to generate detailed and accurate recipes, advancing the application of large language models in culinary domains.

Contribution

The paper introduces LLaVA-Chef, a novel multi-stage training approach that adapts large language models to the food domain for improved recipe generation.

Findings

LLaVA-Chef outperforms existing models in recipe detail and ingredient accuracy.

The model effectively integrates visual food images with textual recipe generation.

Enhanced linguistic quality of generated recipes through a custom loss function.

Abstract

In the rapidly evolving landscape of online recipe sharing within a globalized context, there has been a notable surge in research towards comprehending and generating food recipes. Recent advancements in large language models (LLMs) like GPT-2 and LLaVA have paved the way for Natural Language Processing (NLP) approaches to delve deeper into various facets of food-related tasks, encompassing ingredient recognition and comprehensive recipe generation. Despite impressive performance and multi-modal adaptability of LLMs, domain-specific training remains paramount for their effective application. This work evaluates existing LLMs for recipe generation and proposes LLaVA-Chef, a novel model trained on a curated dataset of diverse recipe prompts in a multi-stage approach. First, we refine the mapping of visual food image embeddings to the language space. Second, we adapt LLaVA to the food domain by fine-tuning it on relevant recipe data. Third, we utilize diverse prompts to enhance the model's recipe comprehension. Finally, we improve the linguistic quality of generated recipes by penalizing the model with a custom loss function. LLaVA-Chef demonstrates impressive improvements over pretrained LLMs and prior works. A detailed qualitative analysis reveals that LLaVA-Chef generates more detailed recipes with precise ingredient mentions, compared to existing approaches.