A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task

TL;DR

This paper introduces a new Japanese recipe dataset with ingredient state annotations to evaluate and improve large language models' ability to understand and track ingredient states during cooking procedures.

Contribution

It presents a novel dataset and three tasks for assessing LLMs' understanding of ingredient state transitions in recipes, enhancing their procedural comprehension.

Findings

LLMs can learn ingredient state knowledge to improve cooking process understanding

Performance of Llama3.1-70B and Qwen2.5-72B approaches that of commercial LLMs

The dataset is publicly available for further research

Abstract

Large Language Models (LLMs) are trained on a vast amount of procedural texts, but they do not directly observe real-world phenomena. In the context of cooking recipes, this poses a challenge, as intermediate states of ingredients are often omitted, making it difficult for models to track ingredient states and understand recipes accurately. In this paper, we apply state probing, a method for evaluating a language model's understanding of the world, to the domain of cooking. We propose a new task and dataset for evaluating how well LLMs can recognize intermediate ingredient states during cooking procedures. We first construct a new Japanese recipe dataset with clear and accurate annotations of ingredient state changes, collected from well-structured and controlled recipe texts. Using this dataset, we design three novel tasks to evaluate whether LLMs can track ingredient state transitions and identify ingredients present at intermediate steps. Our experiments with widely used LLMs, such as Llama3.1-70B and Qwen2.5-72B, show that learning ingredient state knowledge improves their understanding of cooking processes, achieving performance comparable to commercial LLMs. The dataset are publicly available at: https://huggingface.co/datasets/mashi6n/nhkrecipe-100-anno-1