Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDL

TL;DR

This paper presents a robot cooking system that combines food state recognition, large language models, and classical planning to execute new recipes in real-world environments, demonstrating practical robot cooking capabilities.

Contribution

The study introduces an integrated system using foundation models and PDDL for real-world robot cooking based on recipe understanding and food state recognition.

Findings

Successful execution of new recipes by PR2 robot in real-world settings

Effective food state recognition with limited data using VLM

Demonstrated practical robot cooking with integrated planning and recognition

Abstract

Although there is a growing demand for cooking behaviours as one of the expected tasks for robots, a series of cooking behaviours based on new recipe descriptions by robots in the real world has not yet been realised. In this study, we propose a robot system that integrates real-world executable robot cooking behaviour planning using the Large Language Model (LLM) and classical planning of PDDL descriptions, and food ingredient state recognition learning from a small number of data using the Vision-Language model (VLM). We succeeded in experiments in which PR2, a dual-armed wheeled robot, performed cooking from arranged new recipes in a real-world environment, and confirmed the effectiveness of the proposed system.