Recognition of Heat-Induced Food State Changes by Time-Series Use of Vision-Language Model for Cooking Robot

TL;DR

This paper presents a novel approach using a vision-language model to recognize heat-induced food state changes in cooking robots, addressing a key challenge in robotic cooking tasks.

Contribution

It introduces a unified method leveraging time-series vision-language models for recognizing food state changes during cooking, validated with real robot data.

Findings

Effective recognition of four typical cooking state changes

Comparison of natural language prompts and image regions for accuracy

Demonstrated applicability in real robot cooking scenarios

Abstract

Cooking tasks are characterized by large changes in the state of the food, which is one of the major challenges in robot execution of cooking tasks. In particular, cooking using a stove to apply heat to the foodstuff causes many special state changes that are not seen in other tasks, making it difficult to design a recognizer. In this study, we propose a unified method for recognizing changes in the cooking state of robots by using the vision-language model that can discriminate open-vocabulary objects in a time-series manner. We collected data on four typical state changes in cooking using a real robot and confirmed the effectiveness of the proposed method. We also compared the conditions and discussed the types of natural language prompts and the image regions that are suitable for recognizing the state changes.