# Everything robots need to know about cooking actions: creating actionable knowledge graphs to support robotic meal preparation

## TL;DR

This paper introduces a system that helps robots understand and execute cooking tasks by mapping recipe instructions to robotic actions using knowledge graphs.

## Contribution

The paper introduces Actionable Knowledge Graphs and Action Cores to systematically translate recipe instructions into robotic actions.

## Key findings

- Action Cores were matched to recipe verbs with 76.5% coverage using direct matching and cosine similarity.
- Neuro-symbolic methods were used to handle unmatched verbs, expanding the system's adaptability.
- The approach was validated in a real-world meal preparation scenario, demonstrating its practicality.

## Abstract

This paper addresses the challenge of enabling robots to autonomously prepare meals by bridging natural language recipe instructions and robotic action execution. We propose a novel methodology leveraging Actionable Knowledge Graphs to map recipe instructions into six core categories of robotic manipulation tasks, termed Action Cores cutting, pouring, mixing, preparing, pick and place, and cook and cool. Each AC is subdivided into Action Groups which represent a specific motion parameterization required for task execution. Using the Recipe1M + dataset (Marín et al., IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43, 187–203), encompassing over one million recipes, we systematically analysed action verbs and matched them to ACs by using direct matching and cosine similarity, achieving a coverage of 76.5%. For the unmatched verbs, we employ a neuro-symbolic approach, matching verbs to existing AGs or generating new action cores utilizing a Large Language Model Our findings highlight the versatility of AKGs in adapting general plans to specific robotic tasks, validated through an experimental application in a meal preparation scenario. This work sets a foundation for adaptive robotic systems capable of performing a wide array of complex culinary tasks with minimal human intervention.