ARChef: An iOS-Based Augmented Reality Cooking Assistant Powered by Multimodal Gemini LLM

TL;DR

This paper introduces an iOS-based augmented reality cooking assistant that integrates multimodal large language models and computer vision to enhance meal planning, ingredient identification, and nutritional guidance, improving accessibility and user experience.

Contribution

It presents a novel AR cooking app leveraging Gemini LLM and ARKit, offering personalized meal suggestions and nutritional info, addressing limitations of existing AR cooking tools.

Findings

High user satisfaction in surveys

Effective ingredient recognition and recipe generation

Enhanced accessibility for diverse users

Abstract

Cooking meals can be difficult, causing many to resort to cookbooks and online recipes. However, relying on these traditional methods of cooking often results in missing ingredients, nutritional hazards, and unsatisfactory meals. Using Augmented Reality (AR) can address these issues; however, current AR cooking applications have poor user interfaces and limited accessibility. This paper proposes a prototype of an iOS application that integrates AR and Computer Vision (CV) into the cooking process. We leverage Google's Gemini Large Language Model (LLM) to identify ingredients in the camera's field of vision and generate recipe choices with detailed nutritional information. Additionally, this application uses Apple's ARKit to create an AR user interface compatible with iOS devices. Users can personalize their meal suggestions by inputting their dietary preferences and rating each meal. The application's effectiveness is evaluated through three rounds of user experience surveys. This application advances the field of accessible cooking assistance technologies, aiming to reduce food wastage and improve the meal planning experience.