VisualChef: Generating Visual Aids in Cooking via Mask Inpainting

TL;DR

VisualChef is a novel method that generates contextual visual aids for cooking by using mask inpainting to produce images of actions and outcomes, maintaining environmental consistency without relying on detailed textual annotations.

Contribution

It introduces a mask-based visual grounding approach for generating cooking visual aids, simplifying alignment and enabling targeted modifications based on action relevance.

Findings

Outperforms state-of-the-art methods quantitatively

Provides high-quality visual aids in cooking scenarios

Works effectively across multiple egocentric video datasets

Abstract

Cooking requires not only following instructions but also understanding, executing, and monitoring each step - a process that can be challenging without visual guidance. Although recipe images and videos offer helpful cues, they often lack consistency in focus, tools, and setup. To better support the cooking process, we introduce VisualChef, a method for generating contextual visual aids tailored to cooking scenarios. Given an initial frame and a specified action, VisualChef generates images depicting both the action's execution and the resulting appearance of the object, while preserving the initial frame's environment. Previous work aims to integrate knowledge extracted from large language models by generating detailed textual descriptions to guide image generation, which requires fine-grained visual-textual alignment and involves additional annotations. In contrast, VisualChef simplifies alignment through mask-based visual grounding. Our key insight is identifying action-relevant objects and classifying them to enable targeted modifications that reflect the intended action and outcome while maintaining a consistent environment. In addition, we propose an automated pipeline to extract high-quality initial, action, and final state frames. We evaluate VisualChef quantitatively and qualitatively on three egocentric video datasets and show its improvements over state-of-the-art methods.