Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

TL;DR

This paper introduces OSCAR, a pipeline using object status recognition to track recipe progress in non-visual cooking, aiding visually impaired individuals by providing real-time contextual feedback.

Contribution

The paper presents a novel pipeline for object status recognition in cooking, along with a real-world dataset and insights for improving assistive systems for visually impaired users.

Findings

Object status recognition improves step prediction accuracy.

Performance is affected by implicit tasks, camera placement, and lighting.

The pipeline is effective in real-world non-visual cooking scenarios.

Abstract

Cooking plays a vital role in everyday independence and well-being, yet remains challenging for people with vision impairments due to limited support for tracking progress and receiving contextual feedback. Object status - the condition or transformation of ingredients and tools - offers a promising but underexplored foundation for context-aware cooking support. In this paper, we present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking. OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking. We evaluate OSCAR on 173 instructional videos and a real-world dataset of 12 non-visual cooking sessions recorded by BLV individuals in their homes. Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions, such as implicit tasks, camera placement, and lighting. We contribute the pipeline of context-aware recipe progress tracking, an annotated real-world non-visual cooking dataset, and design insights to guide future context-aware assistive cooking systems.