CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision

TL;DR

CookAR is a wearable AR system that uses computer vision to provide real-time visual cues of kitchen tool affordances, enhancing safety and independence for people with low vision during cooking.

Contribution

This work introduces a novel egocentric dataset, a fine-tuned affordance segmentation model, and an AR system for assistive kitchen interactions tailored for low vision users.

Findings

Model outperforms baseline on affordance dataset

Participants prefer affordance augmentations over whole object cues

System improves safety and efficiency in kitchen tasks for low vision users

Abstract

Cooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.