SpotLight: Robotic Scene Understanding through Interaction and Affordance Detection

TL;DR

SpotLight is a framework that enables robots to understand and interact with functional household elements like light switches through affordance detection and physical interaction, improving scene understanding and task execution.

Contribution

The paper introduces a novel framework combining affordance prediction, interaction, and scene understanding, with a new dataset and detection model for light switches, extending to other functional elements.

Findings

Achieved up to 84% success in real-world light switch operation

Developed a new dataset with 715 images for light switch detection

Demonstrated robot learning through interaction and scene graph exploration

Abstract

Despite increasing research efforts on household robotics, robots intended for deployment in domestic settings still struggle with more complex tasks such as interacting with functional elements like drawers or light switches, largely due to limited task-specific understanding and interaction capabilities. These tasks require not only detection and pose estimation but also an understanding of the affordances these elements provide. To address these challenges and enhance robotic scene understanding, we introduce SpotLight: A comprehensive framework for robotic interaction with functional elements, specifically light switches. Furthermore, this framework enables robots to improve their environmental understanding through interaction. Leveraging VLM-based affordance prediction to estimate motion primitives for light switch interaction, we achieve up to 84% operation success in real world experiments. We further introduce a specialized dataset containing 715 images as well as a custom detection model for light switch detection. We demonstrate how the framework can facilitate robot learning through physical interaction by having the robot explore the environment and discover previously unknown relationships in a scene graph representation. Lastly, we propose an extension to the framework to accommodate other functional interactions such as swing doors, showcasing its flexibility. Videos and Code: timengelbracht.github.io/SpotLight/