Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomalies

TL;DR

This paper presents a method for household robots to proactively detect hazards by generating diverse anomaly scenarios using foundational models and multi-agent collaboration, enhancing safety in home environments.

Contribution

It introduces a novel multi-agent scenario generation approach leveraging foundational models to create diverse, realistic environments for hazard detection training in robots.

Findings

Generated environments outperform others in diversity and realism.

Robots trained in these environments better detect household hazards.

Approach reduces reliance on manual data labeling.

Abstract

Existing household robots have made significant progress in performing routine tasks, such as cleaning floors or delivering objects. However, a key limitation of these robots is their inability to recognize potential problems or dangers in home environments. For example, a child may pick up and ingest medication that has fallen on the floor, posing a serious risk. We argue that household robots should proactively detect such hazards or anomalies within the home, and propose the task of anomaly scenario generation. We leverage foundational models instead of relying on manually labeled data to build simulated environments. Specifically, we introduce a multi-agent brainstorming approach, where agents collaborate and generate diverse scenarios covering household hazards, hygiene management, and child safety. These textual task descriptions are then integrated with designed 3D assets to simulate realistic environments. Within these constructed environments, the robotic agent learns the necessary skills to proactively discover and handle the proposed anomalies through task decomposition, and optimal learning approach selection. We demonstrate that our generated environment outperforms others in terms of task description and scene diversity, ultimately enabling robotic agents to better address potential household hazards.