Robustness Testing of Multi-Modal Models in Varied Home Environments for Assistive Robots

TL;DR

This paper evaluates the robustness of multi-modal assistive robot models in varied home environments, emphasizing the importance of diverse sensory processing for real-world applicability.

Contribution

It introduces a framework for testing multi-modal models under environmental disturbances in virtual settings, highlighting the need for robustness in assistive robotics.

Findings

Models are sensitive to lighting and environmental changes.

Multi-modal integration improves task performance in varied conditions.

Assessment framework aids in identifying robustness gaps.

Abstract

The development of assistive robotic agents to support household tasks is advancing, yet the underlying models often operate in virtual settings that do not reflect real-world complexity. For assistive care robots to be effective in diverse environments, their models must be robust and integrate multiple modalities. Consider a caretaker needing assistance in a dimly lit room or navigating around a newly installed glass door. Models relying solely on visual input might fail in low light, while those using depth information could avoid the door. This demonstrates the necessity for models that can process various sensory inputs. Our ongoing study evaluates state-of-the-art robotic models in the AI2Thor virtual environment. We introduce disturbances, such as dimmed lighting and mirrored walls, to assess their impact on modalities like movement or vision, and object recognition. Our goal is to gather input from the Geriatronics community to understand and model the challenges faced by practitioners.