Liquid Pouring Monitoring via Rich Sensory Inputs

TL;DR

This paper presents a hierarchical LSTM-based system that uses synchronized visual and inertial sensory data to monitor liquid pouring success, incorporating auxiliary tasks to enhance robustness and accuracy.

Contribution

It introduces a novel multi-sensory monitoring framework with auxiliary tasks for improved robustness in liquid pouring success detection.

Findings

Achieves ~8% better accuracy on unseen containers.

Achieves ~11% better accuracy on unseen users.

Utilizes auxiliary tasks to enhance representation learning.

Abstract

Humans have the amazing ability to perform very subtle manipulation task using a closed-loop control system with imprecise mechanics (i.e., our body parts) but rich sensory information (e.g., vision, tactile, etc.). In the closed-loop system, the ability to monitor the state of the task via rich sensory information is important but often less studied. In this work, we take liquid pouring as a concrete example and aim at learning to continuously monitor whether liquid pouring is successful (e.g., no spilling) or not via rich sensory inputs. We mimic humans' rich sensories using synchronized observation from a chest-mounted camera and a wrist-mounted IMU sensor. Given many success and failure demonstrations of liquid pouring, we train a hierarchical LSTM with late fusion for monitoring. To improve the robustness of the system, we propose two auxiliary tasks during training: inferring (1) the initial state of containers and (2) forecasting the one-step future 3D trajectory of the hand with an adversarial training procedure. These tasks encourage our method to learn representation sensitive to container states and how objects are manipulated in 3D. With these novel components, our method achieves ~8% and ~11% better monitoring accuracy than the baseline method without auxiliary tasks on unseen containers and unseen users respectively.