CLEVER: Stream-based Active Learning for Robust Semantic Perception from Human Instructions

TL;DR

CLEVER is a novel stream-based active learning system that enhances the robustness of DNNs for semantic perception tasks by incorporating human feedback and domain knowledge, demonstrated on real robots.

Contribution

It introduces a Bayesian formulation for online DNN adaptation with human-in-the-loop in stream settings, a first for real robot applications.

Findings

Improves DNN robustness in real-time perception tasks.

Validated through user studies and robot experiments.

Demonstrates effective online adaptation with human support.

Abstract

We propose CLEVER, an active learning system for robust semantic perception with Deep Neural Networks (DNNs). For data arriving in streams, our system seeks human support when encountering failures and adapts DNNs online based on human instructions. In this way, CLEVER can eventually accomplish the given semantic perception tasks. Our main contribution is the design of a system that meets several desiderata of realizing the aforementioned capabilities. The key enabler herein is our Bayesian formulation that encodes domain knowledge through priors. Empirically, we not only motivate CLEVER's design but further demonstrate its capabilities with a user validation study as well as experiments on humanoid and deformable objects. To our knowledge, we are the first to realize stream-based active learning on a real robot, providing evidence that the robustness of the DNN-based semantic perception can be improved in practice. The project website can be accessed at https://sites.google.com/view/thecleversystem.