Sampling Training Data for Continual Learning Between Robots and the Cloud

TL;DR

HarvestNet is an on-board sampling algorithm for robots that selectively stores rare and valuable sensory events, significantly reducing cloud storage and computation costs while improving perception model accuracy.

Contribution

The paper introduces HarvestNet, a novel on-board sampling method that enhances data efficiency and model accuracy for continual learning in robotic perception.

Findings

Reduces cloud storage and annotation time by up to 81.3%.

Improves model accuracy by 1.05-2.58x over baseline algorithms.

Scales effectively on embedded deep learning hardware.

Abstract

Today's robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data, such as rare scenes of road construction sites, to steadily improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central compute servers (or the "cloud") places an enormous time and cost burden on network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to steadily improve perception models re-trained in the cloud. HarvestNet significantly improves the accuracy of machine-learning models on our novel dataset of road construction sites, field testing of self-driving cars, and streaming face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline algorithms and scalably runs on embedded deep learning hardware. We provide a suite of compute-efficient perception models for the Google Edge Tensor Processing Unit (TPU), an extended technical report, and a novel video dataset to the research community at https://sites.google.com/view/harvestnet.