Learning in the Wild with Incremental Skeptical Gaussian Processes

TL;DR

This paper introduces ISGP, a Gaussian Process-based skeptical learning method that effectively handles noisy supervision and evolving class sets in real-world incremental classification tasks.

Contribution

The paper proposes a novel redesign of skeptical learning called ISGP, utilizing GPs to improve label querying and model robustness in noisy, dynamic environments.

Findings

ISGP outperforms original skeptical learning in noisy settings.

It maintains accuracy as new classes are added.

It effectively reduces over-confidence in predictions.

Abstract

The ability to learn from human supervision is fundamental for personal assistants and other interactive applications of AI. Two central challenges for deploying interactive learners in the wild are the unreliable nature of the supervision and the varying complexity of the prediction task. We address a simple but representative setting, incremental classification in the wild, where the supervision is noisy and the number of classes grows over time. In order to tackle this task, we propose a redesign of skeptical learning centered around Gaussian Processes (GPs). Skeptical learning is a recent interactive strategy in which, if the machine is sufficiently confident that an example is mislabeled, it asks the annotator to reconsider her feedback. In many cases, this is often enough to obtain clean supervision. Our redesign, dubbed ISGP, leverages the uncertainty estimates supplied by GPs to better allocate labeling and contradiction queries, especially in the presence of noise. Our experiments on synthetic and real-world data show that, as a result, while the original formulation of skeptical learning produces over-confident models that can fail completely in the wild, ISGP works well at varying levels of noise and as new classes are observed.