From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback

TL;DR

This paper introduces CLIC, a human-in-the-loop method that uses set-valued action targets from human corrections to improve imitation learning robustness, especially with imperfect feedback.

Contribution

It proposes a novel approach that replaces pointwise action labels with set-valued targets, enhancing robustness to noisy and partial human feedback in imitation learning.

Findings

CLIC performs well with accurate data, comparable to state-of-the-art methods.

It is significantly more robust under noisy, relative, and partial feedback.

Experiments include both simulation and real-robot scenarios.

Abstract

Behavior cloning (BC) optimizes policies by treating human demonstrations as pointwise action labels. While effective with accurate action labels, this formulation is brittle in practice: when human-provided actions are imperfect, treating each label as an exact target can steer the policy away from the underlying desired behavior, particularly when expressive models are used (e.g., energy-based models). As a result, we propose a human-in-the-loop alternative that replaces pointwise supervision with set-valued action targets. We introduce Contrastive policy Learning from Interactive Corrections (CLIC). CLIC leverages human corrections to construct and refine sets of desired actions, and optimizes a policy to place probability mass over these sets rather than over a single action target. This formulation naturally accommodates both absolute and relative corrections and can represent complex multi-modal behaviors. Extensive simulation and real-robot experiments show that the proposed approach leads to effective policy learning across diverse settings: CLIC remains competitive with the state of the art under accurate data while being substantially more robust under noisy, relative, and partial feedback. Our implementation is publicly available at https://clic-webpage.github.io/.