Learning To Defer To A Population With Limited Demonstrations

TL;DR

This paper presents a semi-supervised, meta-learning framework for learning to defer to experts with limited demonstrations, enabling scalable and adaptive human-AI collaboration.

Contribution

It introduces a novel context-aware meta-learning approach that generates expert-specific embeddings from few demonstrations for improved data efficiency and on-the-fly adaptation.

Findings

Models trained on synthetic pseudo-labels achieve near-oracle performance.

The approach significantly reduces data requirements for learning to defer.

Experimental results validate rapid adaptation and scalability.

Abstract

This paper addresses the critical data scarcity that hinders the practical deployment of learning to defer (L2D) systems to the population. We introduce a context-aware, semi-supervised framework that uses meta-learning to generate expert-specific embeddings from only a few demonstrations. We demonstrate the efficacy of a dual-purpose mechanism, where these embeddings are used first to generate a large corpus of pseudo-labels for training, and subsequently to enable on-the-fly adaptation to new experts at test-time. The experiment results on three different datasets confirm that a model trained on these synthetic labels rapidly approaches oracle-level performance, validating the data efficiency of our approach. By resolving a key training bottleneck, this work makes adaptive L2D systems more practical and scalable, paving the way for human-AI collaboration in real-world environments. To facilitate reproducibility and address implementation details not covered in the main text, we provide our source code and training configurations at https://github.com/nil123532/learning-to-defer-to-a-population-with-limited-demonstrations.