L2CU: Learning to Complement Unseen Users

TL;DR

L2CU introduces a framework that improves human-AI cooperative classification for unseen users by identifying user profiles and matching new users to these profiles, enhancing accuracy despite sparse and noisy annotations.

Contribution

The paper proposes a novel L2C framework that captures user variability through profiles and matches unseen users to these profiles, addressing generalization challenges.

Findings

L2CU outperforms existing methods on multiple datasets.

It effectively handles sparse and noisy user annotations.

L2CU is model-agnostic and improves cooperative accuracy.

Abstract

Recent research highlights the potential of machine learning models to learn to complement (L2C) human strengths; however, generalizing this capability to unseen users remains a significant challenge. Existing L2C methods oversimplify interaction between human and AI by relying on a single, global user model that neglects individual user variability, leading to suboptimal cooperative performance. Addressing this, we introduce L2CU, a novel L2C framework for human-AI cooperative classification with unseen users. Given sparse and noisy user annotations, L2CU identifies representative annotator profiles capturing distinct labeling patterns. By matching unseen users to these profiles, L2CU leverages profile-specific models to complement the user and achieve superior joint accuracy. We evaluate L2CU on datasets (CIFAR-10N, CIFAR-10H, Fashion-MNIST-H, Chaoyang and AgNews), demonstrating its effectiveness as a model-agnostic solution for improving human-AI cooperative classification.