Classification Under Human Assistance

TL;DR

This paper develops classifiers optimized for varying levels of automation, demonstrating that such models can outperform fully automated systems and humans in medical diagnosis tasks, supported by theoretical and empirical analysis.

Contribution

It introduces a novel approach to designing classifiers for different automation levels, including a new theoretical formulation and approximation algorithms for convex margin-based classifiers.

Findings

Classifiers under human assistance outperform full automation models.

The problem is NP-hard but can be approximated using greedy algorithms.

Experiments show improved performance in medical diagnosis applications.

Abstract

Most supervised learning models are trained for full automation. However, their predictions are sometimes worse than those by human experts on some specific instances. Motivated by this empirical observation, our goal is to design classifiers that are optimized to operate under different automation levels. More specifically, we focus on convex margin-based classifiers and first show that the problem is NP-hard. Then, we further show that, for support vector machines, the corresponding objective function can be expressed as the difference of two functions f = g - c, where g is monotone, non-negative and {\gamma}-weakly submodular, and c is non-negative and modular. This representation allows a recently introduced deterministic greedy algorithm, as well as a more efficient randomized variant of the algorithm, to enjoy approximation guarantees at solving the problem. Experiments on synthetic and real-world data from several applications in medical diagnosis illustrate our theoretical findings and demonstrate that, under human assistance, supervised learning models trained to operate under different automation levels can outperform those trained for full automation as well as humans operating alone.