When Does Confidence-Based Cascade Deferral Suffice?

TL;DR

This paper investigates the effectiveness of confidence-based cascade deferral strategies, providing a theoretical framework for understanding their limitations and proposing alternative methods that perform better under certain challenging conditions.

Contribution

It offers a theoretical characterization of when confidence-based deferral fails and introduces post-hoc mechanisms that outperform it in specific scenarios.

Findings

Confidence-based deferral can fail with specialist models, label noise, and distribution shift.

Post-hoc deferral mechanisms improve performance in challenging settings.

Theoretical analysis clarifies optimal deferral strategies.

Abstract

Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set.