Making Reliable and Flexible Decisions in Long-tailed Classification

TL;DR

This paper introduces RF-DLC, a novel framework for reliable and flexible decision-making in long-tailed classification, addressing the risk of critical errors and adapting to task-specific utility matrices.

Contribution

RF-DLC leverages Bayesian Decision Theory and variational optimization to improve reliability and flexibility in long-tailed classification tasks.

Findings

Demonstrates improved reliability in real-world long-tailed tasks.

Introduces False Head Rate metric for tail-sensitivity risk.

Shows adaptability to diverse utility matrices.

Abstract

Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method.