OCCAM: Towards Cost-Efficient and Accuracy-Aware Classification Inference

TL;DR

OCCAM is a method that optimally assigns classifiers to maximize accuracy within cost constraints, significantly reducing inference costs while maintaining performance across various datasets.

Contribution

We introduce OCCAM, a novel approach that computes the optimal classifier portfolio for cost-efficient, accuracy-aware inference using an unbiased accuracy estimator and integer linear programming.

Findings

Achieves 40% cost reduction on real-world datasets

Maintains accuracy with minimal loss despite cost savings

Effectively balances classifier accuracy and inference cost

Abstract

Classification tasks play a fundamental role in various applications, spanning domains such as healthcare, natural language processing and computer vision. With the growing popularity and capacity of machine learning models, people can easily access trained classifiers as a service online or offline. However, model use comes with a cost and classifiers of higher capacity (such as large foundation models) usually incur higher inference costs. To harness the respective strengths of different classifiers, we propose a principled approach, OCCAM, to compute the best classifier assignment strategy over classification queries (termed as the optimal model portfolio) so that the aggregated accuracy is maximized, under user-specified cost budgets. Our approach uses an unbiased and low-variance accuracy estimator and effectively computes the optimal solution by solving an integer linear programming problem. On a variety of real-world datasets, OCCAM achieves 40% cost reduction with little to no accuracy drop.