Adaptive Contracts for Cost-Effective AI Delegation

TL;DR

This paper introduces adaptive contracts for AI delegation that optimize evaluation efforts by selectively performing detailed assessments, leading to cost savings and improved efficiency in pay-for-performance scenarios.

Contribution

It develops efficient algorithms for optimal adaptive contracts, explores randomized models, and empirically demonstrates their advantages over non-adaptive methods.

Findings

Adaptive contracts reduce evaluation costs.

Algorithms perform well under natural assumptions.

Empirical results show improved cost-effectiveness.

Abstract

When organizations delegate text generation tasks to AI providers via pay-for-performance contracts, expected payments rise when evaluation is noisy. As evaluation methods become more elaborate, the economic benefits of decreased noise are often overshadowed by increased evaluation costs. In this work, we introduce adaptive contracts for AI delegation, which allow detailed evaluation to be performed selectively after observing an initial coarse signal in order to conserve resources. We make three sets of contributions: First, we provide efficient algorithms for computing optimal adaptive contracts under natural assumptions or when core problem dimensions are small, and prove hardness of approximation in the general unstructured case. We then formulate alternative models of randomized adaptive contracts and discuss their benefits and limitations. Finally, we empirically demonstrate the benefits of adaptivity over non-adaptive baselines using question-answering and code-generation datasets.