On Calibrated Predictions for Auction Selection Mechanisms

TL;DR

This paper examines the challenges of achieving calibration in CTR predictions used in auction mechanisms, highlighting conditions where calibration and efficiency maximization are compatible or impossible.

Contribution

It identifies fundamental limitations of calibration in auction-based prediction systems and provides conditions for when calibration and efficiency can be simultaneously achieved.

Findings

Certain calibration notions are impossible to attain depending on auction details.

Maximizing auction efficiency with calibrated predictions can be mutually exclusive.

Calibration is achievable when bids and queries do not contain additional CTR information.

Abstract

Calibration is a basic property for prediction systems, and algorithms for achieving it are well-studied in both statistics and machine learning. In many applications, however, the predictions are used to make decisions that select which observations are made. This makes calibration difficult, as adjusting predictions to achieve calibration changes future data. We focus on click-through-rate (CTR) prediction for search ad auctions. Here, CTR predictions are used by an auction that determines which ads are shown, and we want to maximize the value generated by the auction. We show that certain natural notions of calibration can be impossible to achieve, depending on the details of the auction. We also show that it can be impossible to maximize auction efficiency while using calibrated predictions. Finally, we give conditions under which calibration is achievable and simultaneously maximizes auction efficiency: roughly speaking, bids and queries must not contain information about CTRs that is not already captured by the predictions.