The Model Selection Curse

TL;DR

This paper examines how model selection impacts truth-telling incentives in statistical procedures, highlighting potential issues in human interactions with machine learning algorithms.

Contribution

It introduces a simple model illustrating the effects of model selection on truth-telling incentives, with implications for human-algorithm interactions.

Findings

Model selection can distort incentives for truth-telling.

The analysis reveals potential misalignments in strategic reporting.

Implications for designing better human-algorithm interaction protocols.

Abstract

A "statistician" takes an action on behalf of an agent, based on the agent's self-reported personal data and a sample involving other people. The action that he takes is an estimated function of the agent's report. The estimation procedure involves model selection. We ask the following question: Is truth-telling optimal for the agent given the statistician's procedure? We analyze this question in the context of a simple example that highlights the role of model selection. We suggest that our simple exercise may have implications for the broader issue of human interaction with "machine learning" algorithms.