Competitive Machine Learning: Best Theoretical Prediction vs Optimization

TL;DR

This paper explores how in competitive machine learning scenarios, models that are not the best theoretically can outperform optimal predictions by strategic modifications, highlighting the importance of practical optimization over pure prediction accuracy.

Contribution

It introduces a game-theoretic model showing that inferior models can gain a competitive edge through strategic adjustments, challenging the assumption that the best predictive model always wins.

Findings

Inferior models can outperform optimal predictions through strategic modifications.

Practical optimization can be more beneficial than static prediction improvements.

Game-theoretic analysis reveals benefits of model adaptation in competitive settings.

Abstract

Machine learning is often used in competitive scenarios: Participants learn and fit static models, and those models compete in a shared platform. The common assumption is that in order to win a competition one has to have the best predictive model, i.e., the model with the smallest out-sample error. Is that necessarily true? Does the best theoretical predictive model for a target always yield the best reward in a competition? If not, can one take the best model and purposefully change it into a theoretically inferior model which in practice results in a higher competitive edge? How does that modification look like? And finally, if all participants modify their prediction models towards the best practical performance, who benefits the most? players with inferior models, or those with theoretical superiority? The main theme of this paper is to raise these important questions and propose a theoretical model to answer them. We consider a study case where two linear predictive models compete over a shared target. The model with the closest estimate gets the whole reward, which is equal to the absolute value of the target. We characterize the reward function of each model, and using a basic game theoretic approach, demonstrate that the inferior competitor can significantly improve his performance by choosing optimal model coefficients that are different from the best theoretical prediction. This is a preliminary study that emphasizes the fact that in many applications where predictive machine learning is at the service of competition, much can be gained from practical (back-testing) optimization of the model compared to static prediction improvement.