Using Experts' Opinions in Machine Learning Tasks

TL;DR

This paper introduces a three-step framework to incorporate experts' opinions into machine learning models, demonstrating improved stability and performance in NCAA basketball game predictions compared to previous top solutions.

Contribution

The paper presents a novel framework for integrating expert insights into machine learning, with four models applied to sports prediction, showing enhanced stability and competitive results.

Findings

Models achieved lower log loss (best at 0.489) than previous top solutions.

Models reached top 1%, 10%, and 1% in 2017, 2018, and 2019 Kaggle competitions.

Proposed approach yields more consistent and reliable predictions.

Abstract

In machine learning tasks, especially in the tasks of prediction, scientists tend to rely solely on available historical data and disregard unproven insights, such as experts' opinions, polls, and betting odds. In this paper, we propose a general three-step framework for utilizing experts' insights in machine learning tasks and build four concrete models for a sports game prediction case study. For the case study, we have chosen the task of predicting NCAA Men's Basketball games, which has been the focus of a group of Kaggle competitions in recent years. Results highly suggest that the good performance and high scores of the past models are a result of chance, and not because of a good-performing and stable model. Furthermore, our proposed models can achieve more steady results with lower log loss average (best at 0.489) compared to the top solutions of the 2019 competition (>0.503), and reach the top 1%, 10% and 1% in the 2017, 2018 and 2019 leaderboards, respectively.