Confidence Calibration for Recommender Systems and Its Applications

TL;DR

This paper introduces a model calibration framework for recommender systems to estimate confidence levels accurately, enabling applications like improved training and dynamic item presentation based on confidence estimates.

Contribution

It proposes a novel calibration framework for recommender systems and demonstrates two practical applications leveraging confidence estimates.

Findings

Effective confidence calibration improves recommendation reliability.

Confidence-guided training enhances small model performance.

Dynamic item presentation based on confidence increases user utility.

Abstract

Despite the importance of having a measure of confidence in recommendation results, it has been surprisingly overlooked in the literature compared to the accuracy of the recommendation. In this dissertation, I propose a model calibration framework for recommender systems for estimating accurate confidence in recommendation results based on the learned ranking scores. Moreover, I subsequently introduce two real-world applications of confidence on recommendations: (1) Training a small student model by treating the confidence of a big teacher model as additional learning guidance, (2) Adjusting the number of presented items based on the expected user utility estimated with calibrated probability.