Off-Policy Evaluation from Logged Human Feedback

TL;DR

This paper introduces methods for evaluating AI models using previously collected human feedback without additional data collection, enabling efficient assessment and optimization of new models.

Contribution

It formalizes off-policy evaluation from logged human feedback and proposes both model-based and model-free estimators for policy value assessment.

Findings

Estimators can accurately predict policy values

Estimators can rank different policies effectively

Methods are suitable for optimizing policies based on logged feedback

Abstract

Learning from human feedback has been central to recent advances in artificial intelligence and machine learning. Since the collection of human feedback is costly, a natural question to ask is if the new feedback always needs to collected. Or could we evaluate a new model with the human feedback on responses of another model? This motivates us to study off-policy evaluation from logged human feedback. We formalize the problem, propose both model-based and model-free estimators for policy values, and show how to optimize them. We analyze unbiasedness of our estimators and evaluate them empirically. Our estimators can predict the absolute values of evaluated policies, rank them, and be optimized.