Argumentative Reward Learning: Reasoning About Human Preferences

TL;DR

This paper introduces argumentative reward learning, a neuro-symbolic framework that enhances reinforcement learning from human feedback by better generalizing preferences and reducing user burden.

Contribution

It presents a novel neuro-symbolic approach that combines preference-based argumentation with reinforcement learning, improving robustness and generalization of reward models.

Findings

Enhanced generalization of human preferences

Reduced user burden in feedback collection

Improved robustness of reward models

Abstract

We define a novel neuro-symbolic framework, argumentative reward learning, which combines preference-based argumentation with existing approaches to reinforcement learning from human feedback. Our method improves prior work by generalising human preferences, reducing the burden on the user and increasing the robustness of the reward model. We demonstrate this with a number of experiments.