Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors

TL;DR

This paper uncovers that in interactive reinforcement learning, agents prefer conservative, low-reward teachers over higher-reward ones, highlighting a bias towards safety and consistency that impacts learning dynamics and decision-making.

Contribution

The study reveals a strong conservative bias in agent teacher selection in IRL, supported by extensive experiments, and discusses implications for human-robot collaboration and safety-critical applications.

Findings

Agents prefer low-reward teachers 93.16% of the time.

Critical thresholds at teacher availability rho >= 0.6 and accuracy omega >= 0.6.

Framework improves performance by 159% over baseline Q-learning.

Abstract

Interactive reinforcement learning (IRL) has shown promise in enabling autonomous agents and robots to learn complex behaviours from human teachers, yet the dynamics of teacher selection remain poorly understood. This paper reveals an unexpected phenomenon in IRL: when given a choice between teachers with different reward structures, learning agents overwhelmingly prefer conservative, low-reward teachers (93.16% selection rate) over those offering 20x higher rewards. Through 1,250 experimental runs in navigation tasks with multiple expert teachers, we discovered: (1) Conservative bias dominates teacher selection: agents systematically choose the lowest-reward teacher, prioritising consistency over optimality; (2) Critical performance thresholds exist at teacher availability rho >= 0.6 and accuracy omega >= 0.6, below which the framework fails catastrophically; (3) The framework achieves 159% improvement over baseline Q-learning under concept drift. These findings challenge fundamental assumptions about optimal teaching in RL and suggest potential implications for human-robot collaboration, where human preferences for safety and consistency may align with the observed agent selection behaviour, potentially informing training paradigms for safety-critical robotic applications.