The Sample Complexity of Teaching-by-Reinforcement on Q-Learning

TL;DR

This paper analyzes the minimum number of samples needed for effective teaching in reinforcement learning using Q-learning, introducing optimal teaching algorithms and connecting to existing sample complexity results.

Contribution

It characterizes the teaching dimension for Q-learning under various teacher control levels and develops matching optimal teaching algorithms.

Findings

Derived the teaching dimension for Q-learning with different teachers.

Presented optimal teaching algorithms matching the theoretical bounds.

Connected teaching sample complexity to standard PAC RL results.

Abstract

We study the sample complexity of teaching, termed as "teaching dimension" (TDim) in the literature, for the teaching-by-reinforcement paradigm, where the teacher guides the student through rewards. This is distinct from the teaching-by-demonstration paradigm motivated by robotics applications, where the teacher teaches by providing demonstrations of state/action trajectories. The teaching-by-reinforcement paradigm applies to a wider range of real-world settings where a demonstration is inconvenient, but has not been studied systematically. In this paper, we focus on a specific family of reinforcement learning algorithms, Q-learning, and characterize the TDim under different teachers with varying control power over the environment, and present matching optimal teaching algorithms. Our TDim results provide the minimum number of samples needed for reinforcement learning, and we discuss their connections to standard PAC-style RL sample complexity and teaching-by-demonstration sample complexity results. Our teaching algorithms have the potential to speed up RL agent learning in applications where a helpful teacher is available.