Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning

TL;DR

This paper introduces a coreset-based task selection method for meta-reinforcement learning that improves sample efficiency by selecting diverse and informative tasks, leading to faster adaptation and reduced sample complexity.

Contribution

It proposes a novel task selection approach based on gradient diversity, with theoretical guarantees and empirical validation demonstrating improved sample efficiency in meta-RL.

Findings

Reduces sample complexity by a factor of O(1/ε)

Guarantees faster adaptation to unseen tasks

Validated benefits across multiple RL benchmarks

Abstract

We study task selection to enhance sample efficiency in model-agnostic meta-reinforcement learning (MAML-RL). Traditional meta-RL typically assumes that all available tasks are equally important, which can lead to task redundancy when they share significant similarities. To address this, we propose a coreset-based task selection approach that selects a weighted subset of tasks based on how diverse they are in gradient space, prioritizing the most informative and diverse tasks. Such task selection reduces the number of samples needed to find an $\epsilon$-close stationary solution by a factor of O(1/$\epsilon$). Consequently, it guarantees a faster adaptation to unseen tasks while focusing training on the most relevant tasks. As a case study, we incorporate task selection to MAML-LQR (Toso et al., 2024b), and prove a sample complexity reduction proportional to O(log(1/$\epsilon$)) when the task specific cost also satisfy gradient dominance. Our theoretical guarantees underscore task selection as a key component for scalable and sample-efficient meta-RL. We numerically validate this trend across multiple RL benchmark problems, illustrating the benefits of task selection beyond the LQR baseline.