Tune to Learn: How Controller Gains Shape Robot Policy Learning

TL;DR

This paper investigates how position controller gains influence robot policy learning, revealing that optimal gains depend on the learning method rather than task-specific compliance or stiffness.

Contribution

It systematically analyzes the impact of controller gains on behavior cloning, reinforcement learning, and sim-to-real transfer, offering guidance for gain selection based on learning paradigms.

Findings

Behavior cloning benefits from compliant and overdamped gains.

Reinforcement learning succeeds across all gain regimes with proper hyperparameter tuning.

Stiff and overdamped gains impair sim-to-real transfer.

Abstract

Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn