When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL

TL;DR

This paper investigates hyperparameter sensitivity in offline goal-conditioned RL, finding that robustness varies by algorithm and data quality, with insights into gradient interference affecting stability and sensitivity.

Contribution

It reveals that hyperparameter sensitivity in offline RL is less severe than online RL and links this to gradient interference, guiding more robust algorithm design.

Findings

Offline RL shows greater hyperparameter robustness than online RL.

QRL maintains stability with modest expert data, unlike HIQL.

Gradient interference correlates with hyperparameter sensitivity.

Abstract

Hyperparameter sensitivity in Deep Reinforcement Learning (RL) is often accepted as unavoidable. However, it remains unclear whether it is intrinsic to the RL problem or exacerbated by specific training mechanisms. We investigate this question in offline goal-conditioned RL, where data distributions are fixed, and non-stationarity can be explicitly controlled via scheduled shifts in data quality. Additionally, we study varying data qualities under both stationary and non-stationary regimes, and cover two representative algorithms: HIQL (bootstrapped TD-learning) and QRL (quasimetric representation learning). Overall, we observe substantially greater robustness to changes in hyperparameter configurations than commonly reported for online RL, even under controlled non-stationarity. Once modest expert data is present ($\approx$ 20\%), QRL maintains broad, stable near-optimal regions, while HIQL exhibits sharp optima that drift significantly across training phases. To explain this divergence, we introduce an inter-goal gradient alignment diagnostic. We find that bootstrapped objectives exhibit stronger destructive gradient interference, which coincides directly with hyperparameter sensitivity. These results suggest that high sensitivity to changes in hyperparameter configurations during training is not inevitable in RL, but is amplified by the dynamics of bootstrapping, offering a pathway toward more robust algorithmic objective design.