MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand,, Igor Gilitschenski
TL;DR
MAD-TD introduces a model-augmented data approach to stabilize high update ratios in deep RL, improving training stability and performance without neural network resets.
Contribution
The paper proposes MAD-TD, a novel method that uses learned world model data to stabilize high update-to-data ratios in RL training, reducing instability and eliminating the need for parameter resets.
Findings
MAD-TD achieves competitive performance on DeepMind control tasks.
The method effectively combats value overestimation.
MAD-TD enhances training stability in high update ratio regimes.
Abstract
Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradient steps for every new sample. While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process. Previous approaches need to rely on periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval. In this paper, we focus on one of the core difficulties of stable training with limited samples: the inability of learned value functions to generalize to unobserved on-policy actions. We mitigate this issue directly by augmenting the off-policy RL training…
Peer Reviews
Decision·ICLR 2025 Spotlight
1. The paper tackles an interesting and critical problem setting in sample-efficient reinforcement learning (RL), which has significant implications for many real-world applications where data collection is costly or time-consuming. 2. The authors provide both empirical evidence and theoretical analysis to demonstrate the importance of addressing incorrect Q-value learning in off-policy RL with limited samples, making a compelling case for their proposed solution. 3. The proposed method, MAD-TD,
1. Limited Baseline Comparison: The paper only compares the proposed method, MAD-TD, against two baselines (BRO and TD-MPC) and their variants, which may not be sufficient to demonstrate its performance comprehensively. A more extensive comparison with other state-of-the-art methods would strengthen the paper's claims. 2. Lack of Ablation Study: Although the authors outline critical design choices in Section 4.1, they do not conduct an ablation study to investigate the importance of these choice
I found this an interesting paper and the authors approach seems to generate useful results. The paper is well written and the presentation allows the reader to understand the rationale behind most of the work. The paper combines some mathematical and intuitive insight into the stability problem. It then uses this insight as motivation for their new approach MAD-TD which seems to show some success based on the experimental results. I thought the authors gave quite a balanced presentation of t
The main limitation of the paper is precisely that which the authors point out themeselves i.e. that the assumption that a sufficiently high fidelity of the model can be learned online is valid. This is necessary as the "augmented data" is generated from this model. However, the authors have been quite up-front with this and, despite this short-coming, their results seem to show success with their MAD-TD approach. Another shortfall of the techniques is that of course, the main "proof" of the r
1. Authors propose a method that works effectively with high UTD without resets. 2. The topic is relevant. 3. The paper is well-written.
1. The results look pretty good, but then there is a question about robustness. Authors sometimes refer to [1], but the same paper points out that different effects can be obtained in RL in different environments. I agree with the theses in the manuscript, but I think it is valuable to validate these strong results on other benchmarks so that scientists in the future will know to what extent this is a general solution. Please see question 1 for more details. 2. Minor: Equations are without numer
Videos
Taxonomy
TopicsDistributed and Parallel Computing Systems
MethodsFocus