Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning

TL;DR

This paper introduces SoCo-DT, a novel soft conflict-resolution method for offline multi-task reinforcement learning that dynamically adjusts parameter importance and sparsity to mitigate gradient conflicts and enhance learning efficiency.

Contribution

The paper proposes a dynamic, importance-based masking and sparsity adjustment strategy for multi-task RL, improving over fixed masks and enhancing knowledge sharing across tasks.

Findings

SoCo-DT outperforms state-of-the-art methods by 7.6% on MT50.

It achieves a 10.5% improvement on the suboptimal dataset.

Experimental results validate the effectiveness of dynamic conflict mitigation.

Abstract

Multi-task reinforcement learning (MTRL) seeks to learn a unified policy for diverse tasks, but often suffers from gradient conflicts across tasks. Existing masking-based methods attempt to mitigate such conflicts by assigning task-specific parameter masks. However, our empirical study shows that coarse-grained binary masks have the problem of over-suppressing key conflicting parameters, hindering knowledge sharing across tasks. Moreover, different tasks exhibit varying conflict levels, yet existing methods use a one-size-fits-all fixed sparsity strategy to keep training stability and performance, which proves inadequate. These limitations hinder the model's generalization and learning efficiency. To address these issues, we propose SoCo-DT, a Soft Conflict-resolution method based by parameter importance. By leveraging Fisher information, mask values are dynamically adjusted to retain important parameters while suppressing conflicting ones. In addition, we introduce a dynamic sparsity adjustment strategy based on the Interquartile Range (IQR), which constructs task-specific thresholding schemes using the distribution of conflict and harmony scores during training. To enable adaptive sparsity evolution throughout training, we further incorporate an asymmetric cosine annealing schedule to continuously update the threshold. Experimental results on the Meta-World benchmark show that SoCo-DT outperforms the state-of-the-art method by 7.6% on MT50 and by 10.5% on the suboptimal dataset, demonstrating its effectiveness in mitigating gradient conflicts and improving overall multi-task performance.