A Controlled Study of Double DQN and Dueling DQN Under Cross-Environment Transfer

TL;DR

This study empirically compares Double DQN and Dueling DQN in transfer learning across different environments, revealing that DDQN is more robust to negative transfer than Dueling DQN.

Contribution

It provides the first controlled empirical analysis of how architectural differences affect transfer performance in deep reinforcement learning.

Findings

DDQN avoids negative transfer in the tested setup

Dueling DQN exhibits degraded rewards and instability

Statistical analysis confirms significant performance differences

Abstract

Transfer learning in deep reinforcement learning is often motivated by improved stability and reduced training cost, but it can also fail under substantial domain shift. This paper presents a controlled empirical study examining how architectural differences between Double Deep Q-Networks (DDQN) and Dueling DQN influence transfer behavior across environments. Using CartPole as a source task and LunarLander as a structurally distinct target task, we evaluate a fixed layer-wise representation transfer protocol under identical hyperparameters and training conditions, with baseline agents trained from scratch used to contextualize transfer effects. Empirical results show that DDQN consistently avoids negative transfer under the examined setup and maintains learning dynamics comparable to baseline performance in the target environment. In contrast, Dueling DQN consistently exhibits negative transfer under identical conditions, characterized by degraded rewards and unstable optimization behavior. Statistical analysis across multiple random seeds confirms a significant performance gap under transfer. These findings suggest that architectural inductive bias is strongly associated with robustness to cross-environment transfer in value-based deep reinforcement learning under the examined transfer protocol.