Optimistic Temporal Difference Learning for 2048

TL;DR

This paper introduces an optimistic initialization approach to enhance exploration in TD learning for 2048, significantly improving performance and reducing network size needed for high scores.

Contribution

It proposes using optimistic initialization in TD and TC learning for 2048, leading to better exploration, higher scores, and smaller networks compared to prior methods.

Findings

Significant performance improvement with optimistic initialization

Reduced network size for achieving high scores

State-of-the-art results with combined techniques

Abstract

Temporal difference (TD) learning and its variants, such as multistage TD (MS-TD) learning and temporal coherence (TC) learning, have been successfully applied to 2048. These methods rely on the stochasticity of the environment of 2048 for exploration. In this paper, we propose to employ optimistic initialization (OI) to encourage exploration for 2048, and empirically show that the learning quality is significantly improved. This approach optimistically initializes the feature weights to very large values. Since weights tend to be reduced once the states are visited, agents tend to explore those states which are unvisited or visited few times. Our experiments show that both TD and TC learning with OI significantly improve the performance. As a result, the network size required to achieve the same performance is significantly reduced. With additional tunings such as expectimax search, multistage learning, and tile-downgrading technique, our design achieves the state-of-the-art performance, namely an average score of 625 377 and a rate of 72% reaching 32768 tiles. In addition, for sufficiently large tests, 65536 tiles are reached at a rate of 0.02%.