Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning

TL;DR

This paper introduces strategic link scores to quantify dependencies between decisions in long-term planning, enhancing understanding and performance in reinforcement learning, decision support, and traffic simulation.

Contribution

It proposes a novel metric, strategic link scores, to analyze decision dependencies, with applications in explaining RL agents, improving decision support, and characterizing planning horizons.

Findings

Strategic link scores reveal decision dependencies in RL and real-world scenarios.

Applying these scores improves explanation and performance of decision-making systems.

Analysis of traffic routing demonstrates the method's ability to characterize planning horizons.

Abstract

Long-term planning, as in reinforcement learning (RL), involves finding strategies: actions that collectively work toward a goal rather than individually optimizing their immediate outcomes. As part of a strategy, some actions are taken at the expense of short-term benefit to enable future actions with even greater returns. These actions are only advantageous if followed up by the actions they facilitate, consequently, they would not have been taken if those follow-ups were not available. In this paper, we quantify such dependencies between planned actions with strategic link scores: the drop in the likelihood of one decision under the constraint that a follow-up decision is no longer available. We demonstrate the utility of strategic link scores through three practical applications: (i) explaining black-box RL agents by identifying strategically linked pairs among decisions they make, (ii) improving the worst-case performance of decision support systems by distinguishing whether recommended actions can be adopted as standalone improvements or whether they are strategically linked hence requiring a commitment to a broader strategy to be effective, and (iii) characterizing the planning processes of non-RL agents purely through interventions aimed at measuring strategic link scores - as an example, we consider a realistic traffic simulator and analyze through road closures the effective planning horizon of the emergent routing behavior of many drivers.