Discounting and Drug Seeking in Biological Hierarchical Reinforcement Learning

TL;DR

This paper models addiction within a hierarchical reinforcement learning framework, showing how increased temporal discounting amplifies drug-seeking behavior and aligns with empirical findings, offering new insights into addiction as a hierarchical decision-making disorder.

Contribution

It introduces a biologically grounded HRL model that integrates discounting effects, explaining the neural and behavioral mechanisms of addiction in a hierarchical context.

Findings

High discounting increases drug-seeking across the hierarchy.

Model aligns with empirical data on temporal discounting in addiction.

Predicts addiction as a disorder of hierarchical decision-making.

Abstract

Despite a strong desire to quit, individuals with long-term substance use disorder (SUD) often struggle to resist drug use, even when aware of its harmful consequences. This disconnect between knowledge and compulsive behavior reflects a fundamental cognitive-behavioral conflict in addiction. Neurobiologically, differential cue-induced activity within striatal subregions, along with dopamine-mediated connectivity from the ventral to the dorsal striatum, contributes to compulsive drug-seeking. However, the functional mechanism linking these findings to behavioral conflict remains unclear. Another hallmark of addiction is temporal discounting: individuals with drug dependence exhibit steeper discount rates than non-users. Assuming the ventral-dorsal striatal organization reflects a gradient from cognitive to motor representations, addiction can be modeled within a hierarchical reinforcement learning (HRL) framework. However, integrating discounting into biologically grounded HRL remains an open challenge. In this work, we build on a model showing how action choices reinforced with drug rewards become insensitive to the negative consequences that follow. We address the integration of discounting by ensuring natural reward values converge across all levels in the HRL hierarchy, while drug rewards diverge due to their dopaminergic effects. Our results show that high discounting amplifies drug-seeking across the hierarchy, linking faster discounting with increased addiction severity and impulsivity. We demonstrate alignment with empirical findings on temporal discounting and propose testable predictions, establishing addiction as a disorder of hierarchical decision-making.