Reply to Cecchi and Palminteri: On the need to model temporal variation in learning rates

Abstract

Figures

Full text

We appreciate Cecchi and Palminteri’s [CP, (1)] interest in our work (2). We begin by clarifying that (2) does not claim that human behavior is best described by models with decreasing learning rates. Indeed, we explicitly document ways in which human behavior deviates from Bayesian predictions (SI Appendix, section 4). Our central point is methodological: temporal variation in learning rates can mimic confirmation bias. Therefore, failing to account for temporally varying learning rates before introducing valence-based asymmetries risks overestimating the magnitude of bias.

CP challenge our comparison of Bayesian and biased RL models (Fig. 4B), but two methodological choices explain the discrepancy:

1. CP fit a model designed for stationary environments (with learning rate $[eqn]$ ) to both stationary and nonstationary environments.
2. CP fit one set of parameters to all episodes of a subject.

Regarding (1), applying a stationary Bayesian update rule to reversal blocks is inappropriate: the model is misspecified by design, so poor fits are expected. Our analysis intentionally restricts itself to stationary environments to isolate the confound between temporal learning-rate variation and confirmation bias. If nonstationary environments are included, Bayesian models with assumed volatility—which also imply time-varying learning rates—provide better fits than asymmetric RL models (3).

Regarding (2), there is no consensus on whether reinforcement-learning parameters should be fit jointly across all episodes of a subject or separately for each episode. Several studies show that RL parameter estimates often exhibit low test–retest reliability (4, 5), and human behavior can vary substantially from block to block due to factors such as attention or fatigue. In our data, parameter reliability across symmetric, asymmetric, and reversal environments is low (Table 1), and cross-validation reveals poor generalization of parameters across environment types (Fig. 1). In light of this variability, we chose to fit parameters separately for each episode, while recognizing that alternative modeling choices are possible.

Finally, CP introduce hybrid models with geometrically decaying, valence-dependent learning rates as evidence that temporal dynamics alone cannot account for behavior. However, these models impose a particular parametric form on learning-rate dynamics. Humans are known to infer environmental volatility, which implies flexible, often nonmonotonic changes in learning rates near change points. Given the wide space of possible temporal dynamics, rejecting one specific form does not negate the broader point: temporal variation must be modeled before attributing behavior to valence-based bias.

All analyses reported here, including parameter reliability and cross-validation tests, are available at: https://github.com/prakhargodara/Bandit-parameters-ICC-and-cross-validation-scores.