Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery

TL;DR

This paper introduces a nonparametric additive model for reinforcement learning value functions that enhances interpretability, enabling personalized healthcare recommendations while capturing nonlinear feature contributions without relying on black-box models.

Contribution

The authors develop a novel nonparametric additive approach combining kernel regression and basis expansion for interpretable value function estimation in reinforcement learning.

Findings

Effective in simulation studies

Successfully applied to spine disease recovery

Generated clinically aligned recommendations

Abstract

We propose a nonparametric additive model for estimating interpretable value functions in reinforcement learning, with an application in optimizing postoperative recovery through personalized, adaptive recommendations. While reinforcement learning has achieved significant success in various domains, recent methods often rely on black-box approaches such as neural networks, which hinder the examination of individual feature contributions to a decision-making policy. Our novel method offers a flexible technique for estimating action-value functions without explicit parametric assumptions, overcoming the limitations of the linearity assumption of classical algorithms. By incorporating local kernel regression and basis expansion, we obtain a sparse, additive representation of the action-value function, enabling local approximation and retrieval of nonlinear, independent contributions of select state features and the interactions between joint feature pairs. We validate our approach through a simulation study and apply it to spine disease recovery, uncovering recommendations aligned with clinical knowledge. This method bridges the gap between flexible machine learning techniques and the interpretability required in healthcare applications, paving the way for more personalized interventions.