Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performance

TL;DR

This paper explores using neural signals from fNIRS to assess and guide reinforcement learning agents' performance, demonstrating feasibility and potential for neural feedback-based training systems.

Contribution

It introduces a novel dataset of fNIRS signals linked to agent performance and develops classifiers and regressors to predict performance levels from neural data.

Findings

Achieved 67% F1 score for binary classification of agent performance.

Demonstrated improved prediction accuracy with subject-specific fine-tuning.

Showed feasibility of using neural signals for reinforcement learning feedback.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a methodology that aligns agent behavior with human preferences by integrating user feedback into the agent's training process. This paper introduces a framework that guides agent training through implicit neural signals, with a focus on the neural classification problem. Our work presents and releases a novel dataset of functional near-infrared spectroscopy (fNIRS) recordings collected from 25 human participants across three domains: Pick-and-Place Robot, Lunar Lander, and Flappy Bird. We train multiple classifiers to predict varying levels of agent performance (optimal, suboptimal, or worst-case) from windows of preprocessed fNIRS features, achieving an average F1 score of 67% for binary and 46% for multi-class classification across conditions and domains. We also train multiple regressors to predict the degree of deviation between an agent's chosen action and a set of near-optimal policy actions, providing a continuous measure of performance. Finally, we evaluate cross-subject generalization and show that fine-tuning pre-trained models with a small sample of subject-specific data increases average F1 scores by 17% and 41% for binary and multi-class models, respectively. Our results demonstrate that mapping implicit fNIRS signals to agent performance is feasible and can be improved, laying the foundation for future Reinforcement Learning from Neural Feedback (RLNF) systems.