Agile Reinforcement Learning through Separable Neural Architecture

TL;DR

This paper introduces SPAN, a spline-based neural architecture for reinforcement learning that improves sample efficiency and success rates over traditional MLPs, especially in resource-constrained environments.

Contribution

The work proposes SPAN, a novel separable neural architecture with learnable preprocessing, enhancing parameter and computational efficiency in RL function approximation.

Findings

SPAN achieves 30-50% better sample efficiency.

SPAN outperforms MLPs with 1.3-9x higher success rates.

SPAN shows robustness to hyperparameter variations.

Abstract

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet the go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Recent spline-based separable architectures - such as Kolmogorov-Arnold Networks (KANs) - have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks), a novel function approximation approach to RL. SPAN adapts the low rank KHRONOS framework by integrating a learnable preprocessing layer with a separable tensor product B-spline basis. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, as well as offline settings (Minari/D4RL). Empirical results demonstrate that SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high performance alternative for learning intrinsically efficient policies in resource-limited settings.