Tensor-Efficient High-Dimensional Q-learning

TL;DR

Tensor-Efficient Q-Learning (TEQL) leverages low-rank tensor structures in high-dimensional RL to improve sample efficiency and exploration, outperforming traditional methods in resource-constrained settings.

Contribution

The paper introduces TEQL, a novel tensor-based Q-learning algorithm that explicitly exploits low-rank structure for efficient exploration and learning in high-dimensional spaces.

Findings

TEQL outperforms matrix-based low-rank and deep RL methods in sample efficiency.

TEQL effectively exploits low-rank tensor structure for better exploration.

Experiments on control tasks validate TEQL's efficiency under limited sampling budgets.

Abstract

High-dimensional reinforcement learning(RL) faces challenges with complex calculations and low sample efficiency in large state-action spaces. Q-learning algorithms struggle particularly with the curse of dimensionality, where the number of state-action pairs grows exponentially with problem size. While neural network-based approaches like Deep Q-Networks have shown success, they do not explicitly exploit problem structure. Many high-dimensional control tasks exhibit low-rank structure in their value functions, and tensor-based methods using low-rank decomposition offer parameter-efficient representations. However, existing tensor-based Q-learning methods focus on representation fidelity without leveraging this structure for exploration. We propose Tensor-Efficient Q-Learning (TEQL), which represents the Q-function as a low-rank CP tensor over discretized state-action spaces and exploits the tensor structure for uncertainty-aware exploration. TEQL incorporates Error-Uncertainty Guided Exploration (EUGE), which combines tensor approximation error with visit counts to guide action selection, along with frequency-aware regularization to stabilize updates. Under matched parameter budgets, experiments on classic control tasks demonstrate that TEQL outperforms both matrix-based low-rank methods and deep RL baselines in sample efficiency, making it suitable for resource-constrained applications where sampling costs are high.