Tiny Reinforcement Learning for Quadruped Locomotion using Decision Transformers

TL;DR

This paper introduces a method to adapt imitation learning for resource-constrained quadruped robots by using decision transformers, model compression, and simulation testing, enabling efficient deployment without significant performance loss.

Contribution

It proposes a novel approach to make imitation learning deployable on low-resource robotic platforms through model compression and decision transformers, demonstrated in simulation.

Findings

Quantization and pruning reduce model size by 30%

The method achieves natural gaits in simulation

Performance remains competitive after compression

Abstract

Resource-constrained robotic platforms are particularly useful for tasks that require low-cost hardware alternatives due to the risk of losing the robot, like in search-and-rescue applications, or the need for a large number of devices, like in swarm robotics. For this reason, it is crucial to find mechanisms for adapting reinforcement learning techniques to the constraints imposed by lower computational power and smaller memory capacities of these ultra low-cost robotic platforms. We try to address this need by proposing a method for making imitation learning deployable onto resource-constrained robotic platforms. Here we cast the imitation learning problem as a conditional sequence modeling task and we train a decision transformer using expert demonstrations augmented with a custom reward. Then, we compress the resulting generative model using software optimization schemes, including quantization and pruning. We test our method in simulation using Isaac Gym, a realistic physics simulation environment designed for reinforcement learning. We empirically demonstrate that our method achieves natural looking gaits for Bittle, a resource-constrained quadruped robot. We also run multiple simulations to show the effects of pruning and quantization on the performance of the model. Our results show that quantization (down to 4 bits) and pruning reduce model size by around 30\% while maintaining a competitive reward, making the model deployable in a resource-constrained system.