Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RL

TL;DR

This paper investigates how independently reducing the size of actor networks in actor-critic reinforcement learning can significantly cut computational costs while maintaining performance, especially useful in resource-limited settings.

Contribution

It demonstrates that smaller actor networks can match larger ones' performance, challenging the common assumption of symmetric architectures in actor-critic models.

Findings

Up to 99% reduction in network weights for actors.

Average 77% weight reduction across multiple algorithms.

Smaller actors maintain comparable policy performance.

Abstract

Actors and critics in actor-critic reinforcement learning algorithms are functionally separate, yet they often use the same network architectures. This case study explores the performance impact of network sizes when considering actor and critic architectures independently. By relaxing the assumption of architectural symmetry, it is often possible for smaller actors to achieve comparable policy performance to their symmetric counterparts. Our experiments show up to 99% reduction in the number of network weights with an average reduction of 77% over multiple actor-critic algorithms on 9 independent tasks. Given that reducing actor complexity results in a direct reduction of run-time inference cost, we believe configurations of actors and critics are aspects of actor-critic design that deserve to be considered independently, particularly in resource-constrained applications or when deploying multiple actors simultaneously.