# Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

## TL;DR

This paper demonstrates that using Variable Impedance Control in End-effector Space (VICES) as the action space in reinforcement learning enhances sample efficiency, safety, and transferability in contact-rich manipulation tasks.

## Contribution

It introduces VICES as a novel action space for deep RL in contact-rich tasks, showing its advantages over traditional action spaces in multiple manipulation scenarios.

## Key findings

- VICES improves sample efficiency in RL for manipulation tasks.
- VICES maintains low energy consumption and safety.
- RL policies with VICES transfer effectively across robots and from simulation to real world.

## Abstract

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.