See and Copy: Generation of complex compositional movements from modular and geometric RNN representations

TL;DR

This paper introduces a modular RNN-based model for motor control that can generalize to complex, unseen movement sequences and adapt rapidly to perturbations, inspired by primate imitation abilities.

Contribution

It presents a novel modular control architecture with separate encoding, motor, and scheduling components, demonstrating improved generalization and biological plausibility.

Findings

Generalizes to longer and more complex trajectories than trained on

Rapidly adapts to perturbations in movement

Recapitulates experimental motor cortex activity patterns

Abstract

A hallmark of biological intelligence and control is combinatorial generalization: animals are able to learn various things, then piece them together in new combinations to produce appropriate outputs for new tasks. Inspired by the ability of primates to readily imitate seen movement sequences, we present a model of motor control using a realistic model of arm dynamics, tasked with imitating a guide that makes arbitrary two-segment drawings. We hypothesize that modular organization is one of the keys to such flexible and generalizable control. We construct a modular control model consisting of separate encoding and motor RNNs and a scheduler, which we train end-to-end on the task. We show that the modular structure allows the model to generalize not only to unseen two-segment trajectories, but to new drawings consisting of many more segments than it was trained on, and also allows for rapid adaptation to perturbations. Finally, our model recapitulates experimental observations of the preparatory and execution-related processes unfolding during motor control, providing a normative explanation for functional segregation of preparatory and execution-related activity within the motor cortex.