SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks

TL;DR

SpawnNet introduces a two-stream architecture that fuses pre-trained visual features to enhance the generalization of visuomotor policies across categories, outperforming prior methods in imitation learning tasks.

Contribution

The paper proposes SpawnNet, a novel architecture that improves categorical generalization of policies by learning to fuse multi-layer pre-trained visual representations.

Findings

Significantly better categorical generalization in simulated experiments.

Effective transfer to real-world scenarios.

Outperforms prior approaches in imitation learning.

Abstract

The existing internet-scale image and video datasets cover a wide range of everyday objects and tasks, bringing the potential of learning policies that generalize in diverse scenarios. Prior works have explored visual pre-training with different self-supervised objectives. Still, the generalization capabilities of the learned policies and the advantages over well-tuned baselines remain unclear from prior studies. In this work, we present a focused study of the generalization capabilities of the pre-trained visual representations at the categorical level. We identify the key bottleneck in using a frozen pre-trained visual backbone for policy learning and then propose SpawnNet, a novel two-stream architecture that learns to fuse pre-trained multi-layer representations into a separate network to learn a robust policy. Through extensive simulated and real experiments, we show significantly better categorical generalization compared to prior approaches in imitation learning settings. Open-sourced code and videos can be found on our website: https://xingyu-lin.github.io/spawnnet.