X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning

TL;DR

X-Distill introduces a cross-architecture knowledge distillation method that transfers rich visual features from large Vision Transformers to compact CNNs, significantly improving data-efficient visuomotor learning in robotics.

Contribution

The paper presents a novel offline distillation approach that combines large ViT models with compact CNNs for improved robotic manipulation performance.

Findings

Outperforms from-scratch ResNet policies on 34 benchmarks

Surpasses fine-tuned DINOv2 encoders in real-world tasks

Achieves state-of-the-art data efficiency in robotic manipulation

Abstract

Visuomotor policies often leverage large pre-trained Vision Transformers (ViTs) for their powerful generalization capabilities. However, their significant data requirements present a major challenge in the data-scarce context of most robotic learning settings, where compact CNNs with strong inductive biases can be more easily optimized. To address this trade-off, we introduce X-Distill, a simple yet highly effective method that synergizes the strengths of both architectures. Our approach involves an offline, cross-architecture knowledge distillation, transferring the rich visual representations of a large, frozen DINOv2 teacher to a compact ResNet-18 student on the general-purpose ImageNet dataset. This distilled encoder, now endowed with powerful visual priors, is then jointly fine-tuned with a diffusion policy head on the target manipulation tasks. Extensive experiments on $34$ simulated benchmarks and $5$ challenging real-world tasks demonstrate that our method consistently outperforms policies equipped with from-scratch ResNet or fine-tuned DINOv2 encoders. Notably, X-Distill also surpasses 3D encoders that utilize privileged point cloud observations or much larger Vision-Language Models. Our work highlights the efficacy of a simple, well-founded distillation strategy for achieving state-of-the-art performance in data-efficient robotic manipulation.