Feature Extractor or Decision Maker: Rethinking the Role of Visual Encoders in Visuomotor Policies

TL;DR

This paper investigates the role of visual encoders in visuomotor policies, revealing that end-to-end trained models utilize encoders for decision-making, unlike pre-trained out-of-domain models, which impacts performance.

Contribution

It introduces Visual Alignment Testing to evaluate the functional role of visual encoders and shows that end-to-end training enables encoders to actively contribute to decisions.

Findings

E2E-trained models' encoders contribute to decision-making.

Pretrained OOD encoders lack decision-making capability.

Performance drops by 42% in OOD-pretrained models.

Abstract

An end-to-end (E2E) visuomotor policy is typically treated as a unified whole, but recent approaches using out-of-domain (OOD) data to pretrain the visual encoder have cleanly separated the visual encoder from the network, with the remainder referred to as the policy. We propose Visual Alignment Testing, an experimental framework designed to evaluate the validity of this functional separation. Our results indicate that in E2E-trained models, visual encoders actively contribute to decision-making resulting from motor data supervision, contradicting the assumed functional separation. In contrast, OOD-pretrained models, where encoders lack this capability, experience an average performance drop of 42\% in our benchmark results, compared to the state-of-the-art performance achieved by E2E policies. We believe this initial exploration of visual encoders' role can provide a first step towards guiding future pretraining methods to address their decision-making ability, such as developing task-conditioned or context-aware encoders.