Efficient Universal Perception Encoder

TL;DR

EUPE is a compact, versatile vision encoder that distills knowledge from multiple domain experts, achieving high performance on diverse tasks efficiently suitable for edge devices.

Contribution

Introduces EUPE, a novel distillation approach that scales up to a large proxy teacher before scaling down, improving efficiency and versatility over previous methods.

Findings

EUPE matches or exceeds individual domain experts in diverse tasks.

EUPE outperforms previous agglomerative encoders in efficiency and performance.

Full EUPE models and code are publicly released.

Abstract

Running AI models on smart edge devices can unlock versatile user experiences, but presents challenges due to limited compute and the need to handle multiple tasks simultaneously. This requires a vision encoder with small size but powerful and versatile representations. We present our method, Efficient Universal Perception Encoder (EUPE), which offers both inference efficiency and universally good representations for diverse downstream tasks. We achieve this by distilling from multiple domain-expert foundation vision encoders. Unlike previous agglomerative methods that directly scale down from multiple teachers to an efficient encoder, we demonstrate the importance of first scaling up to a large proxy teacher and then scaling down from this single teacher. Experiments show that EUPE achieves on-par or better performance than individual domain experts of the same size on diverse task domains and also outperforms previous agglomerative encoders. We release the full family of EUPE models and the code to foster future research.