Distillation of Diffusion Features for Semantic Correspondence

TL;DR

This paper introduces a novel knowledge distillation method that combines large vision models into a smaller, efficient model for semantic correspondence, enhanced by 3D data, achieving superior performance with less computation.

Contribution

The work presents a new distillation technique for combining multiple large models into a single efficient model for semantic correspondence, incorporating 3D data to improve accuracy without human annotations.

Findings

Distilled model outperforms state-of-the-art methods.

Significant reduction in computational cost.

Enhanced performance with 3D data augmentation.

Abstract

Semantic correspondence, the task of determining relationships between different parts of images, underpins various applications including 3D reconstruction, image-to-image translation, object tracking, and visual place recognition. Recent studies have begun to explore representations learned in large generative image models for semantic correspondence, demonstrating promising results. Building on this progress, current state-of-the-art methods rely on combining multiple large models, resulting in high computational demands and reduced efficiency. In this work, we address this challenge by proposing a more computationally efficient approach. We propose a novel knowledge distillation technique to overcome the problem of reduced efficiency. We show how to use two large vision foundation models and distill the capabilities of these complementary models into one smaller model that maintains high accuracy at reduced computational cost. Furthermore, we demonstrate that by incorporating 3D data, we are able to further improve performance, without the need for human-annotated correspondences. Overall, our empirical results demonstrate that our distilled model with 3D data augmentation achieves performance superior to current state-of-the-art methods while significantly reducing computational load and enhancing practicality for real-world applications, such as semantic video correspondence. Our code and weights are publicly available on our project page.