Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings

TL;DR

This paper introduces a novel architecture for dense object tracking that combines the generality of dense embeddings with the accuracy of sparse keypoints by using task adaptation via latent embeddings, enabling zero-shot transfer in robotic tasks.

Contribution

A new architecture inspired by few-shot task adaptation that conditions on keypoint embeddings, balancing generality and accuracy in dense object tracking.

Findings

Achieves accuracy close to sparse keypoint methods

Maintains generality of dense embeddings

Enables zero-shot transfer to new objects in robotics

Abstract

Dense object tracking, the ability to localize specific object points with pixel-level accuracy, is an important computer vision task with numerous downstream applications in robotics. Existing approaches either compute dense keypoint embeddings in a single forward pass, meaning the model is trained to track everything at once, or allocate their full capacity to a sparse predefined set of points, trading generality for accuracy. In this paper we explore a middle ground based on the observation that the number of relevant points at a given time are typically relatively few, e.g. grasp points on a target object. Our main contribution is a novel architecture, inspired by few-shot task adaptation, which allows a sparse-style network to condition on a keypoint embedding that indicates which point to track. Our central finding is that this approach provides the generality of dense-embedding models, while offering accuracy significantly closer to sparse-keypoint approaches. We present results illustrating this capacity vs. accuracy trade-off, and demonstrate the ability to zero-shot transfer to new object instances (within-class) using a real-robot pick-and-place task.