GenCape: Structure-Inductive Generative Modeling for Category-Agnostic Pose Estimation

TL;DR

GenCape introduces a generative framework for category-agnostic pose estimation that infers instance-specific keypoint structures directly from images, improving accuracy and flexibility across diverse categories.

Contribution

The paper proposes a novel structure-aware generative model with iterative variational inference and graph transfer modules, enabling flexible, instance-specific keypoint relationship modeling without predefined skeletons.

Findings

Achieves significant improvements over graph-support baselines in 1- and 5-shot settings.

Maintains competitive performance against text-support methods.

Demonstrates effective structural inference from support images alone.

Abstract

Category-agnostic pose estimation (CAPE) aims to localize keypoints on query images from arbitrary categories, using only a few annotated support examples for guidance. Recent approaches either treat keypoints as isolated entities or rely on manually defined skeleton priors, which are costly to annotate and inherently inflexible across diverse categories. Such oversimplification limits the model's capacity to capture instance-wise structural cues critical for accurate pixel-level localization. To overcome these limitations, we propose GenCape, a Generative-based framework for CAPE that infers keypoint relationships solely from image-based support inputs, without additional textual descriptions or predefined skeletons. Our framework consists of two principal components: an iterative Structure-aware Variational Autoencoder (i-SVAE) and a Compositional Graph Transfer (CGT) module. The former infers soft, instance-specific adjacency matrices from support features through variational inference, embedded layer-wise into the Graph Transformer Decoder for progressive structural priors refinement. The latter adaptively aggregates multiple latent graphs into a query-aware structure via Bayesian fusion and attention-based reweighting, enhancing resilience to visual uncertainty and support-induced bias. This structure-aware design facilitates effective message propagation among keypoints and promotes semantic alignment across object categories with diverse keypoint topologies. Experimental results on the MP-100 dataset show that our method achieves substantial gains over graph-support baselines under both 1- and 5-shot settings, while maintaining competitive performance against text-support counterparts.