Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling

TL;DR

Upsample Anything introduces a simple, universal, and highly effective test-time optimization method that restores low-resolution features to high-resolution outputs without retraining, improving pixel-level tasks across various models and modalities.

Contribution

It proposes a lightweight, per-image optimization framework that learns an anisotropic Gaussian kernel for feature upsampling, outperforming existing methods without dataset-specific retraining.

Findings

Achieves state-of-the-art results in semantic segmentation and depth estimation

Runs in approximately 0.419 seconds per 224x224 image

Effectively transfers across architectures and modalities

Abstract

We present \textbf{Upsample Anything}, a lightweight test-time optimization (TTO) framework that restores low-resolution features to high-resolution, pixel-wise outputs without any training. Although Vision Foundation Models demonstrate strong generalization across diverse downstream tasks, their representations are typically downsampled by 14x/16x (e.g., ViT), which limits their direct use in pixel-level applications. Existing feature upsampling approaches depend on dataset-specific retraining or heavy implicit optimization, restricting scalability and generalization. Upsample Anything addresses these issues through a simple per-image optimization that learns an anisotropic Gaussian kernel combining spatial and range cues, effectively bridging Gaussian Splatting and Joint Bilateral Upsampling. The learned kernel acts as a universal, edge-aware operator that transfers seamlessly across architectures and modalities, enabling precise high-resolution reconstruction of features, depth, or probability maps. It runs in only $\approx0.419 \text{s}$ per 224x224 image and achieves state-of-the-art performance on semantic segmentation, depth estimation, and both depth and probability map upsampling. \textbf{Project page:} \href{https://seominseok0429.github.io/Upsample-Anything/}{https://seominseok0429.github.io/Upsample-Anything/}