From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios

TL;DR

This paper introduces DenseWorld, a comprehensive benchmark for real-world dense prediction tasks, and proposes DenseDiT, a data-efficient model leveraging generative priors to improve real-world generalization with minimal additional parameters.

Contribution

The paper presents DenseWorld, a new benchmark for diverse real-world dense prediction tasks, and DenseDiT, a novel, parameter-efficient model that effectively adapts generative priors for practical applications.

Findings

DenseDiT outperforms existing baselines on DenseWorld.

DenseDiT requires less than 0.01% of training data compared to baselines.

Existing methods show significant performance drops in real-world scenarios.

Abstract

Dense prediction tasks hold significant importance of computer vision, aiming to learn pixel-wise annotated labels for input images. Despite advances in this field, existing methods primarily focus on idealized conditions, exhibiting limited real-world generalization and struggling with the acute scarcity of real-world data in practical scenarios. To systematically study this problem, we first introduce DenseWorld, a benchmark spanning a broad set of 25 dense prediction tasks that correspond to urgent real-world applications, featuring unified evaluation across tasks. We then propose DenseDiT, which exploits generative models' visual priors to perform diverse real-world dense prediction tasks through a unified strategy. DenseDiT combines a parameter-reuse mechanism and two lightweight branches that adaptively integrate multi-scale context. This design enables DenseDiT to achieve efficient tuning with less than 0.1% additional parameters, activating the visual priors while effectively adapting to diverse real-world dense prediction tasks. Evaluations on DenseWorld reveal significant performance drops in existing general and specialized baselines, highlighting their limited real-world generalization. In contrast, DenseDiT achieves superior results using less than 0.01% training data of baselines, underscoring its practical value for real-world deployment.