SG-MIM: Structured Knowledge Guided Efficient Pre-training for Dense Prediction

TL;DR

SG-MIM introduces a structured knowledge guided pre-training framework that enhances dense prediction tasks like depth estimation and segmentation by effectively integrating structured knowledge without extra annotations.

Contribution

It proposes a lightweight relational guidance framework and a selective masking strategy to incorporate structured knowledge into masked image modeling for dense prediction tasks.

Findings

Outperforms existing methods on KITTI, NYU-v2, and ADE20k datasets.

Improves monocular depth estimation accuracy.

Enhances semantic segmentation performance.

Abstract

Masked Image Modeling (MIM) techniques have redefined the landscape of computer vision, enabling pre-trained models to achieve exceptional performance across a broad spectrum of tasks. Despite their success, the full potential of MIM-based methods in dense prediction tasks, particularly in depth estimation, remains untapped. Existing MIM approaches primarily rely on single-image inputs, which makes it challenging to capture the crucial structured information, leading to suboptimal performance in tasks requiring fine-grained feature representation. To address these limitations, we propose SG-MIM, a novel Structured knowledge Guided Masked Image Modeling framework designed to enhance dense prediction tasks by utilizing structured knowledge alongside images. SG-MIM employs a lightweight relational guidance framework, allowing it to guide structured knowledge individually at the feature level rather than naively combining at the pixel level within the same architecture, as is common in traditional multi-modal pre-training methods. This approach enables the model to efficiently capture essential information while minimizing discrepancies between pre-training and downstream tasks. Furthermore, SG-MIM employs a selective masking strategy to incorporate structured knowledge, maximizing the synergy between general representation learning and structured knowledge-specific learning. Our method requires no additional annotations, making it a versatile and efficient solution for a wide range of applications. Our evaluations on the KITTI, NYU-v2, and ADE20k datasets demonstrate SG-MIM's superiority in monocular depth estimation and semantic segmentation.