VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer

TL;DR

VPNeXt introduces a simplified yet effective dense decoding approach for Plain Vision Transformers, utilizing novel modules to improve semantic segmentation performance and surpass existing benchmarks.

Contribution

The paper proposes VPNeXt, a new model that simplifies dense decoding in ViT by replacing complex architectures with the VCR and ViTUp modules, achieving state-of-the-art results.

Findings

Achieved state-of-the-art performance on semantic segmentation tasks.

Significantly surpassed the mIoU barrier on VOC2012 dataset.

Validated effectiveness through ablation studies and visualizations.

Abstract

We present VPNeXt, a new and simple model for the Plain Vision Transformer (ViT). Unlike the many related studies that share the same homogeneous paradigms, VPNeXt offers a fresh perspective on dense representation based on ViT. In more detail, the proposed VPNeXt addressed two concerns about the existing paradigm: (1) Is it necessary to use a complex Transformer Mask Decoder architecture to obtain good representations? (2) Does the Plain ViT really need to depend on the mock pyramid feature for upsampling? For (1), we investigated the potential underlying reasons that contributed to the effectiveness of the Transformer Decoder and introduced the Visual Context Replay (VCR) to achieve similar effects efficiently. For (2), we introduced the ViTUp module. This module fully utilizes the previously overlooked ViT real pyramid feature to achieve better upsampling results compared to the earlier mock pyramid feature. This represents the first instance of such functionality in the field of semantic segmentation for Plain ViT. We performed ablation studies on related modules to verify their effectiveness gradually. We conducted relevant comparative experiments and visualizations to show that VPNeXt achieved state-of-the-art performance with a simple and effective design. Moreover, the proposed VPNeXt significantly exceeded the long-established mIoU wall/barrier of the VOC2012 dataset, setting a new state-of-the-art by a large margin, which also stands as the largest improvement since 2015.