P$^2$HCT: Plug-and-Play Hierarchical C2F Transformer for Multi-Scale Feature Fusion

TL;DR

P$^2$HCT is a lightweight, plug-and-play hierarchical transformer module that improves multi-scale feature fusion efficiency and accuracy in vision models, suitable for resource-constrained environments.

Contribution

The paper introduces P$^2$HCT, a novel hierarchical transformer module that reduces computational overhead while enhancing feature fusion for detection and classification tasks.

Findings

P$^2$HCT improves mAP by up to 0.9% on MS COCO.

Embedding P$^2$HCT into ResNet backbones boosts ImageNet top-1 accuracy.

P$^2$HCT achieves these gains with minimal latency increase.

Abstract

Feature fusion plays a pivotal role in achieving high performance in vision models, yet existing attention-based fusion techniques often suffer from substantial computational overhead and implementation complexity, particularly in resource-constrained settings. To address these limitations, we introduce the Plug-and-Play Hierarchical C2F Transformer (P$^2$HCT), a lightweight module that combines coarse-to-fine token selection with shared attention parameters to preserve spatial details while reducing inference cost. P$^2$HCT is trainable using coarse attention alone and can be seamlessly activated at inference to enhance accuracy without retraining. Integrated into real-time detectors such as YOLOv11-N/S/M, P$^2$HCT achieves mAP gains of 0.9\%, 0.5\%, and 0.4\% on MS COCO with minimal latency increase. Similarly, embedding P$^2$HCT into ResNet-18/50/101 backbones improves ImageNet top-1 accuracy by 6.5\%, 1.7\%, and 1.0\%, respectively. These results underscore P$^2$HCT's effectiveness as a hardware-friendly and general-purpose enhancement for both detection and classification tasks.