Revisiting Vision Transformer from the View of Path Ensemble

TL;DR

This paper presents a novel perspective on Vision Transformers as multi-path ensemble networks, introduces path pruning and re-weighting techniques, and demonstrates improved performance and deeper architectures through these strategies.

Contribution

It reinterprets ViTs as ensemble networks with multiple paths, proposes path pruning and ensemble re-weighting methods, and enhances ViT depth and performance via these new strategies.

Findings

Path pruning improves accuracy by removing underperforming paths.

Re-weighting ensemble components enhances feature representation.

Path strategies enable ViTs to be deeper and filter low-frequency signals.

Abstract

Vision Transformers (ViTs) are normally regarded as a stack of transformer layers. In this work, we propose a novel view of ViTs showing that they can be seen as ensemble networks containing multiple parallel paths with different lengths. Specifically, we equivalently transform the traditional cascade of multi-head self-attention (MSA) and feed-forward network (FFN) into three parallel paths in each transformer layer. Then, we utilize the identity connection in our new transformer form and further transform the ViT into an explicit multi-path ensemble network. From the new perspective, these paths perform two functions: the first is to provide the feature for the classifier directly, and the second is to provide the lower-level feature representation for subsequent longer paths. We investigate the influence of each path for the final prediction and discover that some paths even pull down the performance. Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths. We also demonstrate that our path combination strategies can help ViTs go deeper and act as high-pass filters to filter out partial low-frequency signals. To further enhance the representation of paths served for subsequent paths, self-distillation is applied to transfer knowledge from the long paths to the short paths. This work calls for more future research to explain and design ViTs from new perspectives.