PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context

TL;DR

This paper introduces PETAH, a method for efficient task adaptation in hybrid transformer models for resource-limited vision applications, combining pruning and adaptation to outperform existing techniques.

Contribution

The paper presents PETAH, a novel task adaptation approach for hybrid transformers that improves performance and efficiency in resource-constrained environments.

Findings

PETAH outperforms existing task-adaptation methods for ViTs.

PETAH models require fewer parameters and are more hardware-efficient.

Combining PETAH with pruning yields highly compact, multi-task models.

Abstract

Following their success in natural language processing (NLP), there has been a shift towards transformer models in computer vision. While transformers perform well and offer promising multi-tasking performance, due to their high compute requirements, many resource-constrained applications still rely on convolutional or hybrid models that combine the benefits of convolution and attention layers and achieve the best results in the sub 100M parameter range. Simultaneously, task adaptation techniques that allow for the use of one shared transformer backbone for multiple downstream tasks, resulting in great storage savings at negligible cost in performance, have not yet been adopted for hybrid transformers. In this work, we investigate how to achieve the best task-adaptation performance and introduce PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers. We further combine PETAH adaptation with pruning to achieve highly performant and storage friendly models for multi-tasking. In our extensive evaluation on classification and other vision tasks, we demonstrate that our PETAH-adapted hybrid models outperform established task-adaptation techniques for ViTs while requiring fewer parameters and being more efficient on mobile hardware.