Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique

TL;DR

This paper evaluates the impact of gradient accumulation optimization on Swin Vision Transformer models, finding that it decreases accuracy and increases training time, thus questioning its suitability for such models.

Contribution

The study provides the first detailed analysis of how gradient accumulation optimization affects Swin ViT performance, highlighting potential drawbacks.

Findings

GAO decreases Swin ViT accuracy

GAO increases training time for Swin ViT

Applying GAO may not be suitable for transformer models

Abstract

Vision Transformers (ViTs) have emerged as a promising approach for visual recognition tasks, revolutionizing the field by leveraging the power of transformer-based architectures. Among the various ViT models, Swin Transformers have gained considerable attention due to their hierarchical design and ability to capture both local and global visual features effectively. This paper evaluates the performance of Swin ViT model using gradient accumulation optimization (GAO) technique. We investigate the impact of gradient accumulation optimization technique on the model's accuracy and training time. Our experiments show that applying the GAO technique leads to a significant decrease in the accuracy of the Swin ViT model, compared to the standard Swin Transformer model. Moreover, we detect a significant increase in the training time of the Swin ViT model when GAO model is applied. These findings suggest that applying the GAO technique may not be suitable for the Swin ViT model, and concern should be undertaken when using GAO technique for other transformer-based models.