Shifted Window Fourier Transform And Retention For Image Captioning

TL;DR

This paper introduces SwiFTeR, a Fourier Transform-based architecture for image captioning that is lightweight, efficient, and scalable, aiming to improve performance on resource-constrained devices.

Contribution

SwiFTeR is a novel Fourier Transform and Retention-based architecture that significantly reduces computational complexity and memory usage in image captioning models.

Findings

SwiFTeR has only 20M parameters and requires 3.1 GFLOPs per forward pass.

It can generate 400 captions per second, demonstrating high efficiency.

Current caption quality is lower, but this is due to incomplete training, not architecture limitations.

Abstract

Image Captioning is an important Language and Vision task that finds application in a variety of contexts, ranging from healthcare to autonomous vehicles. As many real-world applications rely on devices with limited resources, much effort in the field was put into the development of lighter and faster models. However, much of the current optimizations focus on the Transformer architecture in contrast to the existence of more efficient methods. In this work, we introduce SwiFTeR, an architecture almost entirely based on Fourier Transform and Retention, to tackle the main efficiency bottlenecks of current light image captioning models, being the visual backbone's onerosity, and the decoder's quadratic cost. SwiFTeR is made of only 20M parameters, and requires 3.1 GFLOPs for a single forward pass. Additionally, it showcases superior scalability to the caption length and its small memory requirements enable more images to be processed in parallel, compared to the traditional transformer-based architectures. For instance, it can generate 400 captions in one second. Although, for the time being, the caption quality is lower (110.2 CIDEr-D), most of the decrease is not attributed to the architecture but rather an incomplete training practice which currently leaves much room for improvements. Overall, SwiFTeR points toward a promising direction to new efficient architectural design. The implementation code will be released in the future.