Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

TL;DR

This paper introduces the concept of input elasticity for vision transformers, allowing flexible patch sampling, and proposes architectural modifications to enhance this elasticity, broadening their applicability in real-world visual exploration tasks.

Contribution

It formalizes input elasticity for vision transformers, develops an evaluation protocol, and proposes architectural changes to improve their flexibility in handling variable input patches.

Findings

Increased elasticity improves adaptability to diverse visual inputs

Proposed modifications enhance transformer flexibility without significant performance loss

Identified challenges in balancing elasticity and accuracy

Abstract

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.