SASE: A Searching Architecture for Squeeze and Excitation Operations

TL;DR

This paper introduces SASE, a neural architecture search method that automatically designs near-optimal attention modules for deep neural networks, improving performance across visual tasks by exploring a novel, subdivided search space.

Contribution

SASE is the first to subdivide the attention search space and search for architectures beyond existing modules, automating the design of effective squeeze-and-excitation attention blocks.

Findings

SASE-attention modules outperform state-of-the-art modules on ResNet-50/101.

The search space includes novel operations not previously used in attention mechanisms.

Extensive experiments validate the effectiveness of SASE in various visual tasks.

Abstract

In the past few years, channel-wise and spatial-wise attention blocks have been widely adopted as supplementary modules in deep neural networks, enhancing network representational abilities while introducing low complexity. Most attention modules follow a squeeze-and-excitation paradigm. However, to design such attention modules, requires a substantial amount of experiments and computational resources. Neural Architecture Search (NAS), meanwhile, is able to automate the design of neural networks and spares the numerous experiments required for an optimal architecture. This motivates us to design a search architecture that can automatically find near-optimal attention modules through NAS. We propose SASE, a Searching Architecture for Squeeze and Excitation operations, to form a plug-and-play attention block by searching within certain search space. The search space is separated into 4 different sets, each corresponds to the squeeze or excitation operation along the channel or spatial dimension. Additionally, the search sets include not only existing attention blocks but also other operations that have not been utilized in attention mechanisms before. To the best of our knowledge, SASE is the first attempt to subdivide the attention search space and search for architectures beyond currently known attention modules. The searched attention module is tested with extensive experiments across a range of visual tasks. Experimental results indicate that visual backbone networks (ResNet-50/101) using the SASE attention module achieved the best performance compared to those using the current state-of-the-art attention modules. Codes are included in the supplementary material, and they will be made public later.