TVConv: Efficient Translation Variant Convolution for Layout-aware Visual Processing

TL;DR

TVConv introduces an efficient, layout-aware convolution method that adapts to spatial variance in images, significantly reducing computation and improving accuracy in face recognition and medical imaging tasks.

Contribution

The paper proposes TVConv, a novel layout-aware convolution that combines affinity maps and a weight-generating block for improved efficiency and accuracy in layout-specific visual applications.

Findings

Reduces computational cost by up to 3.1x in face recognition.

Improves throughput by 2.3x while maintaining high accuracy.

Enhances generalization in medical image segmentation.

Abstract

As convolution has empowered many smart applications, dynamic convolution further equips it with the ability to adapt to diverse inputs. However, the static and dynamic convolutions are either layout-agnostic or computation-heavy, making it inappropriate for layout-specific applications, e.g., face recognition and medical image segmentation. We observe that these applications naturally exhibit the characteristics of large intra-image (spatial) variance and small cross-image variance. This observation motivates our efficient translation variant convolution (TVConv) for layout-aware visual processing. Technically, TVConv is composed of affinity maps and a weight-generating block. While affinity maps depict pixel-paired relationships gracefully, the weight-generating block can be explicitly overparameterized for better training while maintaining efficient inference. Although conceptually simple, TVConv significantly improves the efficiency of the convolution and can be readily plugged into various network architectures. Extensive experiments on face recognition show that TVConv reduces the computational cost by up to 3.1x and improves the corresponding throughput by 2.3x while maintaining a high accuracy compared to the depthwise convolution. Moreover, for the same computation cost, we boost the mean accuracy by up to 4.21%. We also conduct experiments on the optic disc/cup segmentation task and obtain better generalization performance, which helps mitigate the critical data scarcity issue. Code is available at https://github.com/JierunChen/TVConv.