High-Level Parallelism and Nested Features for Dynamic Inference Cost and Top-Down Attention

TL;DR

This paper presents a new neural network topology inspired by human perception that combines dynamic inference cost management with top-down attention, enabling selective activation and improved efficiency.

Contribution

It introduces a novel network design with nested high-level features, a cutout technique for selective activation, and a top-down attention mechanism for dynamic inference control.

Findings

Achieves up to 73.48% parameter exclusion and 84.41% GMAC reduction.

Reduces parameters by an average of 40% and GMACs by 8% across tested models.

Demonstrates improved efficiency and adaptability for various applications.

Abstract

This paper introduces a novel network topology that seamlessly integrates dynamic inference cost with a top-down attention mechanism, addressing two significant gaps in traditional deep learning models. Drawing inspiration from human perception, we combine sequential processing of generic low-level features with parallelism and nesting of high-level features. This design not only reflects a finding from recent neuroscience research regarding - spatially and contextually distinct neural activations - in human cortex, but also introduces a novel "cutout" technique: the ability to selectively activate %segments of the network for task-relevant only network segments of task-relevant categories to optimize inference cost and eliminate the need for re-training. We believe this paves the way for future network designs that are lightweight and adaptable, making them suitable for a wide range of applications, from compact edge devices to large-scale clouds. Our proposed topology also comes with a built-in top-down attention mechanism, which allows processing to be directly influenced by either enhancing or inhibiting category-specific high-level features, drawing parallels to the selective attention mechanism observed in human cognition. Using targeted external signals, we experimentally enhanced predictions across all tested models. In terms of dynamic inference cost our methodology can achieve an exclusion of up to $73.48\,\%$ of parameters and $84.41\,\%$ fewer giga-multiply-accumulate (GMAC) operations, analysis against comparative baselines show an average reduction of $40\,\%$ in parameters and $8\,\%$ in GMACs across the cases we evaluated.