Topological limits to parallel processing capability of network architectures

TL;DR

This paper reveals a fundamental tradeoff in network architectures between the capacity for learning and generalization versus the ability to perform multiple tasks simultaneously, with implications for AI development and understanding the human brain.

Contribution

It formally demonstrates the inherent tension between interactive and independent parallelism in network architectures, quantifying how shared representations limit multitasking.

Findings

Maximum number of tasks grows linearly with network size.

Expected concurrent tasks grow sub-linearly under realistic scenarios.

Shared representations constrain multitasking capacity.

Abstract

The ability to learn new tasks and generalize performance to others is one of the most remarkable characteristics of the human brain and of recent AI systems. The ability to perform multiple tasks simultaneously is also a signature characteristic of large-scale parallel architectures, that is evident in the human brain, and has been exploited effectively more traditional, massively parallel computational architectures. Here, we show that these two characteristics are in tension, reflecting a fundamental tradeoff between interactive parallelism that supports learning and generalization, and independent parallelism that supports processing efficiency through concurrent multitasking. We formally show that, while the maximum number of tasks that can be performed simultaneously grows linearly with network size, under realistic scenarios (e.g. in an unpredictable environment), the expected number that can be performed concurrently grows radically sub-linearly with network size. Hence, even modest reliance on shared representation strictly constrains the number of tasks that can be performed simultaneously, implying profound consequences for the development of artificial intelligence that optimally manages the tradeoff between learning and processing, and for understanding the human brains remarkably puzzling mix of sequential and parallel capabilities.