PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

TL;DR

PARTIME is a Python library that enables scalable, parallel processing of neural networks over streaming data by distributing computations across multiple GPUs, achieving near-linear speedups in online learning and inference.

Contribution

It introduces a pipeline-based parallel processing scheme for neural networks tailored for streaming data, improving scalability and efficiency over existing batch-oriented libraries.

Findings

Achieves near-linear speedup with up to 8 GPUs.

Effectively handles non-i.i.d. streaming data for learning.

Significantly reduces computation time compared to traditional methods.

Abstract

In this paper, we present PARTIME, a software library written in Python and based on PyTorch, designed specifically to speed up neural networks whenever data is continuously streamed over time, for both learning and inference. Existing libraries are designed to exploit data-level parallelism, assuming that samples are batched, a condition that is not naturally met in applications that are based on streamed data. Differently, PARTIME starts processing each data sample at the time in which it becomes available from the stream. PARTIME wraps the code that implements a feed-forward multi-layer network and it distributes the layer-wise processing among multiple devices, such as Graphics Processing Units (GPUs). Thanks to its pipeline-based computational scheme, PARTIME allows the devices to perform computations in parallel. At inference time this results in scaling capabilities that are theoretically linear with respect to the number of devices. During the learning stage, PARTIME can leverage the non-i.i.d. nature of the streamed data with samples that are smoothly evolving over time for efficient gradient computations. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning, distributing operations on up to 8 NVIDIA GPUs, showing significant speedups that are almost linear in the number of devices, mitigating the impact of the data transfer overhead.