Parallel Algorithms for Tensor Train Arithmetic

TL;DR

This paper introduces scalable parallel algorithms for tensor train operations, enabling efficient computation on large tensors across distributed-memory systems, with demonstrated significant speedups and near-linear scaling.

Contribution

The paper develops and analyzes new parallel algorithms for TT tensor operations, optimized for distributed-memory systems, with performance improvements over existing tools.

Findings

Achieved 34x speedup on a 2GB tensor using 40 cores.

Demonstrated near-linear scaling on over 10,000 cores.

Outperformed MATLAB TT-Toolbox in tensor rounding performance.

Abstract

We present efficient and scalable parallel algorithms for performing mathematical operations for low-rank tensors represented in the tensor train (TT) format. We consider algorithms for addition, elementwise multiplication, computing norms and inner products, orthogonalization, and rounding (rank truncation). These are the kernel operations for applications such as iterative Krylov solvers that exploit the TT structure. The parallel algorithms are designed for distributed-memory computation, and we use a data distribution and strategy that parallelizes computations for individual cores within the TT format. We analyze the computation and communication costs of the proposed algorithms to show their scalability, and we present numerical experiments that demonstrate their efficiency on both shared-memory and distributed-memory parallel systems. For example, we observe better single-core performance than the existing MATLAB TT-Toolbox in rounding a 2GB TT tensor, and our implementation achieves a $34\times$ speedup using all 40 cores of a single node. We also show nearly linear parallel scaling on larger TT tensors up to over 10,000 cores for all mathematical operations.