On Optimizing Distributed Tucker Decomposition for Sparse Tensors

TL;DR

This paper introduces a lightweight, near-optimal distribution scheme for parallel Tucker decomposition of sparse tensors, significantly reducing execution time on distributed systems compared to prior methods.

Contribution

We propose a simple, efficient distribution scheme for distributed Tucker decomposition that balances performance and setup cost, outperforming complex hypergraph-based methods.

Findings

Achieves up to 3x faster HOOI execution time.

Distribution setup time is comparable to lightweight schemes.

Scheme is near-optimal on key computational metrics.

Abstract

The Tucker decomposition generalizes the notion of Singular Value Decomposition (SVD) to tensors, the higher dimensional analogues of matrices. We study the problem of constructing the Tucker decomposition of sparse tensors on distributed memory systems via the HOOI procedure, a popular iterative method. The scheme used for distributing the input tensor among the processors (MPI ranks) critically influences the HOOI execution time. Prior work has proposed different distribution schemes: an offline scheme based on sophisticated hypergraph partitioning method and simple, lightweight alternatives that can be used real-time. While the hypergraph based scheme typically results in faster HOOI execution time, being complex, the time taken for determining the distribution is an order of magnitude higher than the execution time of a single HOOI iteration. Our main contribution is a lightweight distribution scheme, which achieves the best of both worlds. We show that the scheme is near-optimal on certain fundamental metrics associated with the HOOI procedure and as a result, near-optimal on the computational load (FLOPs). Though the scheme may incur higher communication volume, the computation time is the dominant factor and as the result, the scheme achieves better performance on the overall HOOI execution time. Our experimental evaluation on large real-life tensors (having up to 4 billion elements) shows that the scheme outperforms the prior schemes on the HOOI execution time by a factor of up to 3x. On the other hand, its distribution time is comparable to the prior lightweight schemes and is typically lesser than the execution time of a single HOOI iteration.