Exploiting Symmetry in Tensors for High Performance: Multiplication with Symmetric Tensors

TL;DR

This paper introduces a blocked storage format and algorithms for symmetric tensors that significantly reduce storage and computation by exploiting symmetry and partial symmetry, with practical implementation and preliminary performance gains.

Contribution

It proposes a novel blocked storage format and block-based algorithms that leverage symmetry in tensors to reduce storage and computational costs.

Findings

Storage reduced by a factor of O(m!)

Computational requirements reduced by a factor of O((m+1)!/2^m)

Preliminary results show reduced computational time

Abstract

Symmetric tensor operations arise in a wide variety of computations. However, the benefits of exploiting symmetry in order to reduce storage and computation is in conflict with a desire to simplify memory access patterns. In this paper, we propose a blocked data structure (Blocked Compact Symmetric Storage) wherein we consider the tensor by blocks and store only the unique blocks of a symmetric tensor. We propose an algorithm-by-blocks, already shown of benefit for matrix computations, that exploits this storage format by utilizing a series of temporary tensors to avoid redundant computation. Further, partial symmetry within temporaries is exploited to further avoid redundant storage and redundant computation. A detailed analysis shows that, relative to storing and computing with tensors without taking advantage of symmetry and partial symmetry, storage requirements are reduced by a factor of $ O\left( m! \right)$ and computational requirements by a factor of $O\left( (m+1)!/2^m \right)$, where $ m $ is the order of the tensor. However, as the analysis shows, care must be taken in choosing the correct block size to ensure these storage and computational benefits are achieved (particularly for low-order tensors). An implementation demonstrates that storage is greatly reduced and the complexity introduced by storing and computing with tensors by blocks is manageable. Preliminary results demonstrate that computational time is also reduced. The paper concludes with a discussion of how insights in this paper point to opportunities for generalizing recent advances in the domain of linear algebra libraries to the field of multi-linear computation.