TenExp: Mixture-of-Experts-Based Tensor Decomposition Structure Search Framework

TL;DR

TenExp introduces a novel mixture-of-experts framework for tensor decomposition structure search, enabling dynamic selection and combination of decompositions to better capture data structures, outperforming existing fixed-family methods.

Contribution

The paper presents TenExp, a flexible framework that allows for dynamic mixture-of-decompositions, overcoming limitations of fixed factor-interaction families in tensor decomposition search.

Findings

TenExp outperforms state-of-the-art methods on synthetic datasets.

TenExp effectively captures complex data structures with decomposition mixtures.

Theoretical error bounds demonstrate TenExp's approximation capabilities.

Abstract

Recently, tensor decompositions continue to emerge and receive increasing attention. Selecting a suitable tensor decomposition to exactly capture the low-rank structures behind the data is at the heart of the tensor decomposition field, which remains a challenging and relatively under-explored problem. Current tensor decomposition structure search methods are still confined by a fixed factor-interaction family (e.g., tensor contraction) and cannot deliver the mixture of decompositions. To address this problem, we elaborately design a mixture-of-experts-based tensor decomposition structure search framework (termed as TenExp), which allows us to dynamically select and activate suitable tensor decompositions in an unsupervised fashion. This framework enjoys two unique advantages over the state-of-the-art tensor decomposition structure search methods. Firstly, TenExp can provide a suitable single decomposition beyond a fixed factor-interaction family. Secondly, TenExp can deliver a suitable mixture of decompositions beyond a single decomposition. Theoretically, we also provide the approximation error bound of TenExp, which reveals the approximation capability of TenExp. Extensive experiments on both synthetic and realistic datasets demonstrate the superiority of the proposed TenExp compared to the state-of-the-art tensor decomposition-based methods.