Mixture of Latent Experts Using Tensor Products

TL;DR

This paper introduces a novel modular language model, TensorPoly, utilizing tensor product-based reparameterization and routing functions, which improves multi-task learning efficiency and performance by mitigating negative transfer.

Contribution

The paper proposes TensorPoly, a modular language model with tensor product reparameterization and innovative routing functions, enhancing multi-task learning and parameter efficiency.

Findings

Modular LMs outperform dense models in multi-task benchmarks.

TensorPoly-I achieves higher parameter efficiency and better performance.

The approach mitigates negative transfer in multi-task learning.

Abstract

In multi-task learning, the conventional approach involves training a model on multiple tasks simultaneously. However, the training signals from different tasks can interfere with one another, potentially leading to \textit{negative transfer}. To mitigate this, we investigate if modular language models can facilitate positive transfer and systematic generalization. Specifically, we propose a novel modular language model (\texttt{TensorPoly}), that balances parameter efficiency with nuanced routing methods. For \textit{modules}, we reparameterize Low-Rank Adaptation (\texttt{LoRA}) by employing an entangled tensor through the use of tensor product operations and name the resulting approach \texttt{TLoRA}. For \textit{routing function}, we tailor two innovative routing functions according to the granularity: \texttt{TensorPoly-I} which directs to each rank within the entangled tensor while \texttt{TensorPoly-II} offers a finer-grained routing approach targeting each order of the entangled tensor. The experimental results from the multi-task T0-benchmark demonstrate that: 1) all modular LMs surpass the corresponding dense approaches, highlighting the potential of modular language models to mitigate negative inference in multi-task learning and deliver superior outcomes. 2) \texttt{TensorPoly-I} achieves higher parameter efficiency in adaptation and outperforms other modular LMs, which shows the potential of our approach in multi-task transfer learning.