Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks

TL;DR

Tensorization reshapes neural network weights into higher-order tensors, offering promising avenues for model compression and interpretability, yet remains underutilized due to practical challenges and limited exploration.

Contribution

This paper highlights the potential of tensorized neural networks for compression and interpretability, emphasizing their unique properties and outlining future research directions.

Findings

Tensorization enables effective model compression.

TNNs offer increased interpretability through latent bond indices.

Potential for deeper understanding of feature evolution across layers.

Abstract

Tensorizing a neural network involves reshaping some or all of its dense weight matrices into higher-order tensors and approximating them using low-rank tensor network decompositions. This technique has shown promise as a model compression strategy for large-scale neural networks. However, despite encouraging empirical results, tensorized neural networks (TNNs) remain underutilized in mainstream deep learning. In this position paper, we offer a perspective on both the potential and current limitations of TNNs. We argue that TNNs represent a powerful yet underexplored framework for deep learning--one that deserves greater attention from both engineering and theoretical communities. Beyond compression, we highlight the value of TNNs as a flexible class of architectures with distinctive scaling properties and increased interpretability. A central feature of TNNs is the presence of bond indices, which introduce new latent spaces not found in conventional networks. These internal representations may provide deeper insight into the evolution of features across layers, potentially advancing the goals of mechanistic interpretability. We conclude by outlining several key research directions aimed at overcoming the practical barriers to scaling and adopting TNNs in modern deep learning workflows.