Generalising Recursive Neural Models by Tensor Decomposition

TL;DR

This paper proposes a tensor-based framework for neural models that improves structural data aggregation, controlling model complexity via Tucker tensor decomposition, and introduces a Tensorial Tree-LSTM for tree classification tasks.

Contribution

It introduces a novel tensor-based aggregation method with parameter control via Tucker decomposition, and develops a Tensorial Tree-LSTM as a new model instance.

Findings

Effective regulation of model expressivity and complexity.

Improved performance on tree classification tasks.

Demonstrated benefits of tensor decomposition in neural models.

Abstract

Most machine learning models for structured data encode the structural knowledge of a node by leveraging simple aggregation functions (in neural models, typically a weighted sum) of the information in the node's neighbourhood. Nevertheless, the choice of simple context aggregation functions, such as the sum, can be widely sub-optimal. In this work we introduce a general approach to model aggregation of structural context leveraging a tensor-based formulation. We show how the exponential growth in the size of the parameter space can be controlled through an approximation based on the Tucker tensor decomposition. This approximation allows limiting the parameters space size, decoupling it from its strict relation with the size of the hidden encoding space. By this means, we can effectively regulate the trade-off between expressivity of the encoding, controlled by the hidden size, computational complexity and model generalisation, influenced by parameterisation. Finally, we introduce a new Tensorial Tree-LSTM derived as an instance of our framework and we use it to experimentally assess our working hypotheses on tree classification scenarios.