Banyan: Improved Representation Learning with Explicit Structure

TL;DR

Banyan is a novel model that uses explicit hierarchical structure and diagonalized message passing to achieve high-quality semantic representations efficiently, especially in low-resource NLP tasks.

Contribution

It introduces an entangled hierarchical tree structure and diagonalized message passing, enabling superior performance with minimal parameters compared to larger transformers.

Findings

Outperforms larger transformer models with only 14 parameters.

Excels in low-resource language settings.

Offers an efficient and interpretable NLP approach.

Abstract

We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.