SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression

TL;DR

The paper introduces SEE, a novel compression method for transformer models that uses sememe-based low-rank approximation and quantum entanglement principles to reduce size and computational costs while maintaining performance.

Contribution

It proposes a new sememe entanglement encoding technique guided by expert knowledge, enabling effective model compression with minimal performance loss.

Findings

Achieves stable performance after compression.

Reduces model size and computational costs.

Applicable to various transformer model sizes.

Abstract

Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resource-constrained scenarios. An effective approach is to eliminate redundant model parameters and computational costs while incorporating efficient expert-derived knowledge structures to achieve a balance between compression and performance. Therefore, we propose the \textit{Sememe Entanglement Encoding (SEE)} algorithm. Guided by expert prior knowledge, the model is compressed through the low-rank approximation idea. In Entanglement Embedding, basic semantic units such as sememes are represented as low-dimensional vectors, and then reconstructed into high-dimensional word embeddings through the combination of generalized quantum entanglement. We adapt the Sememe Entanglement Encoding algorithm to transformer-based models of different magnitudes. Experimental results indicate that our approach achieves stable performance while compressing model parameters and computational costs.