Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices

TL;DR

This paper proposes reducing parameter redundancy in biaffine classifiers by using symmetric and circulant weight matrices, leading to fewer parameters and improved or comparable accuracy in dependency parsing tasks.

Contribution

It introduces a novel approach to decrease parameters in biaffine classifiers through symmetric and circulant matrix assumptions, enhancing efficiency without sacrificing performance.

Findings

Achieved over 16% parameter reduction in dependency parsing models.

Maintained or improved accuracy on most treebanks.

Demonstrated effectiveness of symmetric and circulant matrices in reducing overfitting.

Abstract

Currently, the biaffine classifier has been attracting attention as a method to introduce an attention mechanism into the modeling of binary relations. For instance, in the field of dependency parsing, the Deep Biaffine Parser by Dozat and Manning has achieved state-of-the-art performance as a graph-based dependency parser on the English Penn Treebank and CoNLL 2017 shared task. On the other hand, it is reported that parameter redundancy in the weight matrix in biaffine classifiers, which has O(n^2) parameters, results in overfitting (n is the number of dimensions). In this paper, we attempted to reduce the parameter redundancy by assuming either symmetry or circularity of weight matrices. In our experiments on the CoNLL 2017 shared task dataset, our model achieved better or comparable accuracy on most of the treebanks with more than 16% parameter reduction.