HYDRA -- Hyper Dependency Representation Attentions

TL;DR

HYDRA heads are lightweight, pretrained self-attention modules that inject linguistic knowledge into transformer models, improving performance without extensive retraining, offering a balanced approach between unsupervised learning and rigid knowledge enforcement.

Contribution

The paper introduces HYDRA heads, a novel lightweight method for integrating linguistic knowledge into transformers without retraining from scratch.

Findings

Improved model performance on benchmark datasets.

Lightweight and architecture-friendly design.

Effective transfer of linguistic knowledge into transformers.

Abstract

Attention is all we need as long as we have enough data. Even so, it is sometimes not easy to determine how much data is enough while the models are becoming larger and larger. In this paper, we propose HYDRA heads, lightweight pretrained linguistic self-attention heads to inject knowledge into transformer models without pretraining them again. Our approach is a balanced paradigm between leaving the models to learn unsupervised and forcing them to conform to linguistic knowledge rigidly as suggested in previous studies. Our experiment proves that the approach is not only the boost performance of the model but also lightweight and architecture friendly. We empirically verify our framework on benchmark datasets to show the contribution of linguistic knowledge to a transformer model. This is a promising result for a new approach to transferring knowledge from linguistic resources into transformer-based models.