Efficient Convolutional Neural Networks for Diacritic Restoration

TL;DR

This paper introduces an efficient Acausal Temporal Convolutional Neural Network (A-TCN) for diacritic restoration, demonstrating significant speed and performance improvements over traditional BiLSTM models across multiple languages.

Contribution

It applies and evaluates a novel Acausal TCN architecture for diacritic restoration, showing it matches BiLSTM performance with faster inference.

Findings

A-TCN outperforms TCN in diacritic restoration accuracy.

A-TCN achieves 270%-334% faster inference than BiLSTM.

A-TCN performs comparably to BiLSTM across three languages.

Abstract

Diacritic restoration has gained importance with the growing need for machines to understand written texts. The task is typically modeled as a sequence labeling problem and currently Bidirectional Long Short Term Memory (BiLSTM) models provide state-of-the-art results. Recently, Bai et al. (2018) show the advantages of Temporal Convolutional Neural Networks (TCN) over Recurrent Neural Networks (RNN) for sequence modeling in terms of performance and computational resources. As diacritic restoration benefits from both previous as well as subsequent timesteps, we further apply and evaluate a variant of TCN, Acausal TCN (A-TCN), which incorporates context from both directions (previous and future) rather than strictly incorporating previous context as in the case of TCN. A-TCN yields significant improvement over TCN for diacritization in three different languages: Arabic, Yoruba, and Vietnamese. Furthermore, A-TCN and BiLSTM have comparable performance, making A-TCN an efficient alternative over BiLSTM since convolutions can be trained in parallel. A-TCN is significantly faster than BiLSTM at inference time (270%-334% improvement in the amount of text diacritized per minute).