Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation

TL;DR

This paper introduces a novel end-to-end sign language translation model that eliminates the need for gloss annotations, using stochastic transformer layers with winner-takes-all units, achieving state-of-the-art BLEU-4 scores with reduced memory usage.

Contribution

The paper presents a new Transformer-based SLT model that does not require gloss groundtruth, employing stochastic winner-takes-all layers and variational inference for weights, with efficient compression at inference.

Findings

Achieved top BLEU-4 score on PHOENIX 2014T without gloss supervision.

Reduced memory footprint by over 70%.

Demonstrated effective stochastic layer integration in Transformer networks.

Abstract

Automating sign language translation (SLT) is a challenging real world application. Despite its societal importance, though, research progress in the field remains rather poor. Crucially, existing methods that yield viable performance necessitate the availability of laborious to obtain gloss sequence groundtruth. In this paper, we attenuate this need, by introducing an end-to-end SLT model that does not entail explicit use of glosses; the model only needs text groundtruth. This is in stark contrast to existing end-to-end models that use gloss sequence groundtruth, either in the form of a modality that is recognized at an intermediate model stage, or in the form of a parallel output process, jointly trained with the SLT model. Our approach constitutes a Transformer network with a novel type of layers that combines: (i) local winner-takes-all (LWTA) layers with stochastic winner sampling, instead of conventional ReLU layers, (ii) stochastic weights with posterior distributions estimated via variational inference, and (iii) a weight compression technique at inference time that exploits estimated posterior variance to perform massive, almost lossless compression. We demonstrate that our approach can reach the currently best reported BLEU-4 score on the PHOENIX 2014T benchmark, but without making use of glosses for model training, and with a memory footprint reduced by more than 70%.