Silo NLP's Participation at WAT2022

TL;DR

Silo NLP participated in WAT2022, developing Transformer-based models for multilingual and multimodal translation tasks, achieving top results in several language pairs and modalities.

Contribution

The paper introduces a system combining Transformer models and multimodal features for Asian language translation, with state-of-the-art performance.

Findings

Top performance in multiple translation tasks

Effective use of object tags as visual features

Successful fine-tuning of mBART-50 models

Abstract

This paper provides the system description of "Silo NLP's" submission to the Workshop on Asian Translation (WAT2022). We have participated in the Indic Multimodal tasks (English->Hindi, English->Malayalam, and English->Bengali Multimodal Translation). For text-only translation, we trained Transformers from scratch and fine-tuned mBART-50 models. For multimodal translation, we used the same mBART architecture and extracted object tags from the images to use as visual features concatenated with the text sequence. Our submission tops many tasks including English->Hindi multimodal translation (evaluation test), English->Malayalam text-only and multimodal translation (evaluation test), English->Bengali multimodal translation (challenge test), and English->Bengali text-only translation (evaluation test).