Translation, Sentiment and Voices: A Computational Model to Translate and Analyse Voices from Real-Time Video Calling

TL;DR

This paper introduces VidALL, a system for analyzing emotions and translating speech in real-time video calls, using neural networks to improve translation accuracy and emotional understanding.

Contribution

It presents a novel audio-visual communication system that extracts speech, analyzes emotional polarity, and translates English to French using neural network models.

Findings

Embedded-RNN achieved 88.71% accuracy in translation

Polarity effectively estimates human emotion

System enables real-time emotion and translation analysis

Abstract

With internet quickly becoming an easy access to many, voice calling over internet is slowly gaining momentum. Individuals has been engaging in video communication across the world in different languages. The decade saw the emergence of language translation using neural networks as well. With more data being generated in audio and visual forms, there has become a need and a challenge to analyse such information for many researchers from academia and industry. The availability of video chat corpora is limited as organizations protect user privacy and ensure data security. For this reason, an audio-visual communication system (VidALL) has been developed and audio-speeches were extracted. To understand human nature while answering a video call, an analysis was conducted where polarity and vocal intensity were considered as parameters. Simultaneously, a translation model using a neural approach was developed to translate English sentences to French. Simple RNN-based and Embedded-RNN based models were used for the translation model. BLEU score and target sentence comparators were used to check sentence correctness. Embedded-RNN showed an accuracy of 88.71 percentage and predicted correct sentences. A key finding suggest that polarity is a good estimator to understand human emotion.