Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencing

TL;DR

This paper introduces a scalable system architecture for real-time multilingual video translation using generative AI, addressing latency and computational challenges to enable practical multi-user video conferencing.

Contribution

It proposes a novel architecture with turn-taking and segmented processing to reduce complexity and latency, validated through implementation and performance analysis on various hardware.

Findings

Achieves real-time throughput ($\tau < 1.0$) on modern GPUs.

Reduces computational complexity from quadratic to linear in multi-user scenarios.

User study confirms high acceptability of initial delay for seamless experience.

Abstract

The real-time deployment of cascaded generative AI pipelines for applications like video translation is constrained by significant system-level challenges. These include the cumulative latency of sequential model inference and the quadratic ($\mathcal{O}(N^2)$) computational complexity that renders multi-user video conferencing applications unscalable. This paper proposes and evaluates a practical system-level framework designed to mitigate these critical bottlenecks. The proposed architecture incorporates a turn-taking mechanism to reduce computational complexity from quadratic to linear in multi-user scenarios, and a segmented processing protocol to manage inference latency for a perceptually real-time experience. We implement a proof-of-concept pipeline and conduct a rigorous performance analysis across a multi-tiered hardware setup, including commodity (NVIDIA RTX 4060), cloud (NVIDIA T4), and enterprise (NVIDIA A100) GPUs. Our objective evaluation demonstrates that the system achieves real-time throughput ($\tau < 1.0$) on modern hardware. A subjective user study further validates the approach, showing that a predictable, initial processing delay is highly acceptable to users in exchange for a smooth, uninterrupted playback experience. The work presents a validated, end-to-end system design that offers a practical roadmap for deploying scalable, real-time generative AI applications in multilingual communication platforms.