Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipeline

TL;DR

This paper presents a high-performance pipeline for generating listener head videos in dyadic interactions, enhancing semantic extraction and achieving top results in a competitive challenge.

Contribution

It introduces improvements to the hierarchical semantic extraction in the audio encoder and refines the decoder, renderer, and post-processing modules for listener head video synthesis.

Findings

Achieved first place on the official leaderboard for listening head generation.

Enhanced hierarchical semantic extraction improves video synthesis quality.

Proposed pipeline outperforms baseline methods in the challenge.

Abstract

In dyadic speaker-listener interactions, the listener's head reactions along with the speaker's head movements, constitute an important non-verbal semantic expression together. The listener Head generation task aims to synthesize responsive listener's head videos based on audios of the speaker and reference images of the listener. Compared to the Talking-head generation, it is more challenging to capture the correlation clues from the speaker's audio and visual information. Following the ViCo baseline scheme, we propose a high-performance solution by enhancing the hierarchical semantic extraction capability of the audio encoder module and improving the decoder part, renderer and post-processing modules. Our solution gets the first place on the official leaderboard for the track of listening head generation. This paper is a technical report of ViCo@2023 Conversational Head Generation Challenge in ACM Multimedia 2023 conference.