Hi Sheldon! Creating Deep Personalized Characters from TV Shows

TL;DR

This paper introduces a new task called Deep Personalized Character Creation (DPCC) that generates multimodal responses matching a character's personality from TV shows, supported by a large new dataset and baseline methods.

Contribution

The paper proposes the DPCC task, introduces the DPCD dataset with extensive multimodal character data, and develops baseline models for personalized multimodal character generation.

Findings

Baseline models can generate character-matched multimodal responses.

The DPCD dataset is approximately ten times larger than existing datasets.

Subjective and objective evaluations show promising results for personalized character responses.

Abstract

Imagine an interesting multimodal interactive scenario that you can see, hear, and chat with an AI-generated digital character, who is capable of behaving like Sheldon from The Big Bang Theory, as a DEEP copy from appearance to personality. Towards this fantastic multimodal chatting scenario, we propose a novel task, named Deep Personalized Character Creation (DPCC): creating multimodal chat personalized characters from multimodal data such as TV shows. Specifically, given a single- or multi-modality input (text, audio, video), the goal of DPCC is to generate a multi-modality (text, audio, video) response, which should be well-matched the personality of a specific character such as Sheldon, and of high quality as well. To support this novel task, we further collect a character centric multimodal dialogue dataset, named Deep Personalized Character Dataset (DPCD), from TV shows. DPCD contains character-specific multimodal dialogue data of ~10k utterances and ~6 hours of audio/video per character, which is around 10 times larger compared to existing related datasets.On DPCD, we present a baseline method for the DPCC task and create 5 Deep personalized digital Characters (DeepCharacters) from Big Bang TV Shows. We conduct both subjective and objective experiments to evaluate the multimodal response from DeepCharacters in terms of characterization and quality. The results demonstrates that, on our collected DPCD dataset, the proposed baseline can create personalized digital characters for generating multimodal response.Our collected DPCD dataset, the code of data collection and our baseline will be published soon.