Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation
Satwik Kottur, Seungwhan Moon, Aram H. Markosyan, Hardik Shah, Babak, Damavandi, Alborz Geramifard

TL;DR
This paper introduces task-oriented dialog systems for interactive media montage creation, enabling seamless search and editing through multi-turn conversations, supported by a new dataset and a real-world mobile demo.
Contribution
It is the first to leverage multi-turn dialogs for media montage creation, introducing a new dataset and benchmarking language models for this challenging task.
Findings
State-of-the-art models face multimodal challenges in this task.
The dataset enables benchmarking of conversational media creation.
A mobile demo demonstrates real-world applicability.
Abstract
People capture photos and videos to relive and share memories of personal significance. Recently, media montages (stories) have become a popular mode of sharing these memories due to their intuitive and powerful storytelling capabilities. However, creating such montages usually involves a lot of manual searches, clicks, and selections that are time-consuming and cumbersome, adversely affecting user experiences. To alleviate this, we propose task-oriented dialogs for montage creation as a novel interactive tool to seamlessly search, compile, and edit montages from a media collection. To the best of our knowledge, our work is the first to leverage multi-turn conversations for such a challenging application, extending the previous literature studying simple media retrieval tasks. We collect a new dataset C3 (Conversational Content Creation), comprising 10k dialogs conditioned on media…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Video Analysis and Summarization · Digital Storytelling and Education
