C3LLM: Conditional Multimodal Content Generation Using Large Language Models

TL;DR

C3LLM introduces a unified multimodal framework that leverages large language models to generate audio from video, text, or audio inputs, enhancing fidelity and semantic alignment across modalities.

Contribution

The paper presents a novel hierarchical, discrete-token-based approach for multimodal audio generation using LLMs, integrating video, audio, and text tasks into a single model.

Findings

Improved semantic alignment over previous methods

Enhanced audio fidelity through hierarchical token generation

Unified multimodal generation in an end-to-end framework

Abstract

We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner. Our contributions are as follows. First, we adapt a hierarchical structure for audio generation tasks with pre-trained audio codebooks. Specifically, we train the LLM to generate audio semantic tokens from the given conditions, and further use a non-autoregressive transformer to generate different levels of acoustic tokens in layers to better enhance the fidelity of the generated audio. Second, based on the intuition that LLMs were originally designed for discrete tasks with the next-word prediction method, we use the discrete representation for audio generation and compress their semantic meanings into acoustic tokens, similar to adding "acoustic vocabulary" to LLM. Third, our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model, providing more versatility in an end-to-end fashion. Our C3LLM achieves improved results through various automated evaluation metrics, providing better semantic alignment compared to previous methods.