Automatic background animation generation aligned with LLM-generated lyrics for children’s songs

TL;DR

This paper introduces an AI-based system that automatically creates animated videos for children's songs by generating lyrics and matching background animations.

Contribution

The novelty lies in combining language models for lyric generation with diffusion models for background animation, tailored for children's songs.

Findings

CascadeSD outperformed conventional diffusion models in generating background images.

Landscape and image-style prompting improved the quality of generated animations.

The proposed pipeline produced better results than existing text-to-video models for children's songs.

Abstract

Media content creation is a labor-intensive and expensive process requiring significant time. Recent developments in artificial intelligence have introduced generative models, which have significant potential in the entertainment industry. Meanwhile, demand for video content tailored to children’s songs has steadily increased, reflecting their significant contribution to early education and entertainment. In this paper, we present a generative model-based approach to automated video creation for children’s songs. The proposed pipeline consists of three key steps: generating lyrics using a language model, producing background images with a diffusion model, and overlaying dynamic visual effects to enhance the final output. Our experiments include a comparison of conventional diffusion models and prompt engineering methods, highlighting the superior performance of CascadeSD and the efficacy of landscape or image-style prompting. Lastly, we provide experimental results comparing text-to-video models with our pipeline. The code for our project is available in the following repository: https://github.com/KhrTim/BAGen.

Figures