Automated Composition of Picture-Synched Music Soundtracks for Movies

TL;DR

This paper presents a system that automatically composes video-synchronized music soundtracks by analyzing visual features and generating music that aligns with visual events, aiming to automate soundtrack creation.

Contribution

It introduces an innovative system combining video analysis and machine learning to generate synchronized soundtracks directly from video content, advancing automated film scoring.

Findings

User evaluations were positive about the generated soundtracks.

The system successfully synchronizes music with visual scene transitions.

Preliminary results show potential for automated soundtrack composition.

Abstract

We describe the implementation of and early results from a system that automatically composes picture-synched musical soundtracks for videos and movies. We use the phrase "picture-synched" to mean that the structure of the automatically composed music is determined by visual events in the input movie, i.e. the final music is synchronised to visual events and features such as cut transitions or within-shot key-frame events. Our system combines automated video analysis and computer-generated music-composition techniques to create unique soundtracks in response to the video input, and can be thought of as an initial step in creating a computerised replacement for a human composer writing music to fit the picture-locked edit of a movie. Working only from the video information in the movie, key features are extracted from the input video, using video analysis techniques, which are then fed into a machine-learning-based music generation tool, to compose a piece of music from scratch. The resulting soundtrack is tied to video features, such as scene transition markers and scene-level energy values, and is unique to the input video. Although the system we describe here is only a preliminary proof-of-concept, user evaluations of the output of the system have been positive.