AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Generating Background Music for Videos using Large-Scale Web Data

Generating background music for videos is a time-consuming and complex task. However, new research is making it easier than ever for people to create engaging videos with the perfect music. Researchers at UNC Chapel Hill and ByteDance Inc. have developed a new model called VMAS, which stands for Video-Music Alignment Scheme. This model can generate realistic and diverse background music for videos by learning from a massive collection of web videos and their accompanying music soundtracks.

Previous video-to-music generation methods were limited by their reliance on symbolic music annotations, such as MIDI files. These annotations are limited in quantity and diversity, and they cannot capture the full range of music’s expressiveness. Additionally, these methods were typically trained on small-scale datasets, which limited their ability to generalize to different video types and musical styles.

VMAS, however, leverages the massive amount of data available on the internet. The researchers have created a new dataset called DISCO-MV which is orders of magnitude larger than any previously used dataset for video music generation. This dataset contains 2.2 million video-music samples, spanning a diverse range of genres and musical styles.

One of the key innovations of VMAS is its video-music alignment scheme. This scheme ensures that the generated music is closely aligned with the visual content of the video. It incorporates two strategies:

Another innovation in VMAS is a temporal video encoder. This encoder allows the model to efficiently process videos with many densely sampled frames, which captures subtle visual cues that are important for generating realistic background music.

The researchers evaluated VMAS against a range of existing methods, including methods that rely on symbolic music annotations, waveform music generation, and text-to-music generation. VMAS outperformed all of these methods in terms of music quality and alignment with the video. In human evaluations, participants consistently preferred videos with music generated by VMAS.

The development of VMAS is a significant step forward in the field of video-to-music generation. It opens up new possibilities for creating engaging videos, and it demonstrates the potential of using large-scale web data to train powerful and versatile machine learning models. This research is likely to lead to even more innovative approaches for generating music for videos in the future.