Benchmarking Soundtrack Recommendation Systems with SRBench
Aleksandar Stupar, Sebastian Michel

TL;DR
This paper introduces SRBench, a comprehensive benchmark for evaluating soundtrack recommendation systems based on preference judgments, involving diverse music genres, emotions, and query themes, with assessments from user studies and Mechanical Turk.
Contribution
It presents a novel benchmark for soundtrack recommendation evaluation using preference-based relevance judgments and compares state-of-the-art systems on this benchmark.
Findings
Relevance judgments vary with user agreement levels.
State-of-the-art systems show differing performance on SRBench.
Benchmark facilitates nuanced evaluation of soundtrack recommendation systems.
Abstract
In this work, a benchmark to evaluate the retrieval performance of soundtrack recommendation systems is proposed. Such systems aim at finding songs that are played as background music for a given set of images. The proposed benchmark is based on preference judgments, where relevance is considered a continuous ordinal variable and judgments are collected for pairs of songs with respect to a query (i.e., set of images). To capture a wide variety of songs and images, we use a large space of possible music genres, different emotions expressed through music, and various query-image themes. The benchmark consists of two types of relevance assessments: (i) judgments obtained from a user study, that serve as a "gold standard" for (ii) relevance judgments gathered through Amazon's Mechanical Turk. We report on an analysis of relevance judgments based on different levels of user agreement and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic and Audio Processing · Mobile Crowdsensing and Crowdsourcing · Data Stream Mining Techniques
