It's Only Words And Words Are All I Have

TL;DR

This paper demonstrates that distributed representations of lyrics can effectively predict music genre and popularity, surpassing traditional handcrafted features and weak indicators, thus enhancing music mining and NLP tasks.

Contribution

It introduces a novel approach using distributed lyric representations for genre and popularity prediction, overcoming limitations of prior handcrafted feature methods.

Findings

Achieved around 0.6 F1 score for genre classification

Achieved around 0.6 F1 score for popularity prediction

Showed lyrics improve distributed representation quality

Abstract

The central idea of this paper is to demonstrate the strength of lyrics for music mining and natural language processing (NLP) tasks using the distributed representation paradigm. For music mining, we address two prediction tasks for songs: genre and popularity. Existing works for both these problems have two major bottlenecks. First, they represent lyrics using handcrafted features that require intricate knowledge of language and music. Second, they consider lyrics as a weak indicator of genre and popularity. We overcome both the bottlenecks by representing lyrics using distributed representation. In our work, genre identification is a multi-class classification task whereas popularity prediction is a binary classification task. We achieve an F1 score of around 0.6 for both the tasks using only lyrics. Distributed representation of words is now heavily used for various NLP algorithms. We show that lyrics can be used to improve the quality of this representation.

Figures

Tables

Table 1. Table 1: F1-Scores for Genre Prediction. Highest value for each genre is in bold.

	Metal	Country	Religious	Rap	R&B	Reggae	Folk	Blues	Average
SVM	0.575	0.493	0.634	0.815	0.534	0.608	0.437	0.532	0.579
KNN	0.463	0.457	0.557	0.729	0.457	0.547	0.428	0.515	0.519
Random Forest	0.552	0.536	0.644	0.791	0.525	0.599	0.474	0.559	0.585
Genre Vector	0.605	0.551	0.641	0.738	0.541	0.716	0.475	0.59	0.607
CNN	0.543	0.466	0.668	0.801	0.504	0.628	0.471	0.563	0.580
GRU	0.479	0.467	0.558	0.745	0.462	0.601	0.355	0.531	0.525
Bi-GRU	0.494	0.471	0.567	0.752	0.492	0.609	0.372	0.488	0.531

Table 2. Table 3: Results of Word Analogy tasks. Highest value for each task is in bold.

Tasks	Lyrics	Wikipedia	Sampled Wiki	Lyrics+Wiki
1) capital-common-countries	10.95	87.75	89.43	87.94
2) capital-world	07.26	90.35	79.25	90.00
3) currency	02.94	05.56	01.85	05.56
4) city-in-state	07.87	66.55	61.71	66.73
5) family	81.05	94.74	82.82	94.15
6) gram1-adjective-to-adverb	08.86	35.71	25.53	36.51
7) gram2-opposite	19.88	51.47	33.46	51.10
8) gram3-comparative	83.56	91.18	80.66	90.00
9) gram4-superlative	53.33	75.72	59.49	77.83
10) gram5-present-participle	74.32	73.33	60.82	75.81
11) gram6-nationality-adjective	06.29	97.01	92.60	97.08
12) gram7-past-tense	54.44	68.07	65.47	69.00
13) gram8-plural	73.19	87.60	76.81	89.52
14) gram9-plural-verbs	60.00	71.85	63.32	72.62
Overall Across All Tasks	50.33	75.71	66.6	78.11

Full text

1 Introduction

The dramatic growth in streaming music consumption in the past few years has fueled the research in music mining [2]. More than 85% of online music subscribers search for lyrics [1]. It indicates that lyrics are an important part of the musical experience. This work is motivated by the observation that lyrics are not yet used to their true potential for understanding music and language computationally. There are three main components to experiencing a song: visual through video, auditory though music, and linguistic through lyrics. As compared to video and audio components, lyrics have two main advantages when it comes to analyzing songs. First, the purpose of the song is mainly conveyed through the lyrics. Second, lyrics as a text data require far fewer resources to analyze computationally. In this paper, we focus on lyrics to demonstrate their value for two broad domains: music mining and NLP.

A recent trend in NLP is to move away from handcrafted features in favor of distributed representation. Methods such as word2vec and doc2vec have achieved tremendous success for various NLP tasks in conjunction with Deep Learning [14]. Given a song, we focus on two prediction tasks: genre and popularity. We apply distributed representation learning methods to jointly learn the representation of lyrics as well as genre & popularity labels. Using these learned vectors, we experiment with various traditional supervised machine learning and Deep Learning models. We apply the same methodology for popularity prediction. Please refer to Figures 2 and  2 for overview of our approach.

Our work has three research contributions. First, this is the first work that demonstrates the strength of distributed representation of lyrics for music mining and NLP tasks. Second, contrary to existing work, we show that lyrics alone can be good indicators of genre and popularity. Third, the quality of words vectors can be improved by capitalizing on knowledge encoded in lyrics.

2 Dataset

Lyrics are protected by copyright and cannot be shared directly. Most researchers in the past have used either small datasets that are manually curated or large datasets that represent lyrics as a bag of words [12, 11, 3, 13, 5, 7, 4]. Small datasets are not enough for training distributed representation. Bag of words representation lacks information about the order of words in lyrics. Such datasets cannot be used for training distributed representation. To get around this problem, we harvested lyrics from user-generated content on the Web. Our dataset contains around 400,000 songs in English. We had to do extensive preprocessing to remove text that is not part of lyrics. We also had to detect and remove duplicate lyrics. Metadata about lyrics that is genre and popularity was obtained from Fell and Sporleder [4]. However, for genre and popularity prediction, we were constrained to use only a subset of dataset due to class imbalance problem.

3 Genre Prediction

Our dataset contains songs from eight genres: Metal, Country, Religious, Rap, R&B, Reggae, Folk, and Blues. Our dataset had a severe problem of class imbalance with genres such as Rap dominating. Using complete dataset was resulting in prediction models that were highly biased towards the dominant classes. Hence, we use undersampling technique to generate balanced training and test datasets. We repeated this method to generate ten different versions of training and test datasets. Each version of dataset had about 8000 songs with about 1000 songs for each genre. Lyrics of each genre were randomly split into two partitions: 80% for training and 20% for testing. Experimental results reported here are average across these ten datasets. We did not observe any significant variance in results across different instances of training and test datasets, indicating the robustness of the results.

Distributed representation of lyrics and genres were jointly learned using doc2vec model [6]. This model gave eight genre vectors (a vector representation for each genre) and vector representation for each song in the training and test dataset. We experimented with vector dimensionality and found 300 as the optimal dimensionality for our task. Using this vector representation, we experimented with both traditional machine learning models ( SVM, KNN, Random Forest and Genre Vectors) and deep learning models (CNN, GRU, and Bidirectional GRU) for genre prediction task. Please refer to Table  1. For the KNN model, the genre of a test instance was determined based on genres of K nearest neighbors in the training dataset. Nearest neighbors were determined using cosine similarity. We tried three parameter values for K: 10, 25, and 50. However, there was no significant difference in results. For Genre Vector model, the genre of a test instance was determined based on the cosine similarity of test instance with vectors obtained for each genre. We can observe that Rap is the easiest genre to predict as rap songs have a distinctive vocabulary. The Folk genre is the most difficult to identify. For each genre, the worst performing model is the KNN, indicating that the local neighborhood of a test instance is not the best indicator of the genre. On average, Genre Vector model performs the best.

Please refer to Figure 3. This figure represents confusion matrix for one version of training dataset using Genre Vector model. Each row of the matrix sums to around 200 as the number of instances in test dataset per genre were around 200. We can notice that confusion relationships are asymmetric. We say that a genre X is confused with genre Y if the genre prediction model identifies many songs of genre X as having genre Y. For example, observe the row corresponding to the Folk genre. It is mainly confused with the Religious genre as about 16% of Folk songs are identified as Religious. However for the Religious genre, Folk does not appear as one of the top confused genres. Similarly, genre Reggae is most confused with R&B. However, R&B is least confused with Reggae.

4 Popularity Prediction

Only a subset of songs had user ratings data available with ratings ranging from 1 to 5 [4]. For two genres: Folk and Blues, we did not get popularity data for enough number of songs. For popularity prediction task, the number of genres was thus reduced to six. Number of songs per genre are: Metal(15254), Country(2640), Religious(3296), Rap(19774), R&B(6144), and Reggae(294). Songs of each genre were randomly partitioned into two disjoint sets: 80% for training and 20% for testing. To ensure robustness of results, we performed experiments on ten such versions of the dataset. Experimental results reported here are average across ten runs. We model popularity prediction as a binary classification problem. For each genre, we divided songs into two categories: low popularity (rated 1, 2, or 3) and high popularity(rated 4 or 5). The number of songs included in each class were balanced to avoid any over fitting of model.

Considering the distinctive nature of each genre, we built a separate model per genre for popularity prediction. For each genre using the doc2vec model, we generated two popularity vectors (one each for low and high popularity) and vector representation for each song in training and testing dataset. Similar to the genre prediction task, we experimented with seven prediction models. Please refer to Table 2. We can observe that Deep Learning based models perform better than other models. However, for every genre, the gap between the best and worst model has narrowed down as compared to the genre prediction task.

5 Improving Word Vectors with Lyrics

A large text corpus such as Wikipedia is necessary to train distributed representation of words. Lyrics are a poetic creation that requires significant creativity. Knowledge encoded in them can be utilized by training distributed representation of words. For this task, we used our entire dataset of 400K songs. Using the word2vec model, we generated four sets of word vectors. The four training datasets were: Lyrics only ($D1$, 470 MB), Complete Wikipedia ($D2$, 13 GB), Sampled Wikipedia ($D3$, 470 MB), and Lyrics combined with Wikipedia ($D4$, 13.47 GB). For dataset $D3$, we randomly sampled pages from Wikipedia till we collected dataset of a size comparable to our Lyrics dataset. For dataset $D3$, we created ten such sampled versions of Wikipedia. Results given here for $D3$ are average across ten such datasets.

To compare these four sets of word embeddings, we used 14 tasks of word analogy tests proposed by Mikolov [8]. Please refer to Table 3. Each cell in the table represents accuracy (in percentage) of a particular word vector set for a particular word analogy task. First five tasks in the table consist of finding a related pair of words. These can be grouped as semantic tests. Next nine tasks (6 to 14) check syntactic properties of word vectors using various grammar related tests. These can be grouped as syntactic tests.

By sheer size, we expect $D2$ to beat our dataset $D1$. However, we can observe that for tasks 5, 8, 12, 13, and 14 $D1$ gives results comparable to $D2$. For task 10, $D1$ is able to beat $D2$ despite the significant size difference. Datasets $D3$ and $D1$ are comparable in size. For task 10, $D1$ significantly outperforms $D3$. For all other tasks, the performance gap between $D3$ and $D1$ is reduced noticeably. We can observe that $D1$ performs better on syntactic tests than semantic tests. However, the main takeaway from this experiment is that dataset $D4$ performs the best for a majority of the tasks. Also, $D4$ is the best performing dataset overall. These results indicate that lyrics can be used in conjunction with large text corpus to further improve distributed representation of words.

6 Related Work

Existing works that have used lyrics for genre and popularity prediction can be partitioned into two categories. First, that use lyrics in augmentation with acoustic features of the song [7, 5] and second, that do not use acoustic features [3, 10, 13, 4]. However, all of them represent lyrics using either handcrafted features or bag-of-words models. Identifying features manually requires intricate knowledge of music, and such features vary with the underlying dataset. Mikolov and Le have shown that distributed representation of words and documents is superior to bag-of-words models [9, 6]. To the best of our knowledge, this is the first work that capitalizes on such representation of lyrics for genre and popularity prediction. However, our results cannot be directly compared with existing works as datasets, set of genres, the definition of popularity, and distribution of target classes are not identical. Still, our results stand in contrast with existing works that have concluded that lyrics alone are a weak indicator of genre and popularity. These works report significantly low performance of lyrics for genre prediction task. For example, Rauber et al. report an accuracy of 34% [10], Doraisamy et al. report an accuracy of 40% [13], McKay et al. report an accuracy of 43% [7], and Hu et al. reported accuracy of abysmal 19% [5]. The accuracy of our method is around 63%.

7 Conclusion and Future Work

This work has demonstrated that using distributed representation; lyrics can serve as a good indicator of genre and popularity. Lyrics can also be useful to improve distributed representation of words. Deep Learning based models can deliver better results if larger training datasets are available. Our method can be easily integrated with recent music mining algorithms that use an ensemble of lyrical, audio, and social features.