The latter are slightly more negative, but overall the similarity score of the two bands was 0.99. It turned out that Iron Maiden have a richer vocabulary and use more long words than Metallica. Specifically, I looks for similarities and differences between the two bands. Just for fun, I also used NLTK, SpaCy, and VADER to explore the lyrics. Text analysis of Metallica and Iron Maiden lyrics My model achieved an accuracy of 96.7% on the train set and 68.4% on the test set. For this, the words are vectorized or transformed into numbers. MNBC is typically used in text classification for calculating the probability of a word occurring in a text. I used the Multinomial Naive Bayes Classifier (MNBC), a probabilistic model based on Bayes’ Theorem, which assumes strong independence between the features and uses a multinomial distribution for each of the features. Now that that the dataset was made of clean lyrics, it was ready to train a classification model. This way, I ended up with 86 songs by each band. I went with under-sampling and, after dropping the duplicates, I randomly removed 15 more songs by Metallica. In my case, I had more lyrics by Metallica than by Iron Maiden. weighting: assign higher weights to observation from the minority class during training.SMOTE: generate new samples by interpolation.under-sampling: reduce the number of samples in the majority class.over-sampling: collect more samples for the minority class.There are four main ways to deal with class imbalance: Class imbalance can skew the prediction in the way that the model would predict the majority. if and artist had far more observations than another. But remember:Īnother important part of text processing was to check for class imbalance, i.e. This step was quite easy, mainly because the lyrics were in English. filling words or words that don’t change the meaning of a sentence, like the, a, to). I used TextHero and SpaCy to tokenize the text, make the words lowercase, remove punctuation, numbers, and stop words (i.e. Next, I had to clean and preprocess the scraped lyrics, in order to feed them to the classification model.
#Webscraper python lyrics code#
I found this to be the most difficult and frustrating part, from finding a good lyrics website that doesn’t contain tens of duplicate lyrics, to implementing BeautifulSoup, and transforming the code from a JupyterNotebook into a Python file. The first step in this project was to build a web-scraper in Python with BeautifulSoup. I was super excited about this week, because it was about language models and first steps into NLP, my favorite ML topic! The challenge was to create a Python program that scrapes lyrics from a website, preprocesses them, and predicts the artist from the text. Project completed in week 4 (19.10.-23.10.20) of the Data Science Bootcamp at Spiced Academy in Berlin.
0 kommentar(er)
