ABOUT

Sentiment Anomaly

src="nounproject"

Detecting emotions of the society through internet words

Second Place at Yonsei Big Data Conference 2020

Our emotions are easily reflected onto the words we write and speak. Detecting the anomaly in these words could mean a significant emotional change in the society.

src="nounproject"

Society Sentiment

Reading "Detection of the Change in Blogger Sentiment using Multivariate Control Charts" (2013) by Jeounghoon Moon and Sungim lee, our team member Soojin thought that the current prevalent emotions in the society can also be detected via the 'words of the people'.


Data Anomaly

Can we detect societal problems by detecting the sentiment anomaly from the internet such as the twitter posts? After collecting sentiment scores from blog posts, we predicted if the society at a time was going through the same.


Detecting anomaly from society

Data crawling, anomaly detection, sentiment analysis, data analysis, data visualization

Sentiment analysis, data visualization

Soojin Yoon: project lead, anomaly detection
Sungyoon Kim: data crawling, data analysis
Jihoon Kim: sentiment analysis, data processing

“Social network services generate a considerable amount of social data every day on personal feelings or thoughts. This social data provides changing patterns of information production and consumption but are also a tool that reflects social phenomenon.”

Can the everyday internet words reflect negative emotions of the society?

We sought to collect and analyze the blog articles of Naver, the biggest search engine in Korea, to figure out if the anomaly in their sentiments matched the societal issues and negative emotions at the time.


Data Collection

Using the Selenium crawler, we have collected 1,000 blog posts a day, saving all data into "crawling_naver_api.ipynb: savefile('keyword')". In total, we have crawled 357 days worth of blog posts and times it by 1,000 we have collected 357,000 blog data with the keyword = daily. This was to collect the daily posts like diaries, ones that people write about their typical days and daily life and not others.

Data Collection Web Crawling
Sentiment Analysis
KNU Sentiment dictionary via Bi-LSTM
Sentiment extraction
Sentiment Scoring
Trend visualization
Wordclouds
Anomaly Detection
COVID-19 Trend Comparison
Result analysis

Sentiment Analysis

Using the KNU Korean sentiment dictionary, we managed to dissect the words into negative and positive. This dictionary uses a Bi-LSTM model , that detects the polarity with 89.45% accuracy.

We then scored each of the words to very negtaive, negative, neutral, positive and very positive, and categorized them accordingly. Then, we plotted the scores of 357 blog posts to a scatter graph.

Sentiment Scoring

After crawling and processing the data we scored the sentiment of each word stemming from the blog post. 0 was neutral, -2 was very negative, -1 was negative, 1 was positive and 1 was very positive.

Date Organization

After extracting sentiment data, we sorted the scores according to the dates and looked for the trend.

Result Analysis

We have collected the one-year worth of COVID-19 case data separately. Then, we were able to detect a correlation between COVID-19 case reports and the low sentiment scores accordingly.

Blue: COVID-19 case reports Turquoise: Sentiment score

Sentiment Anomaly

Trend Analysis

It turns out the anomaly in graph represented the societal emotions according to the sentiment score dispersions, after analyzing and crawling the relevant news articles at the time. The negative trend in sentiment anomaly signified the negative societal atmosphere at the time, while positive trend signified there was a good turn in the news or in the society. See below.

Negative Sentiment Anomaly

On February 19th 2020, when COVID cases were breaking out, there were one of the lowest scores of sentiment of the year. Analyzing the news articles of the day, there was a massive rise of cases, the president was sentenced to prison, and worse.



Positive Sentiment Anomaly

On September 26th 2020, there was a skyrocketing score of sentiment, average (0.77). It turns out the COVID was being contained, the Korean society was seeming to handle the lockdown fairly well.

Conclusion

  • Analyzing and detecting sentiment anomaly.
    It as a whole showcased that analyzing and simply detecting the sentiment anomaly, whether positive or negative, can predict the kind of societial and envrironmental news of the day or in the near future.
  • Further Usage.
    We can further do the research on other media platforms, create sentiment dictionary or predict the upcoming social sentiments.
⇽Previous Project
Multisensory Packaging