david melul martin hadid itc

Using Data Mining for #Bitcoin Sentiment Analysis

Abstract 

Twitter is a social media platform where users can voice their opinions through short messages called tweets. This research project focuses on using web scraping to gather and store tweets analyzing how changes in Bitcoin’s price affect peoples’ views on the cryptocurrency. For this, we scrapped Twitter’s main feed, used various APIs to attribute data, and determined tweet’s sentiments with analysis tools such as TextBlob. These sentiments were then contrasted to Bitcoin’s price fluctuations to identify trends and insights.

Challenges

  • Understanding the nature of Twitter’s UI, a complex JavaScript generated application;
  • Using Selenium web driver along with BeautifulSoup to parse the HTML and translate it to Python objects;
  • Overcoming twitter.com http request limits through the same IP using a VPN;
  • Avoiding rate limit errors from Tweepy (Twitter API) using a wait_on_rate_limit flag on the API’s request parameters; and
  • Setting up a headless web driver that runs on memory (with no UI) on AWS.

Achievements

  • Created a reliable application that seamlessly scraped Twitter’s news feed, users feeds and collected Bitcoin pricing data;
  • Set up a MySQL database with a scalable architecture, supporting large quantities of entries as well as dynamic entities;
  • Deployed an application on an AWS machine and connected it to a dockerized redash instance;
  • Collected over 1 million records in under 12 hours of scraping; and
  • Performed sentiment analysis on over 500k gathered tweets. 

 

Roadmap

  • Create a model to predict Bitcoin’s price;
  • Learn how to differentiate between bot-generated and human-generated tweets;
  • Introduce Twitter conversation chains into the data model; and
  • Separate facts from opinions using Natural Language Processing.

 

Link to GitHub: https://github.com/martinhadid/twitter-scraper

 

Authors

Martin Hadid and David Melul Fresco

Would you like to apply for our Data Science Fellows, Cyber Security Fellows or Full-Stack Development program – apply here to receive more information:

I agree to receive information from Israel Tech Challenge

I agree to receive information from Israel Tech Challenge

Share this post

Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on pinterest
Share on email