Scraping the web

An increasing amount of data is available on the web:

• Election results, budget allocations, legislative speeches…
• Social media data, newspaper articles, press releases…
• Geographic information, weather forecasts…

These data are often provided in an unstructured format.

Web scraping is the process of extracting this information automatically and transforming it into a structured dataset.

Two different scenarios:

1. Screen scraping: extract data from source code of website, with html parser and/or regular expressions
   • rvest package
2. Web APIs (application programming interfaces): a set of structure http requests that return JSON or XML data
   • httr to construct API requests
   • packages specific to each API: weatherData, WDI, Rfacebook… Check CRAN Task View on Web Technologies and Services for more examples

1. Respect the hosting site's wishes:
   • First, check if an API exists or if data are available for download
   • Some websites disallow scrapers on their robots.txt files
2. Limit your bandwidth use:
   • Wait one or two seconds after each hit
   • Scrape only what you need, and just once (e.g. store the html file in disk, and then parse it)
3. When using APIs, read documentation
   • Is there a batch download option?
   • Are there any rate limits?
   • Can you share the data?

Workflow:

1. Learn about structure of website
2. Build prototype code
3. Generalize: functions, loops, debugging
4. Data cleaning

1. Data in table format

1. Data in table format
   • Automatic extraction with rvest
2. Data in unstructured format
   • Element identification with selectorGadget
   • Aumatic extraction with rvest
3. Data hidden behind web forms
   • Automation of web browser behavior with selenium

API = Application Programming Interface; a set of structured https requests that return data in JSON or XML format.

Types of APIs:
1. RESTful APIs: queries for static information at current moment (e.g. user profiles, posts, etc.)
2. Streaming APIs: changes in users' data in real time (e.g. new tweets, new FB posts…)

Most APIs are rate-limited:

• Restrictions on number of API calls by user/IP address and period of time.

Constructing a REST API call:

• Baseline URL: https://graph.facebook.com/
• Parameters: ?ids=barackobama,johnmccain
• Authentication token: &access_token=XXXXX

Response is often in JSON Format.

Authentication

• Most common is an open standard called OAuth
  
• Connections without sharing username or password, only temporary tokens that can be refreshed
  
• httr package in R implements most cases (examples)

Twitter:

1. REST API:
   • Data about user profiles, friends and followers, and recent tweets
   • R packages: twitteR or smappR
2. Streaming API:
   • “Stream” of tweets as they are being published
   • Filter by: 1) by keywords, 2) by location, 3) random sampling
   • R package: streamR

Facebook:

• Graph API: data from public Facebook pages (posts, likes, comments)
• R package: Rfacebook

CRAN Web Technologies Task View