There’s Always


an API
Sometimes they make you work for it
Hi, I’m Matt Dennewitz

• VP Product, Pitchfork; Dir. Engineering, Wired

• I consult on data for baseball writers and


MLB clubs

• @mattdennewitz on Twitter, Github

 Agenda
• 101

• Your first scrape: Google Docs

• Interlude: HTML, JSON, XML, XPath

• Scaling up: Python

• What happens when the data isn’t on the page?

• Advanced topics (time allowing)

 What is scraping?
• Extracting information from a document

• Rows from an HTML table

• Text from a PDF

• Images from Craigslist posts or museum websites

• OCR’ing an image and reading its text

• Spidering a website like Google

 Tools

• Google Docs (surprise!)

• Chrome Developer Tools

• Python

• Scrapy
 Strategy

1. “What do I want?”

2. Case the joint

3. Rob it just a little bit

4. Move in
 “What do I want?”
• Envision the data you want, how you need it

• “How will I scrape this data?” Script? Crawler?

• “Do I need to scrape this more than once?”

• “How do I need to shape the data?”

• “What do I need to do to the data after I have it?”

Clean, verify, cross-link with another data set, …?

• “How/to where do I want to output the data?”

 Case the joint
• Does the document seem scrape-ready? Does
access come with preconditions?

• Preconditions: password-protected? Online-only?

Needs a special decoder?

• Look at how the data is presented in the

document. Are there external dependencies, or
is it self-contained?

• External deps: more information on secondary

pages, data in other spreadsheets or workbooks
 Rob it just a little bit
• Prototype using a subset of the information

• Estimate how long scraping will take, determine

imperative needs like throttling or a specific OS

• Validate your ideas about the data you wish


to extract, correct bugs

• Writing unit tests

 Oceans 1101

• You’ve created a stable scraper which emits data

in the format you want (CSV, JSON, XML, SQL, …)
to the location you want

• You understand its performance characteristics

• Go!
 Interlude: formats
• Data is distributed in mercilessly innumerable
formats

• The Big Three of web scraping

• HTML

• JSON

• XML
 Formats: XML

• eXtensible Markup Language

• Well-structured, self-validating, predictable

• Pedantic, though not with its charms

Formats: XML
 Formats: HTML
• Hypertext something something something

• XML-like, without the upside

• Needs stronger class of parser to heal broken

code

• Less predictable, far more susceptible to

changes in the wind
 Formats: HTML
<p>

1.

<strong><span class="playerdef"><a href="http://

www.baseballprospectus.com/card/card.php?id=102123">Alex Reyes</
a></span>, RHP, <span class="teamdef"><a href="http://
www.baseballprospectus.com/team_audit.php?team=SLN"
target="blank">St. Louis Cardinals</a></span></strong><br>

Scouting Report: <a href=“http://www.baseballprospectus.com/

article.php?articleid=30958">LINK</a>

</p>
 Formats: JSON
• JavaScript Object Notation

• Data objects with simple primitives: int, double,

string, boolean, object (key/value pairs), array
(untyped), null.

• Requires waaaaaaay less parsing, much easier to

serialize

• No schemas, but validation tools exist

• Has taken over for XML in web data transmission

 Formats: JSON
{

"prospect_year": "2017",

"player_id": 643217,

"player_first_name": "Andrew",

"player_last_name": "Benintendi",

"rank": 1,

"position": "OF",

"preseason100": 1,

"preseason20": 1,

"team_file_code": "BOS",

}
 Bonus: XPath

• XPath is a way to query XML (and HTML)

• It’s got a super goofy syntax

• Very powerful, essential for scraping the web

 Bonus: XPath
• XPath: //table/tbody/tr

• HTML (fragment): 

<table>

<thead>

<tr>

<th>Name</th><th>HR</th><th>SB</th>

</tr>

</thead>

<tbody>

<tr><td>Mike Trout</td><td>40</td><td>40</td></tr>

</tbody>

</table>

• Result: <tr><td>Mike Trout</td><td>40</td><td>40</td></tr>

 Bonus: XPath

• XPath: //span[@class=“playerdef”]/text()

• HTML: 

<p>1. <strong><span class=“playerdef”>Eloy
Jiminez</span></strong>, OF, …</p>

• Result: “Eloy Jiminez”

Ok, time to scrape
 Google Docs

• Fire up Google Docs, start a new spreadsheet

• IMPORTXML and IMPORTHTML are your friends

• Let’s look at IMPORTHTML

 IMPORTHTML

• Allows you put pull in specific list or tabular data

from a web page

• Syntax:

=IMPORTHTML(url, <“list” or “table”>,
[index])
 IMPORTHTML
• ESPN Home Run Tracker

• Syntax:

=importhtml("http://
www.hittrackeronline.com/?perpage=1000",
"table", 17)

• “Give me the 16th table on the page” (0-based

indexing)
IMPORTHTML
 IMPORTHTML
• Brooks Baseball Player Pitch Logs

• Syntax:

=IMPORTHTML("http://
www.brooksbaseball.net/pfxVB/
tabdel_expanded.php?
pitchSel=458584&game=gid_2016_06_27_bosm
lb_tbamlb_1/
&s_type=&h_size=700&v_size=500",
"table")
IMPORTHTML
 Google Docs

• Useful for pulling in single tables, or keeping

everything in a spreadsheet

• Data doesn’t always exist in a single place

• Spread across several pages

• Spread across several files or APIs

 Google Docs
• Useful for pulling in single tables, or keeping
everything in a spreadsheet

• Data doesn’t always exist in a single place

• Spread across several pages

• Spread across several files or APIs

• Automate as much as you can

 Python time

• Beautiful language. Transcendental even.

• Robust ecosystem for handling data parsing,

cleaning, making net requests, etc

• A+ community

• Runs anywhere
 Python time

• I’m going to use two non-standard packages

today:

• lxml, for HTML parsing and cleaning

• requests, for HTTP fetching

 Strategy (again)

1. “What do I want?”

2. Case the joint

3. Rob it just a little bit

4. Move in
 Strategy (again)

1. “What do I want?”: prospect rankings from BP,

MLB, Baseball America

2. Case the joint

3. Rob it just a little bit

4. Move in
 Strategy (again)
1. “What do I want?”: prospect rankings from BP,
MLB

2. Case the joint: BP has dirty HTML. MLB loads a

JSON file.

3. Rob it just a little bit

4. Move in
 Strategy (again)
1. “What do I want?”: prospect rankings from BP,
MLB

2. Case the joint: BP has dirty HTML. MLB loads a

JSON file.

3. Rob it just a little bit: Get a feel for BP and BA’s

HTML structure, examine MLB’s JSON file.

4. Move in
 Strategy (again)
1. “What do I want?”: prospect rankings from BP,
MLB.

2. Case the joint: BP has dirty HTML. MLB loads a

JSON file.

3. Rob it just a little bit: Get a feel for BP and BA’s

HTML structure, examine MLB’s JSON file.

4. Move in: Write three scripts, one for each.

 Strategy (again)

• Fields to export:

• Name

• Rank

• List type (“BP”, “MLB”, …)

• System ID (MLBAM ID, BP player ID, …)

 BP

• http://www.baseballprospectus.com/article.php?
articleid=31160
 BP

• First thing to do is inspect the source

• Is there a pattern in the HTML you can engineer for,

or an attribute you can target?

• Let’s head to the console! Right click on the one

of the capsules, and click “Inspect”
BP
 BP
• Yes! Player data is in a paragraph tag, <p>, which
contains a <span> with class “playerdef”

• Get used to talking like this

• Using XPath, we can target that <span> and walk

up to its parent element, <p>, which gives us
access to the whole player capsule
 BP
• Beware: the “playerdef” class could be used anywhere.
We need to find a reasonable scope for our XPath.

• Luckily for us, player capsules are in a <div> with class

“article”, and that structure appears only once per
article page across BP.

• XPath: //div[@class=“article”]//
span[@class=“playerdef"]/..

• What else?
 BP

• Code: https://github.com/mattdennewitz/
sloan-scraping/blob/master/bp-
top-101-2017.py

• Output: https://github.com/mattdennewitz/
sloan-scraping/blob/master/bp-2017.csv
BP
 BP

• What did we do?

• Inspected the page

• Found critical path to data, wrote supporting

XPaths

• Scripted collecting and outputting the data

 MLB

• http://m.mlb.com/prospects/2017
 MLB

• Again, start by inspecting the source

• Try to find “Benintendi” or “Moncada”

•
 MLB

• Again, start by inspecting the source

• Try to find “Benintendi” or “Moncada” in


the HTML

• “uhh”
 MLB
• Websites love to load data asynchronously.

• LOVE to

• Let’s head to the Inspector’s Network panel to

poke around and find the source

• In Chrome: Ctrl+Shift+I (Windows) or Cmd+Opt+I

(Mac), then select “Network”
 MLB
• Websites love to load data asynchronously.

• LOVE to

• Let’s head to the Inspector’s Network panel to

poke around and find the source

• In Chrome: Ctrl+Shift+I (Windows) or Cmd+Opt+I

(Mac), then select “Network”

• Let’s start by looking under “XHR”, the typical

place to look for dynamically loaded data
MLB
 MLB

• “playerProspects.json” looks promising

• We know it’s a JSON file

• The filename is a pretty dead giveaway

• When we open it up, it has a ton of prospect data

 MLB

• Here, we have a JSON file

• Let’s inspect the structure to find exactly what

attributes we would like to scrape

• Fast-forward: the “prospect_players” key has

prospects for all teams! And it has the Top 100
under the “prospects” key.
 MLB
{

"prospect_year": "2017",

"player_id": 643217,

"player_first_name": "Andrew",

"player_last_name": "Benintendi",

"rank": 1,

"position": "OF",

"preseason100": 1,

"preseason20": 1,

"team_file_code": "BOS",

}
 MLB
• Using Python’s out-of-box JSON parser, we can
easily parse this file and extract players

• Code: https://github.com/mattdennewitz/
sloan-scraping/blob/master/mlb-
top-100-2017.py

• Output: https://github.com/mattdennewitz/
sloan-scraping/blob/master/mlb-2017.csv
MLB
 Recap
• We’ve used the four step approach to plan for
consistent output across disparate systems

• We’ve used tools like the Inspector to probe for

data

• We’ve written very simple yet powerful scripts in

Python to download prospect lists

• We’ve streamlined the data into a consistent shape

• Our scripts are easily reusable

 Next steps
• Since we were clever and included system IDs,
we can tie it all together using a baseball player
ID registry

• Chadwick Register

• Smart Fantasy Baseball

• Crunchtime
 Tools

• Hopefully there’s time to talk about this

 Tools
• requests: A beautiful HTTP library

• lxml: A beautiful XML and HTML parsing library.

Tricky to install on Windows, binaries are
available.

• BeautifulSoup: another A+ HTML parser

• Scrapy: a very robust Python framework for

crawling websites
 Code

• The code and output from this session is online

at: https://github.com/mattdennewitz/2017-
sloan-data-scraping
 Thanks!

• Questions?

• If we have some time left, we could try a bit of

live coding

• If you have very specific scraping questions, find

me after and let’s talk