Introduction to Internet Measurement: What, Where, Why and How

Nina Taft
Technicolor Research Palo Alto

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Why measure the internet? (1/3)

Cool things to learn

Who wants to know? - business - commercial

Slide source: http://techcrunch.com/2011/05/17/netflix-largest-internet-traffic/

Why measure the internet (2/3)

Just to understand
Why wont the web page download ? Why is my netflix movie so slow ?

Who wants to know? - users !!

Why measure the internet ? (3/3)

Planning for the future
Cisco predictions for 2016:
Global IP traffic will reach 1.3 zettabytes per year The gigabyte equivalent of all movies ever made will cross global IP networks every 3 minutes. Number of devices connected to IP networks will be nearly three times as high as the global population

Who wants to know? - service providers

How is the Internet structured?

How to connect thousands of access service providers, and enterprises?
access net access net access net access net access net enterprise net enterprise net

access net

enterprise net

access net

enterprise net

access net access net access net

enterprise net access net

Slide source: http://www-net.cs.umass.edu/kurose-ross-ppt-6e/

Internet structure
Access Internet Service Providers (ISP) connect to a global transit ISP. Customer and provider ISPs have economic agreement.
access net access net access net access net enterprise net enterprise net access net

ISP A

access net

ISP B
ISP C

enterprise net

access net

enterprise net access net access net access net enterprise net access net

Slide source: http://www-net.cs.umass.edu/kurose-ross-ppt-6e/

Internet structure
Global ISPs must be interconnected
access net access net access net access net enterprise net enterprise net access net

ISP A
enterprise net

access net

ISP B
ISP C
access net access net

access net

peering link
enterprise net access net access net

enterprise net

Slide source: http://www-net.cs.umass.edu/kurose-ross-ppt-6e/

Internet structure
Content provider networks (e.g., Google, Microsoft, Akamai) may run their own network,
access net access net access net access net enterprise net access net

enterprise net

ISP A

Content provider network

access net

ISP B
ISP C

enterprise net

access net

enterprise net access net access net access net enterprise net access net

Slide source: http://www-net.cs.umass.edu/kurose-ross-ppt-6e/

Internet structure: network of networks

Tier 1 ISP

Tier 1 ISP

Google

Regional ISP

Regional ISP

access ISP

access ISP

access ISP

access ISP

access ISP

access ISP

access ISP

access ISP

Slide source: http://www-net.cs.umass.edu/kurose-ross-ppt-6e/

Communication between service providers

Tier-1 ISP
BGP

Tier-1 ISP

Tier-2 ISP

dedicated access ISP

AS (Autonomous System) - a collection of routers under the same technical and administrative domain. BGP (Border Gateway Protocol) - used between two ASs to allow them to exchange routing information so that traffic can be forwarded across AS borders.

A word about BGP Purpose: to share connectivity information

you can reach destinations in address range A via me

AS2 R3
Customer with address range A

AS1 traffic to A

BGP
R2

R1

table at R1: dest next hop A R2

border router internal router

Network structure within one ISP

ISP 3 ISP 2
peering link
border router

border router

ISP 1

access router

access router

customer

access link

customer

So the internet structure is super complicated!

What should we measure? Where should we measure? How do we capture data? It all depends upon what you want to do with the data !

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Where measure ? And why ? (1/5)

ISP 3 ISP 2
Example 1:
peering link
border router

border router

What? Ingress traffic at access router.

ISP 1

Why? ISPs can see if their customer traffic is growing, or if there is unwanted traffic
access router

access router

customer

access link

customer

Where measure ? And why ? (2/5)

ISP 3 ISP 2
Example 2:
peering link
border router border router

What? Aggregate traffic inside backbone traffic.

ISP 1

Why? ISPs need to know when to upgrade a router, or add a new one; when to upgrade link capacity.
access router

access router

customer

access link

customer

Where measure ? And why ? (3/5)

ISP 3 ISP 2
peering link
border router

border router

Example 3: What ? Failures Why? To know if recover well from outages, and how often they occur

ISP 1

access router

access router

customer

access link

customer

Where measure ? And why ? (4/5)

ISP 3
peering link

ISP 2 Example 4: Where ? Peering links What ? BGP routing announcements ISP 1 Why ? Learn who is connected to whom
access router access router

border router

border router

customer

access link

customer

Why ? Make sure no routing loops

Where measure ? And why ? (5/5)

CDN
peering link

ISP 2 Example 5: Where ? In the home Why ? Understand performance of a video streaming
access router

border router

border router

ISP 1

access router

customer

access link

Measurement is hard! (1/2)

Ecosystem of distributed semi-autonomous systems no one sees it all ! Even those who regulate it are confused: The Internet .. is not a big truck, it is a series of tubes (former-US senator) Data ownership and privacy

Measurement is hard! (2/2)

Representativeness:

Did you get enough samples of routers? clients? servers? paths? to make general claim about internet ?
Many things hard to do without access to a large infrastructure. Researchers have created distributed measurement platforms.

Uncertainty principle observer effect

measurements of certain systems cannot be made without affecting the systems

Getting accurate timestamps can be challenging

Two broad categories of measurement techniques

Examples of some well known tools (there are many others) How
Passive Methods pcap: packet capture Netflow: flow capture SNMP: per-link data packet-pair probes ping routeviews: capture BGP routing table views netflow telescope traceroute, ping Active Methods

What
Traffic

Performance Topology Security

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Why do we want to understand traffic?

Link Provisioning

Diurnal traffic patterns

can my network accommodate bursts and peaks? provisioning depends upon aggregate traffic mix delay & loss performance (for example)

Compliance with Service-Level-Agreements SLAs

Long Term Traffic Trends

Application mix constantly evolving

Identify abnormalities

Failures
Security

Passive Measurement Tools

For Traffic Measurement

Approach 1: capture packets Example Tools: Libpcap: for general purpose systems OC192MON: specialized hardware for backbone links (~10Gbps) Analysis tools: tcpdump & Wireshark Approach 2: capture flows Flow commonly defined as 5-tuple:
(srcIPaddr, srcPort, dstIPaddr, dstPort, protocol-id)

Example Tools: Ciscos Netflow, Junipers cflowd Typical data recorded: start-time, end-time, number of bytes, number of packets Reduced trace size compared to packet capture

Wireshark

packet analyzer

Application (browser, email)

pcap

copy packets

Transport (TCP/UDP) Network (IP) Link (Ethernet)

slide from Kurose & Ross Computer Networking, Ch. 1 (online demo)

WIRESHARK DEMO

Traffic Trends

Example of backbone traffic in Tier1 ISP Average growing over time Variability is growing over time Spikes present throughout

Traffic Patterns

Mbps

time

Timescale: zoom in to one month of previous trace We see strong diurnal trends Clear periodicities at 24 and 12 hour periods Weekdays vs. weekends

Recap on Traffic
Who and Why?

ISPs collect packets and flows inside their ISPs to better manage their network, and to be ready for the future. pcap can be put anywhere an end user can do it from their home; hows my traffic composition changing? Enterprise networks also need to understand corporate traffic

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Performance Measurement
What are bandwidth measurements useful for?

Applications that adapt their rate accordingly (video streaming) Verifying SLA (service level agreements) and more Usually measures the latency of a path

What are latency measurements useful for?

Any app that involves choosing among different paths to deliver data: CDNs, P2P, multiuser games,

Measuring such things often involves a measurement process at multiple places in the network

Example: measure the bottleneck bandwidth of a path

link1 link2 link3

Packets of equal size transmitted back-to-back onto a path Slowest link creates fixed delay between start of pkt1 and pkt2 When packets leave narrow link, fixed delay D is preserved

pkt len D capacity

Image source: courtesy of Mark Crovella @ Boston University

pkt1 pkt2

pkt len capacity D

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Topology
A way to visualize the interconnection patterns of Internet components

AS graph

Interconnection pattern of ASes : each node is an AS, and each edge are peering links (or AS exchange points). Each node is a router, and each edge is a one-hop link between 2 routers. Can annotate edges with link capacity.

Router-level graph

(there are others too, but these 2 are common)

Toy Example of an AS graph

7018 AT&T

4809 China telecom

4721 Japan cable

12295 London Link

1239 Sprint

3215 France Telecom

9329 SriLanka telco

7922 Comcast

22390 Xerox

12576 Orange

What does a slice of the AS graph look like?

These are various incoming paths to reach Sprint

Graph source: http://as.robtex.com/

What does the whole AS graph look like?

We cant really be sure, but heres a large chunk of it:

What does a router level topology look like? For one ISP (simple version):

slide from Kurose & Ross Computer Networking, Ch. 1 (online slides)

How to build a these topologies

Need large-scale infrastructures, with monitors all over the world, to obtain needed data
Each location with monitoring equipment called vantage point

Slide source: http://www.monkey.org/~labovit/

How to build a these topologies

Two examples of large-scale infrastructures to obtain needed data
Approach #1: Archipeligo

Combines 2 different data types: traceroute data and BGP routing tables Have ~70 monitors worldwide

Approach #2: Dimes

Built a worldwide community of thousands of users who installed host software (Dimes agent) to run traceroutes and pings (at low rates) from their hosts. Data collected centrally for analysis.

Traceroute data has ambiguities in it. There was much debate over a number of years on the best way to produce these topologies. After settling on a model many researchers liked, the Internet changed again

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Network telescopes (1/2)

Refers to a set of the IPv4 address space that is not used for standard network connectivity; it contains no legitimate hosts. Idea: all inbound traffic to such addresses must be anomalous, so collect it!

Route packets with those addresses to a collection point.

Useful for security:

To detect scanning behavior by worms, and DoS attacks on web servers.

Started using them in 2001. In 2003 Slammer worm released.

Network Telescope (2/2)

First ones appears in 2001. In 2003 Slammer worm released.

Reached very high scanning rate in 3 minutes. Overwhelmed many servers, some disabled, slowed down links

Slammer studied using telescope data, and predictions made: 1) could target small populations; 2) spreading speed could increase.

Both of these predictions were realized by the Witty worm in 2004.

Outline
Motivation & Ecosystem Quick examples of where and why

Areas
traffic performance

topology
security traffic matrices Summary

Whats a traffic matrix?

destination origin City A City B City C

Usually contains average values

City A
City B City C 250 Mbps

Can define variety of matrices

Select timescale City-to-city, or router-to-router

Traditional Traffic Matrix

router-to-router topology:
traffic source: traffic sink

Describes traffic demands for all nodepairs: end-to-end flows within a single network (e.g., AS domain)

ingress node:

egress node:

Example Problem

A 5 3

How much traffic flows between origin-destination pairs? A->D A->C B->C B->D

D SNMP byte counts per link

Example: One Solution

A 5 3

1 C

4 3

4 4

0 D

How much traffic flows between? A->D: 4 A->C: 1 B->C: 3 B->D: 0

Example: Another Solution

A 5 3

3 C

4 1

4 2

2 D

How much traffic flows between? A->D: 2 A->C: 3 B->C: 1 B->D: 2

type of equations:

Link1 = XAD + XBD

Notation: Problem Formulation

routing matrix Link1 Link2 Link3 . Link L

0 1 1 0 0 0 0 0 1 0 0 . .

XAB XAC XAD . . .

Have linear system:

Y=AX

Problem Statement

System: Y = AX We have Y from SNMP link measurements We know A from OSPF link weights (so we can compute shortest paths) problem: find X # origin-destination (OD) pa issue: # links < < # OD pairs => underconstrained system => infinite # of solutions
# links

What does city-to-city traffic look like?

x 10 6
7

Sample OD Flow. 2 weeks in summer 03

5.5

4.5

Bps
4 3.5 3 2.5 0

200

400

600

800 1000 1200 Time in 10-minute units

1400

1600

1800

Summary
What Traffic How packet or flow capture active probes Who ISPs Why
network planning

everyone
CDNs users ISPs

traffic evolution
web download performance Fairness delivering promised performance

Performance

Topology

many vantage points

service providers everyone everyone

vulnerabilities connectedness (isnt that obvious!)

Security

network telescopes, deep packet inspection

Whats Next: The only constant is change

New devices
Web-enabled toaster + weather forecaster

Internet is flattening
Google Comcast
Content & search providers building their own infrastructure

Akamai

New apps, new OSN uses

Tweet-a-watt: monitor energy use Internet refrigerator

Acknowledgements
Book Internet Measurement by Prof. Mark Crovella (Boston University) and Bala Krishnamurthy (AT&T Research) Some slideware thanks to Prof. Jim Kurose (University of Massachusetts at Amherst) online course material from book Computer Networking: Top-Down Approach by Kurose and Ross

Thank You

Questions ?