BRKARC-2003

Cisco ASR 9000 System

Architecture

Yongzhong Peng
Manager, Technical Marketing
SP Routing Infrastructure
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKARC-2003

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• ASR9000 Products Introduction
• ASR9000 System Hardware Architecture
• ASR9000 Distributed Control Plane
• ASR9000 Data Packet Processing
• ASR9000 QoS Architecture
• IOS-XR & IOS-XR 64 Bit
• Conclusion
ASR 9000 Products Introduction
Cisco ASR 9000 System Comprehensive Portfolio
Compact & Powerful High Density Service Edge
Flexible Service Edge
Access/Aggregation and Core
• Small footprint with full IOS- • Optimized for ESE and MSE with high • Scalable, ultra high density service
XR for distributed M-D scale for medium to large sites routers for large, high-growth sites
environments
One Platform, One OS, One Family
ASR 9922
nV Satellites
ASR 9000v, NCS5000 ASR 9912

ASR 9010 ASR 9910

ASR 9906
ASR 9901
ASR 9006
ASR 9904
ASR 9001

Fixed 2RU 2 LC/6RU 4 LC/10RU 4 LC/14RU 8 LC/21RU 8 LC/21RU 10 LC/30RU 20 LC/44RU

240 Gbps 16 Tbps 7 Tbps 32 Tbps 14 Tbps 64 Tbps 80 Tbps 160 Tbps

MSE E-MSE Peering P/PE CE Mobility Broadband

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
ASR 9901 Highlights

Form Factor & BW Ports / Port Density

• 2 RU box with 2 Tomahawk NPU • Fixed ports available; no MPAs
(ASR 9001 is 2 RU with 2 • 42 ports on the faceplate : 16X1G,
Typhoons) 24X1/10G, 2X100G(QSFP28)
• Depth of ~23 inches,(9001 is 18”) • 1G ports : LAN & MACSEC
• 456G Duplex BW 10G ports: LAN & MACSEC
(9001 is 120G Duplex) 100G port : LAN & MACSEC

SW & Licensing Mechanicals & Commons

• Redundant Power & Fan-trays
• 64 bit XR only
• Front to back Airflow
• PAYG mode for 120G,240G,360G
and 456G • NEBS, EMC Compliant
• All ports/power cabling on front
• Full feature parity with Tomahawk plate; fan trays on backside
feature-set
• Typical Power Consumed : 1200W

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
ASR 9901 Port mapping

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Cisco ASR 9906 Overview

 Scalable for Dense 100GE

Applications for future investment 2 Fan Trays
protection
4 Line Card
Slots
 Greater Capacity, Greater Flexibility:
2 RSPs
Start with 2 RSPs, then add fabric
cards to scale beyond 460G and 5 Fabric
Cards
enable N+1 fabric redundancy
1 Power
Tray
 Mid-Plane design allows for compact
chassis footprint

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
RSP880-LT

• RSP880-LT is the 3rd generation route processor card RSP880-LT provides is a cost effective migration
• 880Gbps/slot throughput (redundant configuration) path for RSP440 customers.
• 4 Core Intel Broadwell1.8GHz processor
• Available in both TR (16GB)/ SE (32GB) variants
• Line Cards Supported
• Tomahawk Hardware Differences
• Typhoon
• VSM
RSP880 RSP880-LT
• SIP-700
• Chassis Supported 8 Cores, 1.9Ghz 4 Cores, 2.4Ghz
• 9006/9010/9910/9906/9904
• Feature parity cXR with RSP880 at FCS (eXR support 4 SFP+ external ports No external SFP+ ports
planned for 6.4.x)
• Similar Control Plane Scale to RSP880
2x32GB SSD 2x128GB SSD
• No support for nV-Cluster and 3rd party applications

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Comparison of RSP880-LT to RSP880

RSP880-LT RSP880/A99-RSP

Bandwidth/Slot 880Gbps (Dual RSP)*** 880Gbps (Dual RSP)

RIB Scale** 26M/27M (IPv4/IPv6) 26M/27M (IPv4/IPv6)

RIB Learning Rate 95K IPv4 Routes/sec 78K IPv4 Routes/sec

Platforms Supported ASR 9904, 9006, 9010, 9910, 9906 RSP880: ASR 9904, 9006, 9010

A99-RSP: ASR 9910, 9906

3rd Party Application Support Not Supported Supported

eXR Support Planned for support in 6.4.x Supported

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ASR 9000 Linecards Evolution
1st Gen
Trident Class Trident Octopus Santa Cruz PowerPC
120G 90nm 130nm 130nm Dual Core
15 Gbps 60 Gbps 90 Gbps 1.2 Ghz

2nd Gen
Typhoon Class Typhoon Sacramento
Skytrain PowerPC
360G 55nm 65nm Quad Core
65nm
60 Gbps 220 Gbps 1.5 Ghz
60 Gbps

3rd Gen
Tomahawk Class Tomahawk Tigershark SM15 X86
1.2T 28nm 28nm 28nm 6 Core
240 Gbps 200 Gbps 1.20 Tbps 2 Ghz

4th Gen
Lightspeed Class Lightspeed Skybolt X86
3.2T 400 Gbps 16 Core
BRKARC-2003

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 ASR 9K Ethernet Line Card Overview -TR, -SE

2nd LC Typhoon
NPU: 60Gbps,
~45Mpps

A9K-MOD80 A9K-MOD160

A9K-36x10GE A9K-2x100GE A9K-24x10GE MPAs

20x1GE
2x10GE
4x10GE
1x40GE
A9K-40GE 2x40GE
A9K-4T16GE

3rd LC Tomahawk MPAs

1x100GE
NPU: 240Gbps, 2x100GE
~150Mpps 20x10GE
+
Typhoon
MPAs

A9K-4x100GE A9K-8x100GE A9K-12x100GE MOD400/MOD200 24/48x10/1G LC A9K-400G-DWDM

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Cost Optimized 4x100GE LAN Line card

• Tomahawk based, 5-fabric linecard with: 2x NPU and 5Mbit internal TCAM
HW Specs
• Supported on all ASR 90xx and 99xx systems

• Parity with Tomahawk TR linecards at FCS except for Timings features (post FCS)
Features
• Features not supported: VidMon, Cluster, LISP, NSH, BNG, Satellite, MPLS-TP

• Limited scale for TCAM dependent features; rest of the scale on-par with other Tomahawk –TR
Scale cards

• cXR: 6.2.3, 6.3.2, 6.4.1 and onwards. SMU option possible for 6.1.4, 6.3.1, subject to business case
SW support • eXR: 6.4.1

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Pop Quiz ????

Which of the following RSP is supported in all of ASR9006, ASR9010,

ASR9904, ASR9906 and ASR9910?
• RSP880-LT
• RSP880

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
ASR 9000 Hardware Architecture
 ASR 9000 System Architecture “At-a-Glance”
Control plane split
RSP/RP Network Processor
among RSP/RP
and LC CPU
CPU

RSP FIA NP Bay

CPU CPU
Network Processor CP
BITS/DTI
U
Switch SerDes
FIA Fabric XBAR

FIC
CPU
CPAK

CPAK
PHY
NP FIA FIA NP Bay

CPAK

CPAK
PHY
NP FIA
Switch
CPAK
Fabric Line Card
CPAK
PHY
NP FIA
CPAK

CPAK
PHY
NP FIA Fully Distributed Architecture for
High Performance and High Multi-
Data forwarding is fully Switch Fabric dimensional Control Plane Scale
distributed across NPs
Active-Active Switch Fabric
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 ASR 9000 Switch Fabric High-Level Architecture
3-Stage Non-Blocking Fabric (Separate Unicast and Multicast Crossbars)
Fabric frame format:
Super-frame Stage 1 Stage 2 Stage 3
Fabric load balancing: Active-Active
Unicast is per-packet Fabric
Multicast is per-flow Unicast
Virtual Output Queue Crossbar
Arbitration fabric

fabric fabric
Arbiter FIA
FIA Multicast FIA
FIA RSP0
Crossbar FIA
FIA
Tomahawk
Typhoon LC
fabric
Egress Linecard
Ingress Linecard
Arbiter

RSP1
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ASR90xx – RSP880 and Mixed LC
Stage 1 Stage 2 Stage 3

Egress Linecard
Ingress Linecard Fabric 8x115Gbps
SM15

Fabric
Arbiter FIA
FIA Fabric FIA
FIA RSP880 SM15 FIA
FIA

Tomahawk Line
Typhoon Linecard Card
Fabric
SM15

Fabric bandwidth: 8x55Gbps Fabric bandwidth:

Arbiter
8x55Gbps =440Gbps/slot with dual RSP 8x115Gbps ~ 900Gbps/slot with dual RSP
RSP880
4x55Gbps =220Gbps/slot with single RSP 4x115Gbps ~ 450Gbps/slot with single RSP

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
ASR99xx Switch Fabric Card (FC2)
6+1 All Active 3-Stage Fabric Planes, Scale to 1.6Tbps/LC, Separated Ucast/Mcast Crossbars

5x 2x115G bi-directional 7x2x115G bi-directional

= 1.15Tbps = 1.6Tbps

5-Fabric LC 7-Fabric LC

FIA
FIA Fabric FIA
FIA Fabric
SM15 FIA
SM15
FIA

Tomahawk Line Tomahawk Line

Card Card

Fabric frame format:

Super-frame Fabric bandwidth:
Fabric load balancing: 10x115Gbps ~ 1.15 Tbps/slot with 5x FC2
Unicast is per-packet SFC v2
Multicast is per-flow 14x115Gbps ~ 1.6 Tbps/slot with 7x FC2

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Tomahawk LC: 8x100GE Architecture
Slice Based Architecture
VoQ buffering, Fabric credits, LC CPU
FPOE, Auto-Spread,
L2/L3/L4 lookups, all VPN types, all mcast hashing, scheduler for fabric
CPAK: Macsec Suite B+, G.709, OTN, DWRR, RBH, replication
feature processing, mcast and egress port
100G, 40G, 10G Clocking
replication, QoS/Queuing, ACL, etc …

240G 240G
Tomahawk
PHY FIA Up to
NP 14x115G

240G 240G

Separated Switch Fabric

Tomahawk
PHY FIA
NP

XBAR
240G 240G
Tomahawk
PHY FIA
NP

240G 240G
Per Slice Power Management:
PHY
Tomahawk Tigershark
(100-200W Power Savings)
NP saving location ? sliceFIA
PE1(admin-config)# hw-module power [0-3]

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
4x100GE LAN-based linecard architecture

Interlaken
2x 16x 10.9375G 2x 8x15G = 240G raw bw
= 350G raw bw 3x Fencer ports

QSFP 0

QSFP 1
PHY NP FIA
CPU

Switch Fabric
(SM15)
QSFP 2

QSFP 3
PHY NP FIA
Up to 10x115G

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
A9K-48x10G-1G Architecture
8 Ports
SFPs0-7 Ports0-7 PHY0

IFE0
4 Ports

SFPs8-15 Ports8-15 PHY1 NPU 0 FIA SM15

IFE1
4 Ports
SFPs16-23 Ports16-23 PHY2
8 Ports
• 240G Raw Bandwidth/ 200G
Data Traffic Bandwidth
• Potentially Oversubscribed
SFPs24-31 Ports24-31 PHY3

IFE0
SFPs32-39 Ports32-39 PHY4 NPU 1 FIA SM15

IFE1
SFPs40-47 Ports40-47 PHY5

• Supports all 1G and 10G. 1G and 10G are configurable follow certain rule
• 48-port has a two NPU slices with shared 80Mb TCAM
• Phy does not support OTN Framing or MACSec

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
A9K-48x10G-1G Port Organization
• A9K-48X10GE-1G-SE/TR has two NPU slices, A9K-24X10GE-1G-SE/TR has one
• Each NPU slice consists of two Interface Engines (IFE)
• Interface Ports are organized in port-groups
• Each IFE consists of three port-groups
• Each port-group consists of 4 interface ports

A9K-48X10GE-1G-SE/TR
TOMAHAWK NPU TOMAHAWK NPU

IFE IFE IFE IFE

00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group

12 ports per IFE 12 ports per IFE 12 ports per IFE 12 ports per IFE
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
A9K-48x10-1G Port Speed Rules
• Interface Port Speed is configurable: 1G / 10G
• Per default, all ports are 10G
• The 4 Rules:
1. All ports in a port-group (group of four ports) have to have the same speed.
2. If the first port-group on an IFE (group of three port-groups) is configured for 1G, then all ports
in that same IFE must be 1G.
3. If the first port-group of an IFE is 10G, then the other two port-groups in that same IFE can be
any combination of 1G or 10G.
4. When configuring port speeds, all ports of a given linecard need to be configured together in
one single CLI command.
• Deviations from these rules will results in a CLI rejection
• Changing port speed on an interface does not impact traffic on other interfaces

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Possible Port Speed Combination per IFE
• 5 possible port speed combinations for each IFE
• IFE are independent of each other for port speed configuration

Interface Engine (IFE)

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

Configuration 1 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G

Configuration 2 10G 10G 10G 10G 1G 1G 1G 1G 1G 1G 1G 1G

Configuration 3 10G 10G 10G 10G 10G 10G 10G 10G 1G 1G 1G 1G

Configuration 4 10G 10G 10G 10G 1G 1G 1G 1G 10G 10G 10G 10G

Configuration 5 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 Runs distributed control plane
Line Card Components protocols for increased scale
BFD, CFM, ARP
Main forwarding engine L2 and L3 lookups Receive FIB table from RP and
Multicast replication toward Optics program hardware forwarding table
User level QoS and Security features CPU

NPU P1
PHY P1
FIA
P2
P3 Switch
TM
P2
P3
BE Fabric
BE ASIC

Dedicated queue ASIC – TM (traffic Provides data connection to switch fabric

manager) per NPU for QoS functions
Manage VoQ, Superframe and loadbalancing
User Configurable Queue on TM. data traffic across switch fabric
Default Port Queue Always Created. Mcast replication table for replication toward
NPs
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Network Processor Architecture Details
TR and SE has different
TR and SE has same
memory size
memory size NPU Complex

STATS MEMORY
FIB MAC
LOOKUP Forwarding chip (multi core) FRAME MEMORY
MEMORY TCAM
-

• TCAM: VLAN tag, QoS and ACL classification

• Stats memory: interface statistics, forwarding statistics etc
• Frame memory: buffer, Queues
• Lookup Memory: forwarding tables, FIB, MAC, ADJ
• TR/SE
• Different TCAM/frame/stats memory size for different per-LC QoS, ACL, logical interface scale
• Same lookup memory for same system wide scale mixing different variation of LCs doesn’t impact system
wide scale
-TR: transport optimized, -SE: Service edge optimized
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Pop Quiz ????

For the A9K-48x10G-1G linecard, if the first port-group of an IFE is

configured as 1G, what is the possible port speed for the other two
port-groups of the same IFE?
• 10G
• 10G or 1G
• 1G

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
ASR 9000 Distributed Control Plane
ASR9000 Fully Distributed Control Plane
CPU
LPTS (local packet transport service):
control plane policing
RP
CPU Punt
FPGA FIA

Switch Fabric Switch Fabric

LC
Punt Switch CPU
Control CPAK

packet CPAK
PHY
NP
LPTS FIA
CPAK RP CPU: Routing, MPLS, IGMP, PIM,
CPAK
PHY
NP FIA HSRP/VRRP, etc
Switch
Fabric
CPAK

CPAK
PHY
NP FIA LC CPU: ARP, ICMP, BFD, NetFlow,
CPAK
OAM/CFM, L2 Protocols etc
CPAK
PHY
NP FIA
NP Offloading: BFD
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
L3 Control Plane Architecture

Static BGP OSPF

LDP RSVP-TE
ISIS EIGRP

LSD RIB RSP/RP CPU

RSP/RP

LC

ARP/NDP
SW FIB HW FIB Adjacency
LC NP
AIB
AIB: Adjacency Information Base
LC CPU RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Distributed ARP Processing

Incomplete ADJ Packets RP CPU RP

Incoming ARP Packets
ARP FIA

Switch Fabric
Outgoing ARP Packets

Line card ARP LC

CPU
spio netio

Switch Fabric
SPP

Tsec Driver

Punt Switch

LPTS
PHY NP FIA

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 MAC Learning and Sync
Hardware based MAC learning: ~4Mpps/NP
1 NP learn MAC address in hardware (around
4M pps)
RP
2 NP flood MAC notification (data plane) Punt
CPU
message to all other NPs in the system to sync FPGA FIA
up the MAC address system-wide. MAC
notification and MAC sync are all done in
hardware
Switch Fabric Switch Fabric

CPU LC1
LC2
CPU
Data 3x10GE
SFP + 1NP 2
packet 3x10GE
SFP + NP
FIA 3x10GE
SFP +
NP
FIA
3x10GE
3x10GE
SFP +
NP 2

Fabric ASIC
Switch
SFP + NP
FIA 3x10GE
3x10GE NP

Fabric ASIC
Switch
NP SFP +
SFP + FIA
3x10GE
3x10GE
SFP +
NP
SFP + NP
FIA 3x10GE
3x10GE
SFP +
NP
SFP + NP FIA
3x10GE
3x10GE
SFP +
NP
SFP + NP
3x10GE FIA 3x10GE
NP
NP SFP +
SFP + FIA
3x10GE
SFP +
NP

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Distributed BFD Architecture

RP
OSPF ISIS BGP

BFD Events
BFD session info

BFD Session Tables on RP

BFD Session info BFD Events BFD Session info

LC1-CPU LCn-CPU
BFD Session Tables on LC …… BFD Session Tables on LC

BFD Hellos BFD Hellos

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
HW-offloaded BFD
hw-module bfd-hw-offload enable location 0/0/CPU0

RP
OSPF ISIS BGP

BFD Events
BFD session info

BFD Session Tables on RP

BFD Session info BFD Events BFD Session info

LC1-CPU LCn-CPU
……
BFD BFD BFD BFD BFD BFD BFD BFD
On On On On On On On On
NP NP NP NP NP NP NP NP

BFD Hellos BFD Hellos

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Distributed Netflow Architecture

RP
Netflow Netflow show/clear
• Aggregate 200Kpps netflow •
policer rate
Configure Command Manage flow table
• Export flow
• Evenly Divided among information
netflow-enabled NPs

LC-CPU EXPORT

netIO
NPU:
• Traffic Filtering 1M Record Cache
• Traffic Sampling
• Extract flow
header information
• LPTS Policer

LC
NPU NPU NPU NPU

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Pop Quiz ????

In an ASR9906 system which has 2 RSP, 4 linecards and 5 switch

fabric cards, how many total CPUs are available to host ASR9000
control-plane functionalities?
• 2
• 4
• 6
• 11

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
ASR 9000 Data Packet Processing
 Distributed Two-Stage Packet Processing
• Ingress lookup yields packet egress port and applies ingress features

• Egress lookup performs packet-rewrite and applies egress features

CPU CPU
CPAK 0 CPAK 0
1
PHY NP FIA FIA NP PHY
CPAK 1
CPAK 1

Switch CPAK 2
CPAK 2
Fabric
PHY NP FIA FIA 2
NP PHY
CPAK 3 CPAK 3
Switch Switch

…
…
Fabric Fabric
CPAK 4 CPAK 4
FIA
(SM15) (SM15)
PHY NP FIA NP PHY
CPAK 5
CPAK 5
Up to Up to
14x120G 14x120
CPAK 6 1 Switch G CPAK 6
PHY NP 2 FIA Fabric FIA NP PHY
CPAK 7 CPAK 7

Uniform packet flow for simplicity and predictable performance

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
ASR9000 Life of a Packet – Tomahawk LC
2 4 6 8
• Ingress L2/L3 FIB lookup, • Buffering packet from NP • Egress L2/L3 FIB lookup, • MACSEC Emcryption
ACL/QoS lookup • Requesting fabric credit ACL/QoS lookup • G.709/OTN/WAN-
• Ingress PBR/ABR, ACL, uRPF • Manage superframe and • Egress PBR/ABR, ACL, uRPF PHY/LAN-PHY
• Ingress QoS: classification, load-balancing packet • Egress QoS: classification, • Line Clocking
marking, policing across fabric marking, policing, shaping
• Packet Punting • Manage system VoQ • Incomplete Adj Packet Punting
• Ingress ECMP/LAG hashing • Egress ECMP/LAG hashing

Ingress side of LC CPU CPU 6 8

NP PHY
5 7 TM
1 4
CPAK 0 2 FIA
CPAK 1
PHY NP
3 TM
FIA Switch
Fabric
Egress side of LC
1 3 5
7
• MACSEC Decryption • Ingress Queuing • Re-assembling packets • Egress Queuing
• G.709/OTN/WAN- Processing from superframe Processing
PHY/LAN-PHY • Bypassed in case no • Send packet to
• Line Clocking ingress queuing support corresponding NP
• Release buffer and fabric
credit

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Switch Fabric Arbitration
5: credit return
Fabric
1: Fabric Request ASIC

Fabric
ASIC

Arbitration
Fabric ASIC Fabric ASIC
and VOQ SFC or RSP0 2: Arbitration and VOQ

Fabric
ASIC
3: Fabric Grant
Fabric
ASIC

4: load-balanced Arbitration
transmission SFC or RSP1
across fabric links

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Fabric Super-framing Mechanism

• Multiple unicast frames from/to same destinations aggregated into one super frame
• Super frame is created if there are frames waiting in the queue, up to 32 frames or
when min threshold met, can be aggregated into one super frame
• Super frame only apply to unicast, not multicast
• Super-framing significantly improves total fabric throughput
Packet 1 No super-framing
Packet 2 Packet 1 Min reached
Packet 3 Packet 2 Packet 1 Max reached

Packet 1 Jumbo

Max Min 0 (Empty)

Super-frame Super-frame

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Fabric Load Balancing – Unicast
Fabric
ASIC

Fabric
ASIC

Fabric ASIC Arbitration Fabric ASIC

and VOQ SFC or RSP0 and VOQ 4 3 2 1

Fabric
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1

• Unicast traffic sent across first available fabric link to destination (maximizes efficiency)
• Each frame (or super frame) contains sequencing information
• All destination fabric ASIC have re-sequencing logic
• Additional re-sequencing latency is measured in nanoseconds

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Fabric Load Balancing – Multicast
Fabric
ASIC

Fabric
ASIC

Fabric ASIC Arbitration Fabric ASIC C B A B A A

and VOQ SFC or RSP0 and VOQ 1 2 3 1 2 1

Fabric
Flows exit in-order
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1

• Multicast traffic hashed based on (S,G) info to maintain flow integrity

• Very large set of multicast destinations preclude re-sequencing
• Multicast traffic is non arbitrated – sent across a different fabric plane

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
ASR 9000 QoS Architecture
 ASR9000 Priority-Based QoS Architecture
• Dedicated Traffic Manager(TM) for Traffic Queuing
• User Configurable QoS Policy on Ingress/Egress NP
• End-to-End priority propagation  Guarantee bandwidth, low latency for high priority traffic
• Unicast VOQ and back pressure
Ingress side of LC Egress side of LC

CPU CPU 4
P1 NP0 PHY
P2
P3 3 P1
P1
P2
BE
2
P2
P3 NP1 PHY
CPAK 0 TM P3
BE
FIA BE

CPAK 1
PHY NP FIA Switch NP2 PHY
1 Fabric P1
NP3
TM
P2
P3
PHY
BE

1 2 3
4
• Ingress (sub-)interface • 4xVOQ per VQI 4x Egress Destination Qs per VQI,
QoS Queues Egress (sub-)interface
• Up to 8K VOQs per TSK FIA aggregated at egress port rate
• User Configurable (vs 4k per SKT FIA) QoS Queues
Ingress QoS Policy User-configuration
with Egress MQC
User-configuration Implicit Configuration
with Ingress MQC Not User-controllable
BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Default Implicit Trust Model L2 IF: trust outer Cos
L3 IF: trust DSCP
L3 MPLS: trust outer EXP

IPP=5 DSCP=44
802.1p = 1 L2 Bridging internal
IPP=5 DSCP=44 802.1p = 1 *
cos = 1 IPP=5 DSCP=44

Carried in internal buffer header, ASR 9000 would never modify

by default, internal cos is used for impositioned fields only,
packet DSCP/IP without
For example, added vlan tag, impositioned MPLS label,
It doesn’t include VLAN tag translation or MPLS label swap a policy-map configured

IPP=5 DSCP=44

Untagged L2 Bridging
internal 802.1p = 0 *
IPP=5 DSCP=44 IPP=5 DSCP=44
cos = 0

Ingress line card Egress line card

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
3-Layer Hierarchical QoS (H-QoS)
policy-map child
L0 L1 L2 L3 L4
class Pr1
Port Group Port Grand-Parent
Parent Policy on
L2 (EFP) or L3 Child policy (child of police rate 64 kbps
scheduler scheduler Policy subint. class- Parent) with user- priority level 1
not not
default or physical
port /w user- defined classes & class Pr2
configurable configurable defined classes class-default police rate 10 mbps
priority level 2
class Cl3
bandwidth 3 mbps
class Cl4
bandwidth 1 mbps
!
policy-map parent
class parent1
shape average 100 mbps
service-policy child
class parent2
shape average 25 mbps
service-policy child
class class-default
•N/A •N/A •Shape •Priority Level 1 •Priority Level 1 !
•Priority Level 2 •Priority Level 2
•Bandwidth •Shape, •Shape, policy-map grand-parent
•bandwidth or
remaining bandwidth
remaining
•bandwidth or bandwidth
remaining
class class-default
•1R2C policer •(W)RED •(W)RED shape average 500 mbps
•Police •Police
•Set (marking) •Set (marking) service-policy parent

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
H-QoS – Supported Classification/Policy
Policy-map hierarchy level Classification support Policy Support

• Shape Average
• Bandwidth remaining
Grand-parent Only class-default • 1R2C policer with only drop/transimit
action(no set/mark, Tomahawk card only)

User defined fields with • Priority/WRED Queue and Queue-limit on

Leaf only
Parent restrictions based on • Policer/Shaper/Marking/non-Priority
format/interface types. Queue/Bandwidth/Bandwidth Remaining

User defined fields with • Priority/WRED Queue and Queue-limit on

Leaf only
Child restrictions based on • Policer/Shaper/Marking/non-Priority
format/interface types. Queue/Bandwidth/Bandwidth Remaining

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Default Interface Queues
P1 P2 P3 L

Level 4 Queues

Level 3 Schedulers

Level 2 Schedulers

Level 1 Schedulers

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 MQC Hierarchy in Queuing ASIC
Port default queues MQC queues
policy-map child
P1 P2 P3 L c1 c2 cd-c class c1
priority level 1
police rate 640 kbps
class c2
bandwidth 20 mbps
class class-default cd-c
bandwidth 1 mbps
L4 !
policy-map parent
class class-default cd-p
shape average 35 mbps
service-policy child
L3 cd-p !
interface GigabitEthernet0/0/0/0
service-policy output parent

L2
Inactive entity

Active entity
L1

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 MQC Hierarchy in Queuing ASIC
policy-map child
Port default queues G0/0/0/0.1 G0/0/0/0.2 class c1
P1 P2 P3 L c1 c2 cd-c c1 c2 cd-c priority level 1
police rate 640 kbps
class c2
bandwidth 20 mbps
class class-default cd-c
bandwidth 1 mbps
!
policy-map parent
L4 class class-default cd-p
shape average 35 mbps
service-policy child
!
L3 cd-p cd-p interface GigabitEthernet0/0/0/0.1
service-policy output parent
!
interface GigabitEthernet0/0/0/0.2
service-policy output parent
L2
Inactive entity

L1 Active entity

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Pop Quiz ????

For an incoming untagged layer 2 frame, what is the priority value

used when the frame is processed inside ASR9000?
• 2
• COS bit
• 802.1p Value
• 0

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
IOS-XR & IOS-XR 64 Bit
Cisco IOS – A Recap
Cisco IOS Cisco IOS-XE Classic IOS-XR IOS-XR 64 Bit
Control Data Mgmt
Plane Plane Plane
XR Code v1 XR Code v2

Hosted App 1

Hosted App 2
IOSd

NetFlow
SNMP
OSPF

LPTS

XML
BGP

NetFlow
QoS

NetFlow
ACL
PIM

SNMP
SNMP

OSPF
OSPF

LPTS
LPTS

BGP
BGP

XML
QoS
XML
QoS

ACL
ACL

PIM
PIM
System
IOS “Blob” Admin

Operational Infra Distributed Infra Distributed Infra Distributed Infra

Kernel Kernel Kernel Kernel Kernel

Linux-BinOS QNX, 32bit Linux, 64bit Linux, 64bit Linux, 64bit

Virtualization Layer

1990s 2000s 2003-04 Present Day

Incremental Development, with Industry leading investment protection

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
IOS XR Evolution: XR 64 bit Architecture
IOS-XR IOS-XR Separate Admin Plane
Routing Admin Plane
Apps Routing

RP

RP
Control Plane
System 64-bit IOS XR.
Admin Linux Linux
QNX Linux
IOS XR IOS-XR
Admin Plane
Line Card

Line Card
Linux VM

LC-CPU LC-CPUs

Linux Linux
QNX 64 bit Linux Kernel
Linux

Classic IOS XR Linux-based Virtualized

IOS XR 64 Bit

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
IOS XR 64 Bit Packaging
Bootable Images

Core packages: OS, Admin, Forwarding, Modular Services

Minimum Image asr9k-mini-x64-6.1.2.iso Card, Basic Routing, SNMP, Alarm Correlation

Golden ISO Customized ISO image includes mini ISO + required packages + SMUs + XR config

Optional Feature Packages

asr9k-eigrp-x64-1.0.0.0-r612.x86_64.rpm
asr9k-mpls-x64-2.1.0.0-r612.x86_64.rpm
asr9k-isis-x64-1.1.0.0-r612.x86_64.rpm
asr9k-mcast-x64-2.0.0.0-r612.x86_64.rpm
asr9k-ospf-x64-1.1.0.0-r612.x86_64.rpm
asr9k-optic-x64-1.0.0.0-r612.x86_64.rpm
asr9k-m2m-x64-2.0.0.0-r612.x86_64.rpm
asr9k-li-x64-1.1.0.0-r612.x86_64.rpm
asr9k-mgbl-x64-3.0.0.0-r612.x86_64.rpm
asr9k-k9sec-x64-3.1.0.0-r612.x86_64.rpm
asr9k-mpls-te-rsvp-x64-1.2.0.0-r612.x86_64.rpm

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
IOS XR 64 Bit Packaging - GISO – Golden ISO
• GISO is a customized iso which is built as per individual customer needs.
• GISO contains Mini ISO + rpms + SMUs + config

How to build GISO: use tool provided on the router at /pkg/bin/gisobuild.py in XR domain.

Usage: gisobuild.py [-h] -i BUNDLE_ISO [-r RPMREPO] [-c XRCONFIG] [-l GISOLABEL]
[–m] [-v]
Example: gisobuild.py -i asr9k-mini-x.iso -r . -c config-file -l v1

Script Parameter Expansion Explanation Required/optional

-I BUNDLE_ISO Path to mini ISO Required
-r RPMREPO Path to RPM repo Optional
-c XRCONFIG Optional
-l GISOLABEL GISO Label Optional
-m migration To build migration tar for ASR9K only. Optional
-v version Print script version and exit Optional
-h help Print help menu N/A

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Migrating Classic XR to IOS XR 64 Bit
eXR Migration

eXR

FPD eXR 6.2.x

eXR 6.3.x
Upgrade
cXR 6.2.x
cXR or later

Pre-6.1.3 Target 64 Bit XR

5.x.y/4.x.y Classic XR
Corresponding
To Target 64 Bit XR Two Ways to Migrate:
1. CSM Orchestrated Migration
2. MOP: Manual Migration
Follow below link for details:
https://www.cisco.com/c/en/us/td/docs/routers/asr9000/migration/guide/b-migration-to-ios-xr-64-
bit/b-migration-to-ios-xr-64-bit_chapter_011.html

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Migration Pre-requisites

HW Component Backup cfg to

Upgrade cXR Operational Status External Server
Check
(Calvados, XR)

• All hardware • Any pre-6.1.3 • All hardware • Back up admin/XR

components, release needs to components must configuration to
Chassis, RSP/RP, be upgraded to be in operational external server
LC, FC, FAN and classic XR state before
PEM, should be corresponding to migration
supported in target IOS XR 64
ASR9K 64 bit. Any Bit version
unsupported
hardware may fail
to boot after
migration

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
ASR 9000 System Migration Example to Lightspeed
Classic XR 5.3.4 Classic XR 6.5.1 64 Bit XR 6.5.1 64 Bit XR 6.5.1

RP2/RSP4 RP2/RSP4 RP2/RSP4 RP3/RSP5

RP2/RSP4 RP2/RSP4 RP2/RSP4 RP3/RSP5
SFC2 SFC2 SFC2 SFC3
SFC2 SFC2 SFC2 SFC3
SFC2 SFC2 SFC2 SFC3
SFC2 SFC2 SFC2 SFC3
SFC2 SFC2 SFC2 SFC3
Tomahawk Tomahawk Tomahawk Lightspeed

Upgrade FPD with

cXR 6.5.1 FPD

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Questions?

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Conclusion

• ASR9000 - Truly Carrier-Class Edge Router Provides:

• Rich Features, Flexible Service Capability
• Variety of Hardware to Meet Different Capacity Requirements

• Fully Distributed Architecture for High Performance

and System Scalability
• Uniform, Open and Modularized Software Architecture
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKARC-2003

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be
available for viewing on-demand after the
event at CiscoLive.com/Online.

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions

BRKARC-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Thank you