TRM N55K L2only-Config Tshoot Jdinkin2 2hr 20120208

Cisco Advanced Services
Cisco Nexus 5500 Series

Configuration and
Troubleshooting
Knowledge Transfer
Instructor: Joel Dinkin ([email protected])

Cisco Advanced Services Network Consulting Engineer
2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
Agenda
Nexus 5500 Series Hardware and Architecture
Device Management
In-Service Software Upgrade (ISSU)
Layer 2 Switching
Virtual Port Channel (vPC)
Multicast
Quality of Service (QoS)
Troubleshooting
Cisco Confidential
Nexus 5500 Series

Hardware and Architecture
Cisco Confidential
Nexus 5500 Hardware
Nexus 5548UP
Nexus 5596UP
32 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

Line-rate, Non-blocking 10G FCoE/IEEE DCB
1 Expansion Module Slot
IEEE 1588, FabricPath & Layer 3 Capable
Redundant Fans & Power Supplies
48 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

Line-rate, Non-blocking 10G FCoE/IEEE DCB
3 Expansion Module Slot
IEEE 1588, FabricPath & Layer 3 Capable
Redundant Fans & Power Supplies
Cisco Confidential
Nexus 5500 Hardware

Nexus 5548 (5548P & 5548UP)
32 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Fabric Interconnect
Not Active on Nexus
Console
Out of Band Mgmt

10/100/1000
Fan Module
Expansion Module
USB Flash
Fan Module
Power Entry
N + N Redundant FANs
Power Entry
N + N Power Supplies
Cisco Confidential
Nexus 5500 Hardware

Nexus 5596UP
3 Expansion Modules
48 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Fabric Interconnect
Not Active on Nexus
Power Supply
N + N Power Supplies
Out of Band Mgmt

10/100/1000
Fan Module
Console
Fan Module
Fan Module
USB Flash
Fan Module
N + N Redundant FANs
Cisco Confidential
Nexus 5500 Hardware

Nexus 5500 Expansion Modules
Nexus 5500 expansion slots
Expansion Modules are hot swappable (Future support
for L3 OIR)
Contain forwarding ASIC (UPC-2)
16 x 1/10GE
8 x 1/10GE +
8 x 1/2/4/8G FC
16 unified ports
individually
configurable as 1/10GE
or 1/2/4/8G FC
L3 module for
160G of L3 I/O
bandwidth
Cisco Confidential
1G Support on all ports

Any Ethernet port or Flexible port in N55xx switches can be
configured in 1G mode.
Requires the use of a standard 1G SFP

GLC-T, GLC-SX-MM, GLC-LH-SM, SFP-GE-T, SFP-GE-S, SFPGE-L (DOM capable SFP are supported)
Supports for all features at 1G speed other than Unified I/O

No FCoE (no 1G Converged Network Adapters are shipping)
No Priority Flow Control (standard Pause is available)
CLI to configure 1G
switch(config)# interface Ethernet1/1
switch(config-if)# speed 1000
5.0(3)N1(1)
Required for
1Gbps Support!
Cisco Confidential
Nexus 5500 Layer 3 Options

L3 Hardware
List Price
$5,000
Nexus
Nexus5548P
5548P
Nexus
5548UP
Nexus 5548UP
Nexus
Nexus 5596UP
5596UP
Cisco Confidential
Nexus 5500 Hardware

Nexus 5500 Reversible Air Flow and DC Power Supplies
Nexus 5548UP and 5596UP will support reversible
airflow (new PS and fans)
Nexus 5548UP and 5596UP will support DC power
supplies (not concurrent with reversible airflow)
Note: 5548UP and 5596UP ONLY, not 5548P

Nexus 5500
Hardware
Availability
Front-to-Back Airflow, AC
Power
Nexus
5548P/5548UP/5596UP
Today
Back-to-Front Airflow, AC
Power
Nexus 5548UP/5596UP
Nexus 5548UP
Nexus 5596UP (Future)
Front-to-Back Airflow, DC
Power
Nexus 5548UP/5596UP
Nexus 5548UP
Nexus 5596UP (Future)
Back-to-Front Airflow, DC
Power
N/A
N/A
Cisco Confidential
10
Reverse Air Flow - CLI

CLI enhancements to display air flow direction.
switch# show environment fan detail
--------------------------------------------------Module
Fan
Airflow
Speed(%)
Speed(RPM)
Direction
--------------------------------------------------1
Front-to-Back
40
6733
Front-to-Back
40
6609
Front-to-Back
40
6835
Front-to-Back
40
6792
Front-to-Back
40
6683
Front-to-Back
40
6683
Front-to-Back
40
6758
Front-to-Back
40
6861
Cisco Confidential
11
Nexus 5500 Internals

Data and Control Plane Elements
Expansion Module
10 Gig
Gen 2 UPC
Gen 2 UPC
Gen 2 UPC
DRAM
DDR3
CPU Intel
Jasper
Forest
South
Bridge
Flash
12 Gig
Memory
PCIe x8
Unified Crossbar Fabric

Gen 2
NVRAM
Serial
PEX 8525
4 port PCIE
Switch
Console
PCIe x4
Gen 2 UPC
...
Gen 2 UPC
PCIE
Dual Gig
0 1
PCIE
Dual Gig
0 1
PCIE
Dual Gig
0 1
L2
L1
Mgmt 0
Cisco Confidential
12
Nexus 5500 Hardware Overview

Data Plane Elements - Unified Port Controller (Gen 2)
Each UPC supports eight ports and
contains,
Unified Port
Controller 2
Multimode Media access controllers

(MAC)
Support 1/10 G Ethernet and 1/2/4/8 G
Fibre Channel
All MAC/PHY functions supported on the
UPC (5548UP and 5596UP)
Packet buffering and queuing

MMAC + Buffer +
Forwarding
Cisco Confidential
MMAC + Buffer +
Forwarding
MMAC + Buffer +
Forwarding
MMAC + Buffer +
Forwarding
MMAC + Buffer +
Forwarding
MMAC + Buffer +
Forwarding
Ethernet (Layer 2 and FabricPath) and

Fibre Channel Forwarding and Policy
(L2/L3/L4 + all FC zoning)
MMAC + Buffer +
Forwarding
Forwarding controller
MMAC + Buffer +
Forwarding
640 KB of buffering per port
14

Control Plane Elements Nexus 5500
CPU - 1.7 GHz Intel Jasper Forest (Dual Core)
DRAM - 8 GB of DDR3 in two DIMM slots
Program Store - 2 GB of eUSB flash for base

system storage and partitioned to store image,
configuration, log.
Memory
PCIe x8
On-Board Fault Log (OBFL) - 64 MB of flash to

store hardware related fault and reset reason
Management Interfaces
10/100/1000BASE-T: mgmt0 partitioned
from inbound-hi VLANs
NVRAM
Serial
PEX 8525
4 port PCIE
Switch
PCIe x4
PCIE
Dual Gig
0 1
RS-232 console port: console0
South
Bridge
Flash
Boot/BIOS Flash - 8 MB to store upgradable

and golden version of (Bios + bootloader)
image
NVRAM - 6 MB of SRAM to store Syslog and

licensing information
DRAM
DDR3
CPU Intel
Jasper
Forest
Console
PCIE
Dual Gig
0 1
PCIE
Dual Gig
0 1
Mgmt 0
inbound-hi Data Path
to CPU
Cisco Confidential
15

Control Plane Elements - CoPP
CPU Intel
Jasper
Forest
In-band traffic is identified by the

UPC and punted to the CPU via two
dedicated UPC interfaces, 5/0 and
5/1, which are in turn connected to
eth3 and eth4 interfaces in the CPU
complex
Eth3 handles Rx and Tx of low
priority control pkts
PEX 8525
4 port PCIE
Switch
NIC
0 1
IGMP, CDP, TCP/UDP/IP/ARP (for

management purpose only)
Eth4 handles Rx and Tx of high

priority control pkts
STP, LACP, DCBX, FC and FCoE
control frames (FC packets come to
Switch CPU as FCoE packets)
BPDU
SDP
ICMP
Cisco Confidential
16

CPU Intel
Jasper
Forest
CPU queuing structure provides strict

protection and prioritization of inbound traffic
Each of the two in-band ports has 8 queues

and traffic is scheduled for those queues
based on control plane priority (traffic CoS
value)
Prioritization of traffic between queues on
each in-band interface
CLASS 7 is configured for strict priority
scheduling (e.g. BPDU)
PEX 8525
4 port PCIE
Switch
NIC
0 1
CLASS 6 is configured for DRR scheduling

with 50% weight
Default classes (0 to 5) are configured for DRR

scheduling with 10% weight
Additionally each of the two in-band

interfaces has a priority service order from
the CPU
BPDU
SDP
Eth3 interface has low priority (interrupt

moderation)
ICMP
Eth 4 interface has high priority to service

packets (no interrupt moderation)
Cisco Confidential
17

CPU Intel
Jasper
Forest
On Nexus 5500 an additional level of control

invoked via policers on UPC-2
Software programs a number of egress
policers on the UPC-2 to avoid overwhelming
the CPU (partial list)
STP: 20 Mbps
PEX 8525
4 port PCIE
Switch
NIC
0 1
LACP: 1 Mbps
DCX: 2 Mbps
Satellite Discovery protocol: 2 Mbps
IGMP: 1 Mbps
DHCP: 1 Mbps
Egress
Policiers
...
BPDU
SDP
ICMP
CLI exposed to tune CoPP (Future)
Cisco Confidential
18

Control Plane Elements
Monitoring of in-band traffic via the
NX-OS built-in ethanalyzer
NX-OS
Etheranalyzer
Process
Eth3 is equivalent to inbound-lo

Eth4 is equivalent to inbound-hi
dc11-5548-3# ethanalyzer local sniff-interface ?
inbound-hi
Inbound(high priority) interface
inbound-low Inbound(low priority) interface
mgmt
Management interface
PEX 8525
4 port PCIE
Switch
CLI view of in-band control plane data

dc11-5548-4# sh hardware internal cpu-mac inbound-hi counters
eth3
Link encap:Ethernet HWaddr 00:0D:EC:B2:0C:83
UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:2200
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:630 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:252 (252.0 b) TX bytes:213773 (208.7 KiB)
Base address:0x6020 Memory:fa4a0000-fa4c0000
eth4
NIC
0 1
NIC
0 1
Mgmt 0
Metric:1
Link encap:Ethernet HWaddr 00:0D:EC:B2:0C:84

UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:2200 Metric:1
RX bytes:33960760 (32.3 MiB) TX bytes:25825826 (24.6 MiB)
Base address:0x6000 Memory:fa440000-fa460000
Unified Port
Controller 2
Cisco Confidential
19
Nexus 5500 UPC (Gen 2) and Port Mapping

UPC-2 interfaces are indirectly
mapped to front panel ports
1/1
1/2
1/3
1/4
1/5
1/6
1/7
1/8
...
Mapping of ports to UPC-2 ASIC

The left column identifies the Ethernet
interface identifier, xgb1/8 = e1/8
0 1 2 3 4 5 6 7
UPC #0
Column three and four reflect the UPC

port that is associated with the
physical Ethernet port
UPC #7
...
nexus-5548# show hardware internal carmel all-ports

Carmel Port Info:
name
|log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer
-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+----xgb1/2 |1 |0 |0 -|b7 |dis|dn |0:0:f|0 |92 |0
|0 |1a001000|pass| 4.0b
xgb1/1 |0 |0 |1 -|b7 |dis|dn |1:1:f|1 |88 |0
|0 |1a000000|pass| 4.0b
xgb1/4 |3 |0 |2 -|b7 |dis|dn |2:2:f|2 |93 |0
|0 |1a003000|pass| 4.0b
xgb1/3 |2 |0 |3 -|b7 |dis|dn |3:3:f|3 |89 |0
|0 |1a002000|pass| 4.0b
xgb1/6 |5 |0 |4 -|b7 |dis|dn |4:4:f|4 |90 |0
|0 |1a005000|pass| 4.0b
xgb1/5 |4 |0 |5 -|b7 |dis|dn |5:5:f|5 |94 |0
|0 |1a004000|pass| 4.0b
xgb1/8 |7 |0 |6 -|b7 |dis|dn |6:6:f|6 |95 |0
|0 |1a007000|pass| 4.0b
<snip>
sup0
|32 |4 |4 -|b7 |en |dn |4:4:0|4 |62 |0
|0 |15020000|pass| 0.00
sup1
|33 |4 |5 -|b7 |en |dn |5:5:1|5 |59 |0
|0 |15010000|pass| 0.00
Cisco Confidential
20
5548UP/5596UP UPC (Gen-2) and Unified Ports

All versions of 5500 support 1/10G on all ports
5548UP, 5596UP and N55-M16UP (Expansion Module) support
Unified Port capability on all ports
1G Ethernet Copper/Fibre
10G DCB/FCoE Copper/Fibre
1/2/4/8G Fibre Channel
5548P
5548UP, 5596UP
& N55-M16UP
Unified Port
Controller 2
Ethernet
PHY
SFP+
Cage
Unified Port
Controller 2
Ethernet PHY
1/10G on all ports
SFP+
Cage
PHY removed, all MAC

and PHY functions
performed on UPC-2
1/10G Ethernet and
1/2/4/8G FC capable on all
ports
Cisco Confidential
21
5548UP/5596UP UPC (Gen-2) and Unified Ports

With the 5.0(3)N1 and later releases each module can define any
number of ports as Fibre Channel (1/2/4/8 G) or Ethernet (either 1G or
10G)
Initial SW releases supports only a continuous set of ports

configured as Ethernet or FC within each slot
Eth ports have to be the first set and they have to be one contiguous
range
FC ports have to be second set and they have to be contiguous as well
Future SW release will support per port dynamic configuration

n5k(config)# slot <slot-num>
n5k(config-slot)# port <port-range> type <fc | ethernet>
Slot 2 GEM
Eth Ports
Slot 1
Slot 3 GEM
Eth
FC
Slot 4 GEM
Eth
FC
FC Ports
Eth Ports
Cisco Confidential
22
Nexus 5500
Station (MAC) Table allocation
Nexus 5500 has a 32K Station table entries
4k reserved for multicast (Multicast MAC addresses)
3k assumed for hashing conflicts (very conservative)
Nexus 5500
UPC
Station Table
32k entries
25k effective Layer 2 unicast MAC address entries
4k entries for
IGMP
3k entries for potential hash collision space
25k effective MAC entries for unicast
Cisco Confidential
23
Nexus 5500 Packet Forwarding

Packet ForwardingCut Thru Switching
Packet Header
is serialized into
UPC
Nexus 5500 utilizes a Cut

Thru architecture when
possible
Bits are serialized in from the
ingress port until enough of
the packet header has been
received to perform a
forwarding and policy lookup
Forwarding
Packet is serialized
across Fabric once
forwarding decision
is made
Once a lookup decision has

been made and the fabric has
granted access to the egress
port bits are forwarded
through the fabric
Unified Crossbar
Fabric
Forwarding
Egress Queue is
only used if
Pause Frame
Received while
packet in-flight
Egress port performs any

header rewrite (e.g. CoS
marking) and MAC begins
serialization of bits out the
egress port
Packet Header Re-Write, MAC Learning

and then serialized out egress port
Cisco Confidential
24

Packet ForwardingCut-Through Switching
Nexus 5500 utilizes both cut-through and store and forward switching
Cut-through switching can only be performed when packets are being
sent out as fast as they are received over the fabric
1G to 1G always does store and forward because the fabric is running

0 1 2 3
at 10Gig
The fabric is designed to forward 10G packets in cut-through which
requires that 1G to 1G switching is store and forward mode
Direction of
Packet Flow
Ingress
10G
Ingress
10G
Ingress
1G
Ingress
1G
Unified Crossbar
Fabric
Unified Crossbar
Fabric
Unified Crossbar
Fabric
Unified Crossbar
Fabric
Egress
10G
Egress
1G
Egress
10G
Egress
1G
Cut-Through
Mode
Cut-Through
Mode
Store and Forward

Mode
Store and Forward

Mode
Cisco Confidential
25
For Your
Reference

Forwarding Mode Behavior (Cut-Through or Store and Forward)
Source Interface
Destination Interface
Switching Mode
10 GigabitEthernet
10 GigabitEthernet
Cut-Through
10 GigabitEthernet
1 GigabitEthernet
Cut-Through
1 GigabitEthernet
1 GigabitEthernet
Store-and-Forward
1 GigabitEthernet
10 GigabitEthernet
Store-and-Forward
FCoE
Fibre Channel
Cut-Through
Fibre Channel
FCoE
Store-and-Forward
Fibre Channel
Fibre Channel
Store-and-Forward
FCoE
FCoE
Cut-Through
Cisco Confidential
26

Packet Forwarding - Cut Through Switching
In Cut-Through switching frames are not
dropped due to bad CRC
Bad Fibre
Corrupt Frame
with original
CRC
Nexus 5500 implements a CRC stomp

mechanism to identify frames that have been
detected with a bad CRC upstream
A packet with a bad CRC is stomped, by
replacing the bad CRC with the original CRC
exclusive-ORd with the STOMP value
( a 1s inverse operation on the CRC)
In Cut Through switching frames with invalid
MTU (frames with a larger MTU than allowed)
are not dropped
Frames with a > MTU length are truncated

and have a stomped CRC included in
the frame
Ingress
UPC
Corrupt
Frame with
Stomped
CRC
Unified Crossbar
Fabric
Egress
UPC
Corrupt Frame
with Stomped
CRC
Cisco Confidential
27

Packet ForwardingCut Through Switching
Corrupt or Jumbo frames arriving inbound will
count against the Rx Jumbo or CRC counters
Corrupt or Jumbo frames exiting will be identified

via the Tx output error and Jumbo counters
Eth
1/39
0 1
dc11-5548-4# sh int eth 1/39
<snip>
RX
576 unicast packets 4813153 multicast packets 55273 broadcast packets
4869002 input packets 313150983 bytes
31 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun
0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
Ingress
UPC
Unified Crossbar
Fabric
Egress
UPC
Eth
2/4
dc11-5548-4# sh int eth 2/4

<snip>
TX
112 unicast packets 349327 multicast packets 56083 broadcast packets
405553 output packets 53600658 bytes
31 jumbo packets
31 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble
0 Tx pause
Cisco Confidential
28

Packet ForwardingCut Thru Switching
CRC and stomped frames are tracked internally
between ASICs within the switch as well as on the
interface to determine internal HW errors are
occurring
dc11-5548-4# show hardware internal carmel asic 2 counters interrupt
<snip>
Carmel 2 interrupt statistics:
Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
car_bm_port0_INT_err_ig_mtu_vio
|1f
|0
|1f
<snip>
dc11-5548-4# show hardware internal carmel asic 13 counters interrupt
<snip>
Carmel 13 interrupt statistics:
Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
car_fw2_INT_eg_pkt_err_cb_bm_eof_err
|1f
|0
|1
|0
car_fw2_INT_eg_pkt_err_eth_crc_stomp
|1f
|0
|1
|0
car_fw2_INT_eg_pkt_err_ip_pyld_len_err
|1f
|0
|1
|0
car_mm2_INT_rlp_tx_pkt_crc_err
|1f
|0
|1
|0
<snip>
Ingress
UPC
Unified Crossbar
Fabric
Egress
UPC
Cisco Confidential
29

Packet ForwardingIngress Queuing
Traffic is Queued on all ingress interface
buffers providing a cumulative scaling of
buffers for congested ports
In typical Data Center access

designs, multiple ingress
access ports transmit to a few
uplink ports
Nexus 5500 utilizes an
Ingress Queuing architecture
Packets are stored in ingress
buffers until egress port is
free to transmit
Ingress queuing provides an

additive effective
Egress Queue
0 is full, link
congested
The total queue size available

is equal to [number of ingress
ports x queue depth per port]
Statistically ingress queuing

provides the same
advantages as shared buffer
memory architectures
Cisco Confidential
30

Packet ForwardingVirtual Output Queues
Nexus 5500 use an 8 Queue QoS
model for unicast traffic
VoQ Eth VoQ Eth
1/20
1/8
Packet is able to
be sent to the
fabric for Eth 1/8
Packets
Queued for
Eth 1/20
Egress
Queue 0
is free
Egress
Queue 0
is full
To prevent Head of Line Blocking

(HOLB) Nexus 5500 use a Virtual
Output Queue (VoQ) Model
Each ingress port has a unique set
of 8 virtual output queues for every
egress port (1024 Ingress VOQs =
128 destinations * 8 classes on
every ingress port)
Unified Crossbar
Fabric
Eth 1/8
Traffic is Queued on the Ingress

buffer until the egress port is free to
transmit the packet
If Queue 0 is congested for any port

traffic then Queue 0 in all the other
ports is still able to be transmitted
Eth 1/20
Common shared buffer on ingress,

VoQ are pointer lists and not
physical buffers
Cisco Confidential
31
Nexus 5500 Series

VRF-Lite Support
Prior to 5.0(3)N1(1) , N5k support two VRFs
VRF management & VRF default
vPC Keepalive Dedicated VRF if

using data ports rather than mgmt
port for keepalive
With 5.0(3)N1(1) user can create additional VRFs

VRF-lite,
VRF aware Unicast -BGP/OSPF/RIP
VRF Aware Multicast

Hardware supports 1K VRF
Current Solution testing limit 64 VRFs
Similar to N7K if user data ports are used as

keepalive link, it is now recommended to create
dedicate VRF for keepalive link
interface Vlan123
vrf member vpc_keepalive
ip address 123.1.1.2/30
no shutdown
vpc domain 1
peer-keepalive destination 123.1.1.1 source 123.1.1.2 vrf vpc_keepalive
Cisco Confidential
32
Nexus 5000 & 5500
For Your
Reference
Reference
Nexus
5010
Nexus
5020
Nexus
5548P
Nexus
5548UP
Nexus
5596UP
520Gbps
1.04Tbps
960Gbps
960Gbps
1.92Tbps
1RU
2RU
1RU
1RU
2RU
1 Gigabit Ethernet Port Density
16
48
48
96
10 Gigabit Ethernet Port Density
26
52
48
48
96
8G Native Fibre Channel Port Density
12
16
48
96
~ 3.2us
~ 3.2us
~2.0us
~1.8us
~ 1.8us
512
512
4096
4096
4096
Product Features & Specs

Switch Fabric Throughput
Switch Footprint
Port-to-Port Latency
No. of VLANs
Layer 3 Capability
1 Gigabit Ethernet FEX Port

Scalability (L2 mode)
576
576
1152
1152
1152
10 Gigabit Ethernet FEX Port

Scalability (L2 mode)
384
384
768
768
768
40 Gigabit Ethernet Capable

Reversed Airflow
Presentation_ID
2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
33
Device Management
Cisco Confidential
34
Cisco Nexus 5500

Fundamentals
Config and Troubleshooting
Cisco Confidential
35
Fundamentals
Important Cisco NX-OS and Cisco IOS Differences
In Cisco NX-OS:
When you first log into the NX-OS, you go directly into EXEC mode.
Role Based Access Control (RBAC) determines a users permissions by

default. NX-OS 5.0(2a) introduced privilege levels and two-stage authentication
using an enable secret that can be enabled with the global feature privilege
configuration command.
By default, the admin user has network-admin rights that allow full read/write
access. Additional users can be created with very granular rights to permit or
deny specific CLI commands.
The Cisco NX-OS has a Setup Utility that allows a user to specify the system
defaults, perform basic configuration, and apply a pre-defined Control Plane
Policing (CoPP) security policy.
The Cisco NX-OS uses a feature based license model. An Enterprise Services,
Advanced Services, Transport Services, Scalable Feature and Enhanced Layer 2
license is required depending on the features required. Additional licenses may
be required in the future.
A 120 day license grace period is supported for testing, but features are
automatically removed from the running configuration after the expiration date is
reached. Some features such as Cisco Trustsec that require an Advanced
Services license cannot be configured with a grace period.
Cisco Confidential
36
Fundamentals (contd)
In Cisco NX-OS:
The Cisco NX-OS has the ability to enable and disable features such as OSPF,
BGP, etc using the feature configuration command. Configuration and
verification commands are not available until you enable the specific feature.
Interfaces are labeled in the configuration as Ethernet. There arent any speed
designations.
The Cisco NX-OS has two preconfigured VRF instances by default
(management, default). The management VRF is applied to the supervisor
module out-of-band Ethernet port (mgmt0), and the default VRF instance is
applied to all other I/O module Ethernet ports. The mgmt0 port is the only port
permitted in the management VRF instance and cannot be assigned to another
VRF instance.
SSHv2 server/client functionality is enabled by default. TELNET server
functionality is disabled by default. (The TELNET client is enabled by default and
cannot be disabled.)
VTY and Auxiliary port configurations do not show up in the default
configuration unless a parameter is modified (The Console port is included in
the default configuration). The VTY port supports 32 simultaneous sessions and
the timeout is disabled by default for all three port types
Cisco Confidential
37
Fundamentals (contd)
In Cisco NX-OS:
The Console and VTY ports always prompt the user for a username/password
pair for authentication before granting access to the CLI. The Cisco IOS applies
the login command to the Console and VTY ports by default to enable password
authentication (If the no login command is applied, a user can gain access
without a password.).
A user can execute show commands in configuration mode without using the
do command as in Cisco IOS Software.
When executing a show command, a user has several more options when
using the pipe (|) option such as grep for parsing the output, perl for activating a
script, and xml to format the output for network management applications.
Cisco Confidential
38
Fundamentals
Things You Should Know
The default administer user is predefined as admin. An admin user password
has to be specified when the system is powered up for the first time, or if the
running configuration is erased with the write erase command and system is
repowered.
The license grace-period can be disabled without any impact if the proper
license is installed for a feature within the 120 day grace period.
If you remove a feature with the global no feature configuration command, all
relevant commands related to that feature are removed from the running
configuration. Some features such as LaCP and vPC will not allow you to disable
the feature if they are configured.
The NX-OS uses a kickstart image and a system image. Both images are
identified in the configuration file as the kickstart and system boot variables. The
boot variables determine what version of NX-OS is loaded when the system is
powered on. (The kickstart and system boot variables have to be configured for
the same NX-OS version.)
The show running-config command accepts several options, such as OSPF,
BGP, etc that will display the runtime configuration for a specific feature.
The show tech command accepts several options that will display information
for a specific feature.
Cisco Confidential
39
Fundamentals
The NX-OS has a configuration checkpoint/rollback feature that should be
used when making changes to a production network. A checkpoint configuration
can be saved in EXEC mode with the global checkpoint command and the
rollback procedure can be executed with the rollback command.
Cisco Confidential
40
Fundamentals
Command Comparison: NX-OS vs IOS
Cisco IOS CLI
Cisco NX-OS CLI
Default User Prompt

c6500>
n5000#
Entering Configuration Mode

c6500# configure terminal
n5000# configure terminal
Saving the Running Config to the Startup Config (nvram)

c6500# write memory or
n5000# copy running-config startupc6500# copy running-config startup-config config
Erasing the startup config (nvram)

c6500# write erase
n5000# write erase
Cisco Confidential
41
Fundamentals
Command Comparison: NX-OS vs IOS (contd)
Cisco IOS CLI
Cisco NX-OS CLI
Installing a License
Cisco IOS Software does not require a
license file installation.
n5000# install license

bootflash:license_file.lic
Interface Naming Convention

interface Ethernet 1/1 interface
FastEthernet 1/1
interface GigabitEthernet 1/1
interface TenGigabitEthernet 1/1
interface Ethernet 1/1
Default VRF Configuration (management)

Cisco IOS Software doesnt enable VRFs
vrf context management
by default.
Cisco Confidential
42
Fundamentals
Cisco IOS CLI
Cisco NX-OS CLI
Configuring the Software Image Boot Variables
boot system flash sup-bootdisk:s72033ipservicesk9_wan-mz.122-33.SXH1.bin
boot kickstart bootflash:/n5000-uk9kickstart.5.0.3.N2.2.bin

boot system bootflash:/n5000uk9.5.0.3.N2.2.bin
Enabling Features
Cisco IOS Software does not have the
functionality to enable or disable features.
feature ospf
Enabling TELNET (SSH is recommended)

Cisco IOS Software enables TELNET by
default.
feature telnet
Cisco Confidential
43
Fundamentals
Cisco IOS CLI
Cisco NX-OS CLI
Configuring the Console Timeout

line console 0
exec-timeout 15 0 (minutes seconds)
login
line console
exec-timeout 15 (minutes only)
Configuring the VTY Timeout and Session Limit

line vty 0 9
exec-timeout 15 0 (minutes seconds)
login
line vty
session-limit 10
exec-timeout 15 (minutes only)
Cisco Confidential
44
Fundamentals
Troubleshooting and Verification Commands
Cisco NX-OS
Interface
Cisco IOS Software

Interface
show running-config
show running-config
show startup-config
show startup-config
show interface
show interface
show interface ethernet
show interface <int type>
<x/x>
show interface mgmt 0 show boot
show boot
show clock
show clock
show clock detail

show environment
show clock detail

show environment
Command Description
Displays the running configuration
Displays the startup configuration
Displays the status for all of the interfaces
Displays the status for a specific interface
Displays the status for the mgmt interface

Displays the current boot variables
Displays the system clock and time zone
configuration
Displays the summer-time configuration
Displays all environment parameters
Cisco Confidential
45
Fundamentals
Troubleshooting and Verification Commands (contd)
Cisco NX-OS
show environment
clock
show environment fan
show environment
power
show environment
temperature
-
Cisco IOS
Software
show environment
status clock
show environment
cooling fan-tray
show power
show environment
temperature
-
show feature
show log logfile
show log
show log nvram
show module
show module
show module uptime
Command Description
Displays clock status for A/B and active clock
Displays fan status
Displays power budget
Displays environment data
Displays the features and routing processes

enabled
Displays the local log
Displays persistent log messages (severity 0-2)
stored in NVRAM
Displays installed modules and their status
Displays how long each module has be powered
up
Cisco Confidential
46
Fundamentals
Cisco NX-OS
show process cpu
show process cpu
history
show process cpu
sorted
show system cores
show system
exception-info
show system
resources
Cisco IOS
Software
Command Description
show process cpu

show process cpu
history
show process cpu
sorted
-
Displays the processes running on the CPU

Displays the process history of the CPU in chart
form
show exception
Displays last exception log
show process cpu
Displays CPU and memory usage data
show system uptime -
show tech-support
show tech-support
show tech-support
<name>
show tech-support
<name>
Displays sorted processes running on the CPU

Displays the core dump files if present
Displays system and kernel start time (Displays

active supervisor uptime)
Displays system technical information for Cisco
TAC
Displays feature specific technical information for
Cisco TAC
Hint: Show proc cpu | ex 0.0
Cisco Confidential
47
Fundamentals
Cisco NX-OS
Cisco IOS
Software
show version
show version
show line
show line com1
show line console
show line console
connected
show terminal
show users
show line
show line console 0
Displays running software version, basic

hardware, CMP status and system uptime
Displays console and auxiliary port information
Displays auxiliary port information
Displays console port information
States if the console port is physically connected
show terminal
show users
Displays terminal settings

Displays current virtual terminal settings
Command Description
Cisco Confidential
48
Fundamentals
Cisco NX-OS
show vrf
show vrf <name>
show vrf <name>
detail
show vrf <name>
interface
show vrf default
show vrf detail
show vrf interface
show vrf
management
Cisco IOS
Software
Command Description
show ip vrf
show ip vrf <name>
Displays a list of all configured VRFs

Displays a specified VRF
show vrf detail <name> Displays details for a specified

-
Displays interface assignment for a specified VRF
show vrf detail

show ip vrf interface
Displays a summary of the default VRF

Displays details for all VRF's
Displays VRF interface assignment
Displays a summary of the management VRF
Cisco Confidential
49
Fundamentals
Cisco NX-OS
Cisco IOS
Command Description
Software
show license
show license brief
show license file <name>
show license host-id
show license usage

show license usage <licensetype>
Displays all license file information

Displays the license file names installed
Displays license contents based on a specified
name
Displays the chassis Host-ID used for creating a
license
Displays all licenses used by the system
Displays all licenses used by the system per type
Cisco Confidential
50
Cisco Nexus 5500

Interface
Cisco Confidential
51
Interfaces
In Cisco NX-OS:
SVI command-line interface (CLI) configuration and verification commands are
not available until you enable the SVI feature with the feature interface-vlan
command.
Only 802.1q trunks are supported, so the encapsulation command isn't
necessary when configuring a layer-2 switched trunk interface. (Cisco ISL is not
supported)
An IP subnet mask can be applied using /xx or xxx.xxx.xxx.xxx notation when
configuring an IP address on a layer-3 interface. The IP subnet mask is displayed
as /xx in the configuration and show interface command output regardless
which configuration method is used.
The CLI syntax for specifying multiple interfaces is different in Cisco NX-OS
Software. The range keyword has been omitted from the syntax (IE: interface
ethernet 1/1-2)
When monitoring interface statistics with the show interface CLI command, a
configurable load-interval can be configured per interface with the load-interval
counters command to specify sampling rates for bit-rate and packet-rate
statistics. The Cisco IOS Software supports the load-interval interface command,
but doesn't support multiple sampling rates.
A locator-LED (beacon) that allows remote-hands-support personnel to easily
identify a specific port. The beacon light can be enabled per interface in interface
configuration mode with the beacon CLI command.
Cisco Confidential
52
Interfaces (contd)
In Cisco NX-OS:
An administrator can configure port profiles as templates that can be applied
to a large number of interfaces to simplify the CLI configuration process. Port
profiles are "live" configuration templates, so modifications to a port profile are
automatically applied to the associated interfaces. Cisco IOS uses port macros
to simplify the CLI configuration process, but unlike Port Profiles they are
applied one time.
The out-of-band management ethernet port is configured with the interface
mgmt 0 CLI command.
Proxy ARP is disabled on all interfaces by default.
Cisco Confidential
53
Interfaces
The default port type is configurable for L3 routed or L2 switched in
the setup startup script. (L3 is the default port type prior to running the
script)
A layer-2 switched trunk port sends and receives traffic for all VLANs
by default (This is the same as Cisco IOS Software). Use the switchport
trunk allowed vlan interface CLI command to specify the VLANs allowed
on the trunk.
The clear counters interface ethernet <x/x> CLI command resets the
counters for a specific interface.
An interface configuration can be reset to its default values with the
default interface <x/x> global configuration command.
Cisco Confidential
54
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a Routed Interface

interface gigabitethernet 1/1
interface ethernet 1/1
ip address 192.168.1.1 255.255.255.0
ip address 192.168.1.1/24
no shutdown
no shutdown
Configuring a Switched Interface (VLAN 10)

vlan 10
vlan 10
switchport
switchport
switchport mode access
switchport mode access
switchport access vlan 10
no shutdown
no shutdown
Cisco Confidential
55
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a Switched Virtual Interface (SVI)

ability to enable or disable SVI interfaces
using the feature command.
feature interface-vlan
interface vlan 10
interface vlan 10
ip address 192.168.1.1 255.255.255.0
ip address 192.168.1.1./24
no shutdown
no shutdown
Cisco Confidential
56
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a Switched Trunk Interface

interface GigabitEthernet 1/1
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport trunk native vlan 2
switchport trunk allowed vlan 10,20
no shutdown
no shutdown
Cisco Confidential
57
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a Routed Trunk Sub-Interface
no switchport
no switchport
no shutdown
no shutdown
interface gigabitethernet1/1.10
interface ethernet 1/1.10
encapsulation dot1Q 10
encapsulation dot1q 10
ip address 192.168.1.1 255.255.255.0
ip address 192.168.1.1/24
no shutdown
no shutdown
Cisco Confidential
58
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring Multiple Interfaces

(Examples)
interface range gigabitethernet 1/1-2
or
interface range gigabitethernet 1/1,
gigabitethernet 2/1
interface ethernet 1/1-1

or
interface ethernet 1/1, ethernet 2/1
Configuring the Interface Locator-LED

(Beacon)
ability to enable a located-led per interface. beacon
Cisco Confidential
59
Interfaces
Cisco IOS CLI
Cisco NX-OS CLI
Configuring Port Profiles

port-profile type ethernet Email-Template
switchport
spanning-tree port type edge

ability to configure port profiles.
no shutdown
description Email Server Port
state enabled
interface ethernet 2/1-48
inherit port-profile Email-Template
Cisco Confidential
60
Interfaces
Cisco NX-OS
Interface
Cisco IOS Software

Interface
show interface
show interface
Displays the status and statistics for all

interfaces or a specific interface
Displays the status and statistics for a
FEX host interface
Displays a brief list of the interfaces (type,
mode, status, speed, MTU)
show interface ethernet

<x/x/x>
show interface brief
show interface
capabilities
show interface
counters
show interface
description
Command Description
show interface capabilities Displays interface capabilities
Displays interface counters (input/output

unicast, multicast & broadcast)
Displays all interfaces with configured
show interface description
descriptions
Displays status and statistics for a specific
show interface ethernet show interface ethernet
interface
show interface fexDisplays FEX fabric interface status
fabric
show interface counters
Cisco Confidential
61
Interfaces
Cisco NX-OS Interface
Cisco IOS Software

Command Description
Interface
show interface flowcontrol
show interface
flowcontrol
Displays Flow Control (802.1p) status

and state for all interfaces
show interface loopback
show interface loopback
Displays status and statistics for a

specific loopback interface
show interface macaddress
Displays all interfaces and their

associated MAC Addresses
show interface mgmt
Displays status and statistics for the

management interface located on the
supervisor
show interface portchannel
show interface portchannel

specific port-channel
show interface priorityflow-control
Displays PFC information
show interface pruning
show interface pruning
Displays trunk interfaces VTP pruning

information
show interface snmpifindex
Displays SNMP interface index
Cisco Confidential
62
Interfaces
Cisco IOS Software

Interface
Command Description
show interface status
show interface status
Displays all interfaces and their current

status
show interface switchport show interface switchport
Displays a list of all interfaces that are

configured as switchports
show interface transceiver show interface transceiver
Displays a list of all interfaces and

optic information (calibrations, details)
show interface trunk
Displays a list of all interfaces

configured as trunks
show interface trunk
show interface tunnel <#> show interface tunnel <#>

specific tunnel interface
show interface vlan <#>

specific VLAN interface
show interface vlan <#>
Cisco Confidential
63
Interfaces
Cisco IOS
Software
Interface
Command Description
show port-profile
Displays all port profile information
show port-profile brief
Displays brief port profile information
show port-profile expandinterface
Displays active profile configuration applied to an

interface
show port-profile name
Displays specific port profile
show port-profile sync-status -
Displays interfaces out of sync with port profiles
show port-profile usage
Displays interfaces inherited to a port profile
Cisco Confidential
64
CLI Overview
The Cisco NX-OX CLI shares a lot of concepts as Cisco IOS software, so initial
configuration is very simple. The commands can be abbreviated, the ? provides
online help, and the <TAB> key auto-fills command options.
User Exec Mode:
n5500#
Default prompt - Type exit to log out
Entering Configuration Mode:

n5500# configure terminal
n5500(config)#
Show Running & Startup Configuration:

n5500# show running-config
n5500# show startup-config
Several additional options exist to view the

configuration related to a specific feature
Saving Running Configuration to Startup:

n5500# copy running-config startup-config
No write memory command
Erasing the Startup Configuration:

n5500# write erase
User is prompted to continue
Attaching to a Module:
n5500# attach module 1
Attaching to module 1 ...
module-1#
Type exit or $ to log out of the module

Cisco Confidential
65
Enabling NX-OS Features

The Cisco NX-OS provides the capability to enable and disable features using
the feature command. Configuration CLI and show commands are not
available(displayed) for a feature if it isnt enabled.
n5500(config)# feature ?
bgp
cts
dhcp
dot1x
eigrp
eou
glbp
hsrp
interface-vlan
isis
lacp
msdp
netflow
ospf
ospfv3
pbr
pim
pim6
port-security
private-vlan
rip
scheduler
ssh
tacacs+
telnet
tunnel
udld
vpc
vrrp
vtp
wccp
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Enable/Disable
Border Gateway Protocol (BGP)

CTS
DHCP Snooping
dot1x
Enhanced Interior Gateway Routing Protocol (EIGRP)
eou(l2nac)
Gateway Load Balancing Protocol (GLBP)
Hot Standby Router Protocol (HSRP)
interface vlan
IS-IS Unicast Routing Protocol (IS-IS)
LACP
Multicast Source Discovery Protocol (MSDP)
NetFlow
Open Shortest Path First Protocol (OSPF)
Open Shortest Path First Version 3 Protocol(OSPFv3)
Policy Based Routing(PBR)
Protocol Independent Multicast (PIM)
Protocol Independent Multicast (PIM) for IPv6
port-security
private-vlan
Routing Information Protocol (RIP)
scheduler
ssh
tacacs+
telnet
Tunnel Manager
UDLD
VPC (Virtual Port Channel)
Virtual Router Redundancy Protocol (VRRP)
VTP
Web Cache Communication Protocol (WCCP)
Cisco Confidential
66
Verifying Software Version

Use the show version command to obtain general hardware/software information.
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
Software
BIOS:
version 1.8.0
loader:
version N/A
kickstart: version 5.0(2)N1(1)
system:
version 5.0(2)N1(1)
power-seq: version v3.0, gem: version v1.0
uC:
version v1.0.0.14
BIOS compile time:
10/06/2010
kickstart image file is: bootflash:/n5500-uk9 kickstart.5.0.3.N2.2.bin
kickstart compile time: 10/15/2010 0:00:00 [10/15/2010 04:00:43]
system image file is:
bootflash:/n5500-uk9.5.0.3.N2.2.bin
system compile time:
10/15/2010 0:00:00 [10/15/2010 05:34:05]
Hardware
cisco Nexus5548 Chassis ("O2 32X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU
with 8299548 kB of memory.
Processor Board ID JAF1445APSP
Device name: USPA833NEXUS5548-01
bootflash:
2007040 kB
Kernel uptime is 143 day(s), 1 hour(s), 1 minute(s), 8 second(s)
Last reset
Reason: Unknown
System version: 5.0(2)N1(1)
Service:
NX-OS software
NX-OS versions
File locations
System DRAM (KB)

Bootflash (Size)
Expansion flash
System uptime
plugin
Core Plugin, Ethernet Plugin
`<truncated>
Cisco Confidential
67
Basic Configuration: Configuring the

Management VRF Context
1. Configuring switch name
2. Configuring the management interface
3. Configuring the management VRF context
switch# configure
switch(config)# switchname N5K
N5K(config)# interface mgmt0
N5K(config-if)# ip address 172.18.217.80 255.255.255.0
N5K(config-if)# no shut
N5K(config-if)# exit
N5K(config)# vrf context management
N5K(config-vrf)# ip route 0.0.0.0/0 172.18.217.1/24
Cisco Confidential
68
ACL on mgmt0 / VTY

N5k supports mgmt0 for OOB Mgmt
N5k supports SVI for inband management
- Enable feature interface-vlan
inter mgmt0
ip access-group xx in/out
line vty
Ip access-class xx in/out
Presentation_ID
Cisco Confidential
Management Interface Verification

The following commands verify the management VRF routing table as well as
the interface statistics.
VRF management Routing Table:
n5500# show ip route vrf management
IP Route Table for VRF "management"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
Routing table for management VRF
management VRF default route
0.0.0.0/0, 1 ucast next-hops, 0 mcast next-hops

*via 159.142.1.10, mgmt0, [1/0], 00:01:27, static
Management Interface Statistics:

n5500# show interface mgmt 0
mgmt0 is up
Hardware is GigabitEthernet, address is 001b.54c0.feb8 (bia 001b.54c0.feb8)
Internet Address is 159.142.1.1/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
full-duplex, 1000 Mb/s
Auto-Negotiation is turned on
30 minute input rate 1102814 bytes/sec, 16317 packets/sec
30 minute output rate 42224 bytes/sec, 251 packets/sec
Rx
16422 input packets 6 unicast packets 11734 multicast packets
4682 broadcast packets 1110256 bytes
Tx
254 output packets 164 unicast packets 74 multicast packets
16 broadcast packets 42547 bytes
Cisco Confidential
70
Configuring User Accounts
Creating user accounts
N5K# configure
N5K(config)# username admin password cae123rtp role network-admin
N5K(config)# username operator password oper1234 role network-operator
user:operator is reserved
N5K(config)# username paul password oper1234 role network-operator
N5K(config)# sh run | incl username
username admin password 5 $1$6KdEue0H$vexPxI/qjJNZrRmg8nsIo. role networkadmin
username paul password 5 $1$PvSqwWxh$gxL46OnByOVe8ZC5zOj0b. role networkoperator
N5K(config)# sh run | incl snmp-server
snmp-server user paul network-operator auth md5
0x72fffc91ff1de08468c5b1c3c0acd1
11 priv 0x72fffc91ff1de08468c5b1c3c0acd111 localizedkey
snmp-server user admin network-admin auth md5
0x25bb8f4349b3217abb2672edc84981ac
priv 0x25bb8f4349b3217abb2672edc84981ac localizedkey
Cisco Confidential
71
Configuring Administrative Access:

RSA and SSH
Configure RSA keys (may have to disable SSH server first)
Enable the SSH server process (enabled by default)
Verify that the SSH server is running
N5K(config)# ssh key rsa 1024 force

deleting old rsa key.....
generating rsa key(1024 bits).....
.
generated rsa key
N5K(config)# ssh server enable
N5K(config)# sh ssh server
ssh is enabled
version 2 enabled
Cisco Confidential
72
Role Based Access Control (RBAC)

Users and associated roles are created to secure access to the Cisco NX-OS.
RBAC allows you to create a granular security policy that limits a users access to
the device, so they can only perform the actions they are authorized for. RBAC
can work in conjunction with AAA.
Default User
admin
Default User Roles

network-admin
network-operator
vdc-admin
vdc-operator
User Description
admin user with network-admin role
Role Description
read / write access for default VDC

read access for the default VDC
read / write access for a VDC

read access for a VDC
Note: a user is assigned to the network-operator role if a role isnt specified when the user is created.
Cisco Confidential
73
RBAC Configuration Example

The following example illustrates how to create a role with multiple rules and
assign it to a user.
Only a user with the network-admin or vdc-admin role can create users and
roles.
Create a Role:
n5500(config)# role name ospf-admin
n5500(config-role)# rule 1 permit command show interface *
n5500(config-role)# rule 2 permit command show running-config
n5500(config-role)# rule 3 permit read-write feature router-ospf
n5500(config-role)# rule 4 permit command config t ; interface *
n5500(config-role)# rule 5 permit command copy running-config startup-config
Allow a user to
configure OSPF, verify
the configuration and
save the runningconfiguration
Create a User and Assign a Role:

n5500(config)# username ospf-admin password xxxxxxxx role ospf-admin
If a users role is modified, the changes do not take effect until that user logs
out and back into the system.
Cisco Confidential
74
Logging Configuration and Verification

Multiple logging servers can be enabled with different severity levels. Use the
use-vrf option to specify the VRF where the Syslog server resides.
Logging (Syslog) Configuration:
Specify the logging severity
level per server
n5500(config)# logging server 159.142.1.10 ?

<CR>
<0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug
Specify the VRF the server

should use to send logs
n5500(config)# logging server 159.142.1.10 7 use-vrf management
Logging (Syslog) Verification:
Other common options (logfile & nvram)
n5500# show logging info

Logging console:
enabled (Severity: critical)
Logging monitor:
enabled (Severity: notifications)
Logging linecard:
Logging timestamp:
Seconds
Logging loopback :
disabled
Logging server:
enabled
{159.142.1.10}
server severity:
debugging
server facility:
local7
server VRF:
management
Logging logflash:
Logging logfile:
enabled
Name - messages: Severity - notifications Size 4194304
<Text Omitted>
n5500# clear logging logfile
Syslog Server 159.142.1.10 is enabled

Configured in the management VRF
Clears the
logfile
Cisco Confidential
75
SNMP Configuration and Verification

Basic SNMPv1 configuration. SNMP versions 2c and 3 are also supported.
Community String (RO or RW)
n5500(config)#
n5500(config)#
n5500(config)#
n5500(config)#
n5500(config)#
snmp-server community secret ro

snmp-server host 159.142.1.10 version 1 secret
snmp-server host 159.142.1.10 use_vrf management
snmp-server enable traps
snmp-server contact Lab Manager
Enable default Traps
n5500# show snmp host

-------------------------------------------------------------------------------Host
Port Version Level Type SecName
-------------------------------------------------------------------------------159.142.1.10
162 v1
noauth trap secret
Configured SNMP host
The VRF the host is associated with
Use VRF: management

------------------------------------------------------------------n5500# show snmp trap
Trap type
Enabled
--------------aaa server-state-change Yes
callhome
No
entity fru
Yes
license
Yes
snmp authentication
Yes
link
Yes
bridge topologychange
No
bridge newroot
No
stpx inconsistency
No
stpx loop-inconsistency
No
stpx root-inconsistency
No
Configured Host
V1 is the default
SNMP Traps enabled by default
Cisco Confidential
76
SNMP Community ACL Configuration

An extended ACL can be applied to an SNMP community string to limit access to
SNMP data. An ACL can be applied for read-only and read-write community strings.
The following example restricts SNMP access to one host when accessing the IP
address associated to the mgmt 0 interface.
Configuration:
n5500(config)# interface mgmt0
n5500(config-if)# ip address 10.20.1.21/24
Define an ACL UDP port 161
n5500(config)# ip access-list snmp-ro

n5500(config-acl)# permit udp 10.20.0.20/32 10.20.1.21/32 eq snmp
Define the SNMP community

string and associate the ACL
n5500(config)# snmp-server community cisco123 ro

n5500(config)# snmp-server community cisco123 use-acl snmp-ro
Verification:
n5500# show snmp community
Community
--------cisco123
Group / Access
context acl_filter
-------------------- ---------network-operator
snmp-ro
snmp-ro ACL associated with the

cisco123 community string
Cisco Confidential
77
TACACS+ Configuration and Verification

A basic AAA/TACACS+ configuration is illustrated below that is very similar to the
previous RADIUS configuration. The tacacs+ feature needs be enabled first.
TACACS+ supports command and config-command AAA authorization.
TACACS+ Configuration:
n5500(config)# feature tacacs+
n5500(config)# tacacs-server host 159.142.1.10
warning: no key is configured for the host
n5500(config)# tacacs-server key cisco123
Enable the TACACS+ feature first!

Specify which VRF to use for TACACS+
n5500(config)# aaa group server tacacs+ AAA-Server

n5500(config-tacacs+)# use-vrf management
n5500(config-tacacs+)# server 159.142.1.10
n5500(config)# aaa authentication login default group AAA-Server
n5500(config)# aaa authorization commands default group AAA-Server local
n5500(config)# aaa authorization config-commands default group AAA-Server local
n5500(config)# aaa accounting default group AAA-Server
Optional: Enable AAA

command & configcommand authorization
with local fallback
Optional: Enable AAA Accounting
TACACS+ Server Verification:

n5500# show tacacs
Global TACACS+ shared secret:********
timeout value:5
deadtime value:0
total number of servers:1
following TACACS+ servers are configured:
159.142.1.10:
available on port:49
n5500# show tacacs groups

total number of groups:1
following TACACS+ server groups are configured:
group AAA-Server:
server 159.142.1.10 on port 49
deadtime is 0
vrf is management
Cisco Confidential
78
Cisco Nexus 5500

AAA, RADIUS, and TACACS+
Cisco Confidential
79
AAA, RADIUS and TACACS+

In Cisco NX-OS:
TACACS+ command-line interface (CLI) configuration and verification commands are not
available until you enable the TACACS+ feature with the feature tacacs+ command (The
RADIUS feature is enabled by default and cannot be disabled).
The aaa new-model command is not required to enable AAA authentication,

authorization, or accounting.
The RADIUS vendor-specific attributes (VSA) feature is enabled by default. Cisco IOS
Software requires the global radius-server vsa send configuration command to enable IETF
attribute 26.
Local command authorization can be performed using privilege-levels or role-based

access control (RBAC) without a AAA server. Local privilege-levels or RBAC roles can be
associated to users configured on the AAA server using VSAs (TACACS+ supports
command authorization that can be configured on the AAA server).
If a configured AAA server is not available for authentication, the local database
(username/password) is automatically used for device access.
The RADIUS and TACACS+ host keys are Triple Data Encryption Standard (3DES)
encrypted in the configuration. Cisco IOS Software requires the service password command.
Cisco Confidential
80

In Cisco NX-OS:
All configuration commands are recorded in a local log (NVRAM) with user and time
stamp information by default (no AAA configuration required). The log can be viewed with
the show accounting log command.
The aaa accounting default command enables accounting for start and stop records as
well as command accounting (Exec mode and configuration mode). Cisco IOS Software
requires additional aaa accounting commands to enable both types of accounting.
Cisco Confidential
81

Configuring a protocol for AAA is a multi-step configuration process: Define
the server(s), create the server group, and associate the server group to the
required AAA commands.
If you remove a feature such as TACACS+ with the global no feature <name>
command, all relevant configuration information is removed from the runningconfiguration for the specified feature.
AAA server groups are associated with the default Virtual Route Forwarding
(VRF) instance by default. Associate the proper VRF instance with the AAA
server group if you are using the management port on the supervisor module or
if the AAA server is in a non-default VRF instance.
A RADIUS and TACACS+ source interface can be configured globally or per
AAA server group to specify the source IP address for packets destined to
remote AAA services.
RADIUS and TACACS+ server keys can be specified for a group of servers or
per individual server.
Cisco Confidential
82

By default, RADIUS uses UDP ports 1812 (authentication) and 1813
(accounting), and TACACS+ uses TCP port 49. All server ports can be configured
to use different values.
Directed server requests are enabled by default for RADIUS and TACACS+.
The local option can be used with AAA authorization to fallback to local
privilege-levels or RBAC in the event a AAA server is not available for command
authorization.
RADIUS and TACACS+ support global server test monitoring (Per server
monitoring takes precedence over global monitoring).
Use the show running-config command with the AAA, radius or tacacs+
option to display the running configuration for a specific feature.
Cisco Confidential
83

Cisco IOS CLI
Cisco NX-OS CLI
Configuring a RADIUS Server with a Key

radius-server host 192.168.1.1 key cisco123
radius-server host 192.168.1.1 key 7

"fewhg123" (7=encrypted or 0=cleartext)
Specifying Non defualt RADIUS UDP Ports

radius-server host 192.16.1.1 auth-port 1645 radius-server 192.168.1.1 auth-port 1645
acct-port 1646
acct-port 1646
Specifying the RADIUS Timeout Value (Global)

radius-server host 192.168.1.1 timeout 10
radius-server timeout 10
Specifying the RADIUS Source Interface (Global)

ip radius source-interface loopback0
ip radius source-interface loopback0
Cisco Confidential
84

Cisco IOS CLI
Cisco NX-OS CLI
Enabling TACACS+
Cisco IOS Software does not have the ability to
feature tacacs+
enable or disable TACACS+.
Configuring a TACACS+ Server with a Key

tacacs-server host 192.168.1.1 key cisco123
tacacs-server host 192.168.1.1 key 7

"fewhg123" (7=encrypted or 0=cleartext)
Specifying a Nondefualt TACACS+ TCP Port

tacacs-server host 192.168.1.1 port 85
tacacs-server host 192.168.1.1 port 85
Specifying the TACACS+ Timeout Value (Global)

tacacs-server timeout 10
tacacs-server timeout 10
Specifying the TACACS+ Source Interface (Global)

ip tacacs source-interface loopback0
ip tacacs source-interface loopback0
Cisco Confidential
85

Cisco IOS CLI
Cisco NX-OS CLI
Configuring an AAA Server Group (RADIUS)

aaa group server radius AAA-Servers
server 192.168.1.1

server 192.168.1.1
Configuring an AAA Server Group for a VRF Instance (RADIUS)

server 192.168.1.1
ip vrf forwarding management

server 192.168.1.1
use-vrf management
Cisco Confidential
86

Cisco IOS CLI
Cisco NX-OS CLI
Configuring the AAA Server Group Dead Time (RADIUS)

deadtime 5

deadtime 5
Configuring an AAA Server Group (TACACS+)

aaa group server tacacs+ AAA-Servers
server 192.168.1.1
aaa group server tacacs+ AAA-Servers

server 192.168.1.1
Enabling AAA Authentication with an AAA Server Group

aaa new-model
aaa authentication login default group AAAServers
aaa authentication login default group

AAA-Servers
Cisco Confidential
87

Cisco IOS CLI
Cisco NX-OS CLI
Enabling AAA Authorization with an AAA Server Group

aaa new-model
aaa authorization config-commands
aaa authorization commands 1 default group
AAA-Servers
aaa authorization
config-commands default group AAA-Servers
aaa authorization commands default group AAAServers
Enabling AAA Accounting with an AAA Server Group

aaa new-model
aaa accounting exec default start-stop group
AAA-Servers
aaa accounting default group AAA-Servers
Cisco Confidential
88

Cisco IOS
Command Description
Software Interface
show aaa accounting
show aaa authentication

login ascii-authentication

login chap

login error-enable
login mschap
login mschapv2
show aaa authorization
show aaa groups
show aaa users
show aaa user
Displays the status of AAA accounting

Displays the default and console login
methods
Displays the status of ascii authentication;
enabled or disabled
Displays the status of the Challenge
Handshake authentication protocol (CHAP);
enabled or disabled
Displays the login error message status;
enabled or disabled.
Displays the status of Microsoft CHAP (MSCHAP); enabled or disabled.
Displays the status of MS-CHAPv2; enabled
or disabled)
Displays the AAA authorization configuration
Displays the AAA groups that are configured
Displays the AAA users that authenticated
remotely
Cisco Confidential
89

Cisco IOS Software

Command Description
Interface
Displays the local AAA configuration

accounting log
Displays the RADIUS server configuration
show radius-server
for all servers
show radius-server
Displays a specific RADIUS server
<x.x.x.x>
configuration
show radius-server
Displays the status of the directed-request
directed-request
feature (enabled or disabled)
show radius-server groups show radius server-group Displays RADIUS server groups
show radius-server sorted Displays RADIUS servers sorted by name
show radius-server
Displays RADIUS statistics for a specific
show radius statistics
statistics <x.x.x.x>
server
show accounting log
Cisco Confidential
90

Cisco IOS Software

Command Description
Interface
show tacacs-server
show tacacs
show tacacs-server
<x.x.x.x>
show tacacs-server
directed-request
show tacacs-server groups show tacacs-server sorted show tacacs-server
statistics <x.x.x.x>
show user-account
show users
show users
Displays the TACACS+ server

configuration for all servers
Displays a specific TACACS+ server
configuration
Displays the status of the directed-request
feature (enabled or disabled)
Displays TACACS+ server groups
Displays TACACS+ servers sorted by
name
Displays TACACS+ statistics for a specific
server
Displays a list of locally configured users
Displays the users who are logged in
Cisco Confidential
91
Network Time Protocol Configuration

The Network Time Protocol (NTP) can be used to synchronize the clock
from a reliable time source. The NX-OS can be configured to
synchronize its time with a peer or a server. The NX-OS cannot act
as an NTP server for non-peering clients.
NTP Configuration Options:

n5500(config)# ntp ?
peer NTP Peer address
server NTP server address
source Source of NTP packets
Configures the NX-OS to sync its clock from an NTP server
NTP Configuration:
n5500(config)# ntp server 10.20.8.129 prefer use-vrf management
n5500(config)# ntp server 10.20.8.130 use-vrf management
n5500(config)# ntp source 10.205.225.43
Timezone Configuration:
n5500(config)# clock ?
summer-time Configure summer (daylight savings) time
timezone Configure time zone
Use the prefer option to specify the

primary NTP Server
Specify the source IP address (Optional)
The default time zone is UTC

Cisco Confidential
92
Network Time Protocol Verification

n5500# show ntp peers
-------------------------------------------------Peer IP Address
Serv/Peer
-------------------------------------------------10.20.8.129
Server (configured)
10.20.8.130
Server (configured)
Configured NTP servers
n5500# show ntp peer-status

Total peers : 2
* - selected for sync, + - peer mode(active),
- - peer mode(passive), = - polled in client mode
remote
local
st poll reach delay
vrf
--------------------------------------------------------------------------------------------*10.20.8.129
10.205.225.43 2 64
17 0.00142 management
=10.20.8.130 10.205.225.43 2 64
17 0.00133 management
n5500# show ntp statistics peer ipaddr 10.20.8.129

remote host:
10.20.8.129
local interface:
10.205.225.43
time last received:
30s
time until next send:
21s
reachability change:
190s
packets sent:
26
packets received:
25
bad authentication:
0
bogus origin:
0
duplicate:
0
bad dispersion:
0
bad reference time:
0
candidate order:
6
Preferred Peer selected for sync
NTP packets exchanged with NTP server
Cisco Confidential
93
In-Service Software Upgrade
Cisco Confidential
94
Nexus 5500 ISSU

Differences from Nexus 7000
Although the high-level steps associated with ISSU is common between both the Nexus
5500 and Nexus 7000 platforms, the 2 platforms differ in key fundamental ways. The
Nexus 5500 supports a single supervisor ISSU architecture and performs a stateful
restart of the entire operating system upon execution, whilst leaving data plane
forwarding intact
During this time, control plane functions of the switch undergoing ISSU are temporarily
suspended, and configuration changes disallowed. The control plane will be brought
online again within 80 seconds to allow protocol communications again.
Cisco Confidential
95
Nexus 5500 ISSU

Preconditions
The ISSU process is executed through the installer, and certain conditions must be satisfied
before it can proceed.
Restriction on Configuration changes
Restriction on Topology Changes
CLI and SNMP config change requests are

denied during ISSU operations
Network/Topology changes like STP, FC

Fabric changes that affect zoning, FSPF,
domain manager, Module insertion are not
expected during ISSU operation
Cisco Confidential
96
Nexus 5500 ISSU

VPC Topologies
VPC topologies are fully supported with ISSU. Three types of VPC topologies are supported
for the Nexus 5500 and Nexus 2000 FEX.
Blade or Access Switch
FEX Active-Active
FEX Straight-Through
Throughout the ISSU process, VPC roles will remain intact. It is the peer switchs responsibility to hold
onto its state until ISSU process is complete
Cisco Confidential
97
Nexus 5500 ISSU

STP Topologies
There are some restrictions that need to be placed on Ethernet STP topologies if a nondisruptive ISSU process is required:
1
The Nexus 5500/2000 switch undergoing ISSU must be a leaf on the spanning tree.
The switch should not be a root switch or have any designated non-edge ports in the
STP topology
Bridge Assurance must be disabled for non-disruptive ISSU
STP Primary Root
STP Secondary Root

Non-Disruptive ISSU
Not OK Here
Non-Disruptive ISSU
OK Here
STP Edge Ports
STP Edge Ports
Cisco Confidential
98
Cisco Confidential Do Not Distribute
Nexus 5500 ISSU

Management Services
Prior to the switch being reset for ISSU, inbound-hi and management ports are brought
down, and are brought back up after ISSU completes. Services that depend on inbound-hi
and management ports are impacted during this time
Telnet/SSH
The Telnet/SSH daemons rely on the startup configs of the switch. As the device is
restarted, all Telnet/SSH sessions will be disconnected and need to be re-established
after ISSU completes
AAA/RADIUS
Applications that leverage the AAA Service (such as Login) will be disabled during ISSU
process. Since all Network Management services are disabled during this time, this
behavior is consistent.
HTTP
The HTTP sessions to the Switch will be disconnected during ISSU reboot. After ISSU
reboot, the HTTPd will be restarted and switch will accept HTTP sessions after ISSU
reboot.
NTP
The ntp sessions to and from the switch are disrupted during ISSU reboot. After ISSU
reboot, ntp session will be re-established based on the saved startup configuration.
Telnet/SSH will be dropped, perform ISSU from the

Console!
2009, Cisco Systems, Inc. All rights reserved.
99
ISSU Requirements
Ensure you have enough space to store the images on bootflash:
Ensure no power interruptions occur during any install procedure.
Ensure the system and kickstart images are compatible with each
other.
Run only one installation on a switch at a time ***
Do not issue another command while running the installation
If the fabric extenders are not compatible with the software image
you install on the Nexus 5500 switch, some traffic disruption may
occur depending on the configuration. The install all command
output identifies these commands.
Cisco Confidential
100
Pre-ISSU Check #1
DCN-N5K1# show spanning issu-impact
For ISSU to Proceed, Check the Following Criteria :
1. No Topology change must be active in any STP instance
2. Bridge assurance(BA) should not be active on any port
(except vPC peer-link)
3. There should not be any Non Edge Designated
Forwarding port (except vPC peer-link)
4. ISSU criteria must be met on the VPC Peer Switch as well
Following are the statistics on this switch
No Active Topology change Found!
Criteria 1 PASSED !!
No Ports with BA Enabled Found!
Criteria 2 PASSED!!
List of all the Non-Edge Ports

Port
VLAN Role Sts Tree Type Instance
---------------- ---- ---- --- --------- --------Ethernet1/1
49 Desg FWD PVRST
49
port-channel20 50 Desg FWD PVRST
50
51
52
77
201
Criteria 3 FAILED !!
ISSU Cannot Proceed! Change the above Config
Spanning Tree designated ports

present, upgrade will be
disruptive
Cisco Confidential
101
Pre-ISSU Check #2
show install all impact kickstart <image> system <image>
Displays information describing the impact of the upgrade on
each fabric extender including details such as upgrade image
versions.
This command will also display if the upgrade is
disruptive/non-disruptive and the reason why.
Compatibility check is done:
Module bootable
Impact
------ -------- -------------1
yes non-disruptive
100
yes non-disruptive
FEX
Install-type
-----------reset
rolling
Installation will be non-disruptive
Reason
------
rolling upgrade
means each FEX
updated one at a time
Cisco Confidential
102
Layer 2 Switching
Cisco Confidential
103
VLAN Scalability
The Cisco Nexus 5500 Series
Hardware supports 4096
VLANs
The NXOS reserved VLAN

range doesnt match the
Catalyst reserved VLAN range
Software allows users to

configure the following VLANs:
1 3967 and 4048 to 4093 =
4012 VLANs
But the internal NXOS VLANs

can be mapped to an MST
instance
This is true with or without vPC
Future optimization allows to

shift the reserved VLAN range
Cisco Confidential
104
NXOS Reserved Range

The Cisco Nexus 5500 Series Hardware supports 4096 VLANs
NXOS Reserves the following VLANs:
3968-4031 To support Multicast
4032
Online diagnostics vlan1 - used for internal diags
4033
Online diagnostics vlan2
4034
4035
4036-4047 Reserved - for future use, not used right now
4094 Reserved - for ERSPAN
Out of the NXOS Range, Nexus 5500 Series use:

4041 RSVD_VLAN_DOT1Q_TAG_NATIVE 4041
4042 - for communication with FEX
4043 for communication with adapter
Cisco Confidential
105
VLAN Configuration
VLANs provide layer-2 separation boundaries for unicast, multicast, and broadcast packets.
The Cisco Nexus 5500 Series Hardware supports 4096 VLANs.
Configuration:
n5500(config)# vlan 10
n5500(config-vlan)# ?
ip
Configure IP features
media
Media type of the VLAN
name
Ascii name of the VLAN
no
Negate a command or set its defaults
remote-span Enable remote span VLAN
service-policy Configure service policy for an interface
shutdown
Shutdown VLAN switching
state
Operational state of the VLAN
n5500(config-vlan)# name email-vlan
Created VLAN 10
VLAN 10 name for future reference
n5500(config-vlan)# vlan 11-19

n5500(config-vlan)# vlan 20,30
Verification:
n5500# show vlan
VLAN Name
Status Ports
---- -------------------------------- --------- --------1 default
active
10 email-vlan
active Eth2/1
11 VLAN0011
active
12 VLAN0012
active
<Text Omitted>
VLAN 10 is active and

configured on Ethernet 2/1
Cisco Confidential
106
VTP (*)
NXOS 5.0(2)N1(1) introduced
VTP client/server
Feature vtp
VTPv3 is needed for the full 4k

range, but it is not in this
release
VLANs in the range 1 1006

can be configured in VTP
Inconsistent VTP
configurations are a Type 2
misconfiguration (so it is not
disruptive to vPC)
VLANs beyond this range are

not propagated by VTP
PVLANs requires VTP to be

transparent or off
VTP v1 and v2
vPC + VTP is to be verified
Cisco Confidential
107
VLAN Trunking Protocol (VTP)

All VTP packets received on the Nexus 5500 are dropped by default. Enable VTP in
transparent mode to extend a VTP domain through a Nexus. Once, enabled, VTP
packets received on a trunk port are relayed to all other trunk ports.
Configuration:
Enable the VTP feature first!
n5500(config)# feature vtp

n5500(config)# vtp domain cisco.com
Configure the VTP domain name
n5500(config)# vtp version 2
Enables version 2 version 1 is the default

Note: Select the VTP domain name and version that match the values used in the existing VTP domain.
Verification:
n5500# show vtp status
VTP Version
Configuration Revision
Maximum VLANs supported locally
VTP Operating Mode
VTP Domain Name
VTP Pruning Mode
VTP V2 Mode
VTP Traps Generation
:2
:0
: 1005
: Transparent
: cisco.com
: Disabled
: Enabled
: Disabled
Cisco Confidential
108
Spanning Tree
NX-OS - Spanning Tree Design
NX-OS STP modes
Rapid-PVST+ (Default mode)
MST (Supported)
NR
PVST (Not supported, but

interoperable)
N R
Network Ports
All Send BPDUs
NX-OS always uses Extended System ID

NX-OS uses a fixed STP link cost for
Etherchannel links (based on number of
links configured, not number active as in
IOS)
Understand the three port modes
Access
Edge Ports
No BPDUs
Root port
Alternate port
Designated port
Edge port type replaces

spanning-tree portfast
E
N
R
Edge port
Network port
Root Guard
Network port type for bridge-tobridge links

Normal for generic links in
spanning tree
Cisco Confidential
109
Spanning-Tree Port Types

STP supports three different port types. The default port type is normal. An
edge port type can be configured, so an interface immediately forwards traffic
(IOS Portfast) and the network port type can be configured to enable Bridge
Assurance on an interface.
Port Types:
Edge *
Network
Normal (Default)
* Note: Trunk ports for L3 hosts can be configured with the edge trunk option
Port Configuration:
n5500(config-if-range)# spanning-tree port type ?
edge
Consider the interface as edge port (enable portfast)
network Consider the interface as inter-switch link
normal Consider the interface as normal spanning tree port
edge ports can be

configured on trunks with the
additional trunk option
Port Verification:
n5500# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
<Text Omitted>
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ----------------------Eth2/3
Desg FWD 4
128.259 P2p
Eth2/4
Desg FWD 4
128.260 Edge P2p
Eth2/5
Desg FWD 4
128.261 Network P2p
Normal (Default)
Edge
Network
Cisco Confidential
111
Optimizing the Layer 2 Design

Bridge Assurance
Specifies bi-directional
transmission of BPDUs on all
ports of type network.
Stopped receiving
BPDUS!
Provides IGP like hello-dead

timer behaviour for Spanning
Tree
In all versions of NX-OS,

available in IOS on the
Catalyst 6500 beginning
12.2(33) SXI
Recommended in STP
topologies
Not recommend in vPC
topologies
BPDUs
Root
Protects against
unidirectional links and peer
switch software issues
Malfunctioning
switch
Network
Network
BA Inconsistent
Network
Network
BPDUs
BPDUs
BA Inconsistent
Blocked
Network
Network
Stopped receiving
BPDUS!
Edge
Edge
interface port-channel200
switchport trunk allowed vlan 200-202
spanning-tree port type network
Cisco Confidential
112
Without Bridge Assurance

Root
Malfunctioning
switch
BPDUs
BPDUs
BPDUs
Blocked
Loop!
Cisco Confidential
113
With Bridge Assurance

Stopped receiving
BPDUS!
Malfunctioning
switch
BPDUs
Root
Network
Network
BA Inconsistent
Network
Network
BPDUs
BPDUs
BA Inconsistent
Blocked
Network
Network
Stopped receiving
BPDUS!
Edge
Edge
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48

VLAN0700.
tstevens-dc3-2# sh spanning vl 700 | in -i bkn
Eth2/48
Altn BKN*4
128.304 Network P2p *BA_Inc
tstevens-dc3-2#
Cisco Confidential
114
STP Bridge Assurance

Bridge Assurance prevents a spanning-tree domain from failing in an open
state. When a port configured for Bridge Assurance stops receiving BPDUs, the
port transitions into a blocking state as opposed to remaining in a
forwarding state. This closed state reduces the likelihood for misconfigured devices from creating STP loops.
Configuration:
n5500(config)# spanning-tree bridge assurance
Enabled by default
n5500(config)# interface ethernet 1/25, ethernet 1/26

n5500(config-if-range)# spanning-tree port type network
Change the port type to network
Note: Both ends of the link must have Bridge Assurance enabled
Verification:
n5500# show spanning-tree summary
Switch is in mst mode (IEEE Standard)
Root bridge for: MST0002
Port Type Default
Edge Port [PortFast] BPDU Guard Default
Edge Port [PortFast] BPDU Filter Default
Bridge Assurance
Loopguard Default
Pathcost method used
PVST Simulation
is disabled
is disabled
is disabled
is enabled
is disabled
is long
is enabled
Enabled on all network port types
<Text Omitted>
Cisco Confidential
115
STP (Rapid-PVST+) Configuration

Rapid-PVST is defined in IEEE 802.1w. Rapid-PVST enables one STP instance per
VLAN. Rapid-PVST is enabled by default, so there are very few commands
required to set up a Rapid-PVST domain.
Make sure you create the VLAN(s)
Rapid-PVST is the default
n5500(config)# vlan 20,30

n5500(config)# spanning-tree mode rapid-pvst
Decrements Priority to 24,596 to increase

the probability for it to become root
n5500(config)# spanning-tree vlan 20 root primary

n5500(config)# spanning-tree vlan 30 root secondary
Decrements Priority to 28,672 to increase the

probability for it to become the backup for the root
-OR-
n5500(config)# spanning-tree vlan 20,30 priority 4096
The preferred method to influence the root

selection is to manually set the bridge priority
Verifying STP Root Summary:

n5500# show spanning-tree root
Root Hello Max Fwd
Vlan
Root ID
Cost Time Age Dly
---------------- -------------------- ------- ----- --- --VLAN0020
24596 0018.bad8.58a5
0
2
20 15
VLAN0030
24606 0018.bad8.5825
4
2
20 15
Presentation_ID
Cisco Confidential
Root Port
---------------This bridge is root
Ethernet1/13
Specifies the root or

root port
116
STP (Rapid-PVST+) Verification

n5500# show spanning-tree
VLAN0020
Root ID Priority 24596
Address 0018.bad8.58a5
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 24596 (priority 24576 sys-id-ext 20)
STP Protocol = Rapid-PVST

Root Priority
Root STP ID (MAC Address)
Root Bridge or Root Port
This Bridges Priority and ID
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/13
Desg FWD 4
128.141 P2p
Eth1/14
Desg FWD 4
128.142 P2p
VLAN0030
Root ID Priority 24606
Address 0018.bad8.5825
Cost
4
Port
141 (Ethernet1/13)
Spanning-Tree port States (IE: FWD, BLK)
Bridge ID Priority 28672 (priority 28672 sys-id-ext 30)

Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/13
Root FWD 4
128.141 P2p
Eth1/14
Altn BLK 4
128.142 P2p
Cisco Confidential
117
Multiple Spanning Tree Configuration

MST is defined in IEEE 802.1s. MST maps multiple VLANs into instances that
maintain their own STP topology. MST improves STP scalability by reducing the
number of STP instances and providing fault isolation between STP domains.
Enable MST:
Make sure you create the VLAN(s)
n5500(config)# vlan 10,20

n5500(config)# spanning-tree mode mst
Change from the default RAPID-PVST mode to MST
Configure MST Instances:

n5500(config)# spanning-tree mst configuration
n5500(config-mst)# instance 1 vlan 10
n5500(config-mst)# instance 2 vlan 20
n5500(config-mst)# exit
Map VLANs to MST Instances
Configure the MST Bridge Priority (Optional):

n5500(config)# spanning-tree mst 1 root secondary
n5500(config)# spanning-tree mst 2 root primary
Cisco Confidential
118
Multiple Spanning Tree Verification

MST verification is very similar to Rapid-PVST. Several common show commands
exist for both protocols.
n5500# show spanning-tree mst
##### MST0 vlans mapped: 1-9,11-4094
Bridge
address 0018.bad8.5825 priority
32768 (32768 sysid 0)
Root
this switch for the CIST
Regional Root this switch
Operational hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured hello time 2 , forward delay 15, max age 20, max hops 20
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/25
Desg FWD 20000 128.153 P2p
MST1 with VLAN 10 mapped
##### MST1 vlans mapped: 10

Bridge
address 0018.bad8.5825 priority
28673 (28672 sysid 1)
Root
address 0018.bad8.58a5 priority
24577 (24576 sysid 1)
port Eth1/25
cost
20000 rem hops 19
Root Bridge information
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Eth1/25
Root FWD 20000 128.153 P2p
Ports in MST1 instance
Additional MST Options:

n5500# show spanning-tree mst ?
<CR>
<0-4094>
MST instance range, example: 0-3,5,7-9
>
Redirect it to a file
configuration
MST current region configuration
detail
Detailed information
interface
Spanning Tree interface status and configuration
|
Pipe command output to filter
Cisco Confidential
119
Data Center Architecture

Spanning Tree - Layer 2 Loops
Layer 2 topologies have sometimes proven an operational or
design challenge
Spanning tree protocol itself is not usually the problem, its the
external events that triggers the loop or flooding
L2 has had no native mechanism to dampen down a problem and
no solution to provide link redundancy other than STP
DST MAC 0000.0000.4444
3/2
3/1
3/2
3/1
Switch 1
Switch 2
DST MAC 0000.0000.4444
Cisco Confidential
120
Additional STP Features

The Cisco NX-OS supports several other Spanning-Tree Protocol features that
can be very useful to speed up convergence and reduce the likelihood for layer2 loops. All of the following STP extensions are documented on Cisco.com.
STP Extensions:
BPDU Guard
Shuts down an interface if a BPDU is received.
BPDU Filtering
Prevents a device from sending or receiving BPDUs on

specific ports.
Loop Guard
Prevents a unidirectional-link from creating a bridging loop.
PVST Simulation Allows MST to interoperate with Rapid-PVST+.

Root Guard
Prevents a specified port from becoming a root port.
Cisco Confidential
121
BPDU Guard
Prevents a switch from being plugged in on an Edge port
Port will move to STP BKN (show spanning-tree vlan x)
Recommended on access layer Edge or Edge Trunk ports
Two options for deployment in NX-OS:
Option 1: Enable on an interface:
DCN-N5K1(config-if)# spanning-tree bpduguard enable
Option 2: Enable by default on all Edge ports:

DCN-N5K1(config)# spanning-tree port type edge bpduguard default
Cisco Confidential
122
Global BPDU Filtering

Edge ports should have BPDU Guard enabled
If a BPDU is received port will transition to
err-disable state
Global BPDU Filter compliments BPDU Guard
On link up port will send 10-12 BPDUs and then
stop (in order to reduce CPU load)
If BPDU is received the port will err-disable

Improves CPU scaling in cases with trunk edge
ports (e.g. VMWare servers)
1. X-Connected
patch cable
This is NOT interface level BPDU Filtering
2. BPDU Sent on
Link-Up
dc11-5548-3(config)# spanning-tree port type edge bpdufilter default

dc11-5548-3(config)# interface ethernet 1/7
dc11-5548-3(config-if)# spanning-tree port type edge trunk
dc11-5548-3# show spanning-tree interface ethernet 1/7 detail

<snip>
The port type is edge
Link type is point-to-point by default
Bpdu filter is enabled by default
BPDU: sent 11, received 0
3. BPDU Guard
err-disables
edge port and
prevents loop
4. BPDU are not
sent once link is
up and active
Cisco Confidential
123
Loop Guard
Prevents a port from moving to forwarding upon loss of
BPDUs
Puts the port into loop_inconsistent state until BPDUs
are received again
Minimal benefit and not recommended for switches
running vPC
Deploy on access layer switches that are NOT
connected to the Agg layer using vPC
Global Configuration
n5K-1(config)#spanning-tree loopguard default
Interface Configuration
n5k-1(config-if)#spanning-tree guard loop
Cisco Confidential
126
Root Guard
Prevents Unwanted Changes to
STP Topology
Enable Root Guard on links
connecting to access layer to
protect from edge switches
becoming root and causing suboptimal traffic flow
Forces Layer 2 LAN interface to
be a designated port. If port
receives a superior BPDU, Root
Guard puts the interface into the
root-inconsistent (blocked) state
Channel the trunk between
Distribution Switches so failure
doesnt break topology
Secondary Root
Bridge
Root Bridge
Should never
receive a superior
BPDU
N R
Should never
receive a superior
BPDU
Root port
Alternate port
Designated port
Edge port
Network port
Root Guard
interface Ethernet1/32
description dc10-5548-4
switchport trunk allowed vlan 15,98,180-183
spanning-tree guard root
Cisco Confidential
128
Spanning Tree Recommendations

Port Configuration Overview
Data Center Core
Primary
vPC
vPC
Domain
Primary
Root
-
Network port
Edge or portfast port type
Edge Trunk
Normal port type
BPDU Guard
Rootguard
Loopguard
Secondary
vPC
HSRP
ACTIVE
Aggregation
HSRP
STANDBY
Layer 3
Secondary
Root
-
Layer 2 (STP + Rootguard)
Access
-
E
B
E
B
E
B
E
B
Nexus
1000v
Layer 2 (STP + BPDUguard)
N5K config defaults

loopguard
mode
mst
pathcost
port
vlan
Spanning tree loopguard options

Spanning Tree operating mode
Multiple spanning tree configuration
Spanning tree pathcost options
Spanning tree port options
VLAN Switch Spanning Trees
TM3# show spanning-tree summary

Switch is in rapid-pvst mode
Root bridge for: none
Port Type Default
Edge Port [PortFast] BPDU Guard Default
Edge Port [PortFast] BPDU Filter Default
Bridge Assurance
Loopguard Default
Pathcost method used
is disable
is disabled
is disabled
is disabled
is disabled
is short
Name
Blocking Listening Learning Forwarding STP Active
--------------------------------------------- -------- --------- -------- ---------- ---------VLAN0001
0
0
0
2
2
VLAN0213
0
0
0
3
3
--------------------------------------------- -------- --------- -------- ---------- ---------2 vlans
0
0
0
5
5
Cisco Confidential
130
Data Center Access Architecture

Spanning Tree Design Considerations
Nexus-5500# show spanning-tree interface ethernet 100/1/48 detail
Port 560 (Ethernet100/1/48) of VLAN0100 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.560
Designated root has priority 24776, address 0023.ac64.73c3
Designated bridge has priority 32968, address 000d.eca4.533c
Designated port id is 128.560, designated path cost 2
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
The port type is edge
Link type is point-to-point by default
BPDU Guard Is Enabled by Default and
Bpdu guard is enabled
Cannot be Disabled on FEX Server Ports
BPDU: sent 215784, received 0
udld enable
channel-group 200 mode active
Bridge Assurance Requires

the Port Type to be
Configured as network
udld enable
Nexus5500(config)# spanning-tree port type edge bpdufilter default
Global BPDU Filter
Cisco Confidential
131
Spanning Tree Path Cost Method

Default in NX-OS is short (16-bit values) for link costs
Using the Short method, a 10Gbps interface has a cost
of 2. A port-channel 20Gbps and above will have cost
of 1.
Recommended to change the Path Cost Method to
Long in order to accommodate larger link sizes.
All switches must be configured to use the same Path

Cost Method
DCN-N5K1(config)# spanning-tree pathcost method long
Cisco Confidential
132
Configuring N5K Ethernet Trunk Ports
cae-n5k(config)# int ethernet 1/3, ethernet

1/11, ethernet 1/8, ethernet 1/12
cae-n5k(config-if)# switchport mode trunk
cae-n5k(config-if)# switchport trunk allowed
vlan except 4093
cae-n5k(config-if)# no shut
encapsulation dot1q not required, it is the default.

ISL is not supported
Cisco Confidential
133
Verifying N5K Trunk Ports

cae-n5k# show run
switchport trunk allowed vlan 1-3967,4048-4092
[snip]
switchport trunk allowed vlan 1-3967,4048-4092
[snip]
cae-n5k# show interface ethernet 1/3
Ethernet1/3 is down (linkNotConnected)
Hardware is 10000 Ethernet, address is 000d.ec6b.cd4a (bia
000d.ec6b.cd4a)
Encapsulation ARPA
Port mode is trunk
[snip]
Cisco Confidential
134
Port-channel Count
UPC/Carmel supports 48
hardware port-channels
In Summary Every port can be
a port-channel with either 5548
or 5596
You can bundle up to 16 ports
in a single port-channel
Portchannels configured
on FEX do not take any
resource from the Nexus
5500 switch
More details in the
following slides
All ports can be part of a port-channel simultaneously

Cisco Confidential
135
LACP
Turn on LACP globally first
switch(config)# feature lacp
Channel mode needs to be either active or passive and

one side has to be active
No cisco PAgP supported

Switch 1 mode
Switch 2 mode
Port added to EtherChannel
active
passive
Yes
passive
active
Yes
active
active
Yes
passive
passive
No
active or passive
on
No
on
active or passive
No
on
on
Yes but no LACP negotiation

Cisco Confidential
136
Creating EtherChannel
Three channel group modes: active , passive and on.
Switch(conf)#interface e1/1
switch(config-if)# channel-group 1 mode ?
active
Set channeling mode to ACTIVE
on
Set channeling mode to ON
passive
Set channeling mode to PASSIVE
Channel mode
Description
active
Initiates LACP negotiation
passive
Responds to LACP negotiation
on
No LACP. Adds port to EtherChannel
Best practice is to use LACP in active mode on both sides of

the link
Cisco Confidential
137
Static EtherChannel
Adds port to EtherChannel without negotiataion
channel-group 1 is same as channel-group 1 mode on
switch(config)# int ethernet 1/1
switch(config-if)# channel-group 1
Ethernet1/1 added to port-channel1
switch(config-if)# exit
switch(config-if)# channel-group 1 mode on
Ethernet1/2: already part of port-channel1
switch(config-if)#
Cisco Confidential
138
Etherchannel - Force Keyword

If the physical port parameters do not match that of the
port-channel, the interface cannot be joined to the
Etherchannel
You could try and fix the inconsistency, or you can

force the interface into the channel-group
The config is pushed down from the port-channel to the
physical interface
switch(config-if)# channel-group 1 force mode active
Cisco Confidential
139
Port-Channel (LACP) Verification

Port-Channel Summary:
n5500# show port-channel summary
Flags: D - Down
P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended
r - Module-removed
S - Switched
R - Routed
U - Up (port-channel)
-------------------------------------------------------------------------------Group PortType
Protocol Member
Ports
Channel
-------------------------------------------------------------------------------1 LaCP Port-Channel
1
Po1(RU)
Eth
LACP
Eth1/13(P)
Eth1/14(P)
Traffic Distribution:
n5500# show port-channel traffic
ChanId
Port
Rx-Ucst
------ -------------1
Eth1/13
100.00%
1
Eth1/14
0.0%
with 2 members
Receive and transmit percentages

Tx-Ucst
------100.00%
0.0%
Rx-Mcst Tx-Mcst Rx-Bcst

Tx-Bcst
------- ------- ------- ------94.16%
71.15%
100.00%
100.00%
5.83%
28.84%
0.0%
0.0%
Usage:
n5500# show port-channel usage
Totally 1 port-channel numbers used
====================================
Used :
1
Unused:
2 - 4096
Presentation_ID
Cisco Confidential
140
Port-Channel (LACP) Statistics

n5500# show lacp neighbor
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode
P - Device is in Passive mode
port-channel1 neighbors
Partner's information
Partner
Partner
Partner
Port
System ID
Port Number
Age
Flags
Eth1/13
32768,0-18-ba-d8-58-250x10d
365
SA
LACP Partner
Port Priority
32768
Partner
Oper Key
0x0
Partner's information
Partner
Partner
Port
System ID
Port Number
Eth1/14
32768,0-18-ba-d8-58-250x10e
LACP Partner
Port Priority
32768
Partner
Oper Key
0x0
Neighboring device is
configured for Active mode
and sending Slow PDUs
Partner
Port State
0x3d
Age
284
Partner
Flags
SA
Partner
Port State
0x3d
n5500# show lacp counters

LACPDUs
Marker
Marker Response LACPDUs
Port
Sent
Recv
Sent
Recv
Sent
Recv
Pkts Err
-----------------------------------------------------------------------------------------------port-channel1
Ethernet1/13
34
21
0
0
0
0
0
Ethernet1/14
20
19
0
0
0
0
0
PDU errors
Successful PDUs
Presentation_ID
Cisco Confidential
141
Hash algorithm CLI

CLI to select the fields of frame into the hash calculation
Nexus5500(config)# port-channel load-balance ethernet ?
destination-ip
Destination IP address
destination-mac
Destination MAC address
destination-port
Destination TCP/UDP port
source-destination-ip
Source & Destination IP address
source-destination-mac
Source & Destination MAC address
source-destination-port Source & Destination TCP/UDP port
source-ip
Source IP address
source-mac
Source MAC address
source-port
Source TCP/UDP port
Check the hash algorithm

Nexus5500# sh port-channel load-balance
Port Channel Load-Balancing Configuration:
System: destination-mac
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: destination-mac
IPv4: destination-mac
IPv6: destination-mac
Cisco Confidential
142
Port-channel Load Balancing

CLI to help the user know about the port Nexus 5K picks
for load balancing on a Ethernet port-channel.
show port-channel load-balance [forwarding-path interface port-channel channelnumber] {dst-ip | dst-mac | dst-ipv6 | src-dst-ip | l4-src-port | l4-dst-port | src-ip | srcmac | src-ipv6 }
5548-2# sh port-channel load-balance
System: source-dest-ip
Port Channel Load-Balancing Addresses Used PerProtocol:

Non-IP: source-dest-mac
IP: source-dest-ip source-dest-mac
Example:
DCN-N5k2# show port-channel load-balance forwarding-path interface po20 src-interface e1/1 vlan 49 src-ip 10.122.49.10 dst-ip
172.18.84.183
Missing params will be substituted by 0's.
Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 148 Outgoing port id: Ethernet1/17
Param(s) used to calculate load-balance:
dst-ip: 172.18.84.183
src-ip: 10.122.49.10
dst-mac: 0000.0000.0000
src-mac: 0000.0000.0000
Cisco Confidential
143
Configuring N5K Port Channels
cae-n5k(config)# conf t
cae-n5k(config)# interface ethernet 1/3, ethernet 1/11
cae-n5k(config-if)# channel-group 5 force mode active
Ethernet1/3 Ethernet1/11 added to port channel 5
cae-n5k(config-if)#
cae-n5k(config-if)#
cae-n5k(config-if)#
cae-n5k(config-if)#
interface port-channel 5
switchport trunk allowed vlan except 4093
no shut
Cisco Confidential
144
Cisco Confidential
145
Virtual Port-Channel
Feature Overview
Allow a single device to use a
port channel across two
upstream switches
Eliminate STP blocked ports
Uses all available uplink
bandwidth
Dual-homed server operate in
active-active mode
Provide fast convergence upon
link/device failure
Physical Topology
Logical Topology
Virtual Port Channel

L2
Si
Si
Non-vPC
vPC
Increased BW with vPC
Cisco Confidential
146
Feature Overview
How does vPC help with STP?
Primary
Root
Before vPC
Secondary
Root
STP blocks redundant uplinks

VLAN based load balancing
Loop Resolution relies on STP
Protocol Failure
With vPC
No blocked uplinks
Lower oversubscription
EtherChannel load balancing (hash)
Loop Free Topology
Cisco Confidential
147
vPC Terminology on
N5K-N2K
vPC peer a vPC switch, one of a pair

vPC member port one of a set of ports
(port channels) that form a vPC
vPC the combined port channel
between the vPC peers and the
downstream device
vPC peer
keepalive link
vPC peer link Link used to synchronize

state between vPC peer devices, must
be 10GbE. Also carry
multicast/broadcast/flooding traffic and
data traffic in case of vpc member port
failure
vPC peer link
vPC peer
vPC
vPC
member
port
vPC peer keepalive link the peer

keepalive link between vPC peer
switches. It is used to carry heartbeat
packets
CFS Cisco Fabric Services protocol,
used for state synchronization and
configuration validation between vPC
peer devices
Cisco Confidential
148
How to Configure vPC

vPC configuration on the Cisco Nexus 5500 Series includes these steps:
Enable the vPC feature.

Create a vPC domain and enter vpc-domain mode.
Configure the vPC peer keepalive link.
(Optional) Configure system priority.
(Optional) Configure vPC role priority.
Create the vPC peer link.
Move the PortChannel to vPC.
Cisco Confidential
149

Enable VPC feature on both N5Ks
Configure VPC domain
N5K-1(config)# feature vpc
N5K-1(config)# vpc domain 1

VPC domain ID is an unique number (from 1 to 1000).
Note: The same VPC Domain ID will be configured on the other Nexus
5500.
Note: Each pair of devices in the same layer 2 domain running vPC must
always use a unique Domain ID.
Cisco Confidential
150

Configure system-priority (optional)
N5K-1(config-vpc-domain)# system-priority 4000
Enter the system priority that you want for the specified vPC domain. The range of
values is 1 to 65535. The default value is 32667.
You should manually configure the vPC system priority when you are running
Link Aggregation Control Protocol (LACP) to help ensure that the vPC peer
devices are the primary devices on LACP.
When you manually configure the system priority, make sure that you configure
the same priority value on both vPC peer devices. If these values do not match,
vPC will not be activated.
Cisco Confidential
151

Configure VPC role priority (optional)
N5K-1(config-vpc-domain)# role priority 8192
Each VPC member has a role (primary or secondary), it is calculated by the role
priority value plus local system mac, the lowest value will be elected as primary.
The default role priority is 32768.
Configure one N5500 as primary and the other as secondary by setting role
priority.
Once the election is completed, the VPC role will not change unless the VPC peer
link connection is reset.
Warning: vPCs will be flapped on current primary vPC switch while attempting
role change
Note: VPC Role will indicate none established and have a vPC local role-priority
of zero in the show vpc role command output until the VPC peer link comes up.
Cisco Confidential
152

Configure the VPC peer keepalive link
N5K-1(config-vpc-domain)# peer-keepalive destination 14.1.83.214
source 14.1.83.213 vrf management
It is recommended as best practice to use a separate L3 link for VPC keepalive
exchange and to put the peer keepalive link in a separate VRF.
Typically we will use interface mgmt0 with IP address 14.1.83.213/24 which is uses
vrf management for the peer-keepalive link.
For the destination address, use the mgmt0 IP address of the other N5K.
Cisco Confidential
153

Configure the VPC peer link
Configure interfaces e2/1 and e2/2 as members of PO10 and configure PO10 as
the peer link.
N5K-1(config-if)# int e2/1-2
N5K-1(config-if-range)# switchport mode trunk
N5K-1(config-if-range)#channel-group 10
N5K-1(config)# int po10
N5K-1(config-if)# switchport mode trunk
N5K-1(config-if)# vpc peer-link
First create a port-channel interface, in this example we use PO10 for the peer-link.
The peer-link must be a 10GE link between the VPC members.
Configure trunking on the L2 port-channel interfaces between the two Nexus 5500.
The supported channeling mode is On (which is the default) or LACP (i.e. mode active).
The port mode for interface port-channel 10 is configured as trunk.
NOTE: Spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge
Assurance (which is enabled by default) is not disabled.
NOTE: The port-channel for the peer link and the peer keepalive link will not come up until the
other N5500 is also configured identically.
Cisco Confidential
154
How to Configure vPC (contd)

Move the Downstream PortChannel to vPC
interface port-channel channel-number
vpc number
switch(config)#interface e1/1
switch(config-if)channel-group 20
switch(config-if)# interface port-channel 20
switch(config-if)# vpc 100
Add the interface to the PortChannel and then move the PortChannel to the vPC
to connect to the downstream device. The vPC number ranges from 1 to 4096.
The vPC number does not need to match the PortChannel number, but it must
match the number of the vPC peer switch for that vPC bundle.
A PortChannel is needed even if there is only one member interface for the
PortChannel. When there is only one member for the PortChannel, the hardware
PortChannel resource will not be created.
Cisco Confidential
155
Configuring vPC
The following example enables vPC with LaCP on one side of the vPC
Domain. The same config is required on the other vPC Domain member.
Enable the LaCP and vPC features first!
N5K(config)# feature lacp
N5K(config)# feature vpc
Configure the
vPC domain and
keep-alive link
N5K(config)# vpc domain 1

N5K(config-vpc-domain)# peer-keepalive destination 10.20.0.191 source 10.20.0.190
Note:
--------:: Management VRF will be used as the default VRF ::-------N5K(config)# interface ethernet 3/1,ethernet 4/1
N5K(config-if-range)# switchport
N5K(config-if-range)# switchport mode trunk
N5K(config-if-range)# switchport trunk allowed vlan 9,11-14
N5K(config-if-range)# channel-group 10 mode active
N5K(config-if-range)# no shut
Configure the vPC Peer-Link
Define the vPC Peer-Link
N5K(config-if-range)# interface port-channel 10

N5K(config-if)# vpc peer-link
Please note that spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance
(which is enabled by default) is not disabled.
N5K(config)# interface ethernet 3/2,ethernet 4/2
N5K(config-if-range)# switchport
N5K(config-if-range)# switchport mode trunk
N5K(config-if-range)# switchport trunk allowed vlan 11-14
N5K(config-if-range)# channel-group 20 mode active
N5K(config-if-range)# no shut
N5K(config-if-range)# interface port-channel 20
N5K(config-if)# vpc 20
Configure the downstream link
Define the vPC Port-Channel # for the downstream link

Cisco Confidential
156

vPC Domain 10
vPC Domains
vPC Domain defines the grouping of
switches participating in the vPC
Provides for definition of global vPC
system parameters
vPC Domain 20
The vPC peer devices use the vPC

domain ID to automatically assign a
unique vPC system MAC address
You MUST use utilize unique
Domain ids for all vPC pairs defined
in a contiguous layer 2 domain
! Configure the vPC Domain ID It should be unique within the layer 2 domain
dc11-5548-1(config)# vpc domain 20
! Check the vPC system MAC address
dc11-5548-1# show vpc role
<snip>
vPC system-mac
: 00:23:04:ee:be:14
vPC System MAC identifes the Logical

Switch in the network toplogy
Cisco Confidential
157
Domain ID
vPC System MAC is used for both LACP System Identifier and STP bridge
ID. Uses IETF assigned range of 00:23:04:ee:be:00 -> 00:23:04:ee:c1:ff.
vPC Domain ID is encoded in the vPC System MAC within the last octet
and the trailing 2 bits of the previous octet
10 bits
vPC Domain ID
System Identifier used by LACP to identify

links connected to the same neighbor
Duplicate System ID would result in an
LACP error condition
Could also result in two switches with the
same STP Bridge ID
vPC Domain 10
vPC Domain 20
You MUST use a unique vPC domain ID for

each pair of adjacent vPC peers!
Note: This also applies to VSS domains as well. Always use a unique domain
ID when connecting a vPC domain to VSS
Cisco Confidential
158

802.3ad & LACP System MAC
LACP neighbour needs to see the same System ID from both vPC peers
The vPC system-mac is used by both vPC peers
dc11-5548-1# sh vpc role
<snip>
vPC system-mac
vPC system-priority
vPC local system-mac
vPC local role-priority
:
:
:
:

<snip>
vPC system-mac
vPC system-priority
00:23:04:ee:be:14
1024
00:0d:ec:a4:53:3c
1024
:
:
:
:
00:23:04:ee:be:14
1024
00:0d:ec:a4:5f:7c
32667
dc11-5548-2
dc11-5548-1
1/33
1/34
dc11-4948-1
dc11-4948-1#sh lacp neighbor
<snip>
LACP port
Port
Flags
Priority Dev ID
Gi1/33
SA
32768
0023.04ee.be14
Gi1/34
SA
32768
0023.04ee.be14
Age
9s
21s
Admin
key
0x0
0x0
Oper
Key
0x801E
0x801E
Port
Number
0x4104
0x104
Port
State
0x3D
0x3D
Cisco Confidential
159

802.3ad & LACP System MAC
vPC peers function as independent devices as well as peers

Local system-mac is used for all non vPC PDUs (LACP, STP, )
<snip>
vPC system-mac
vPC system-priority
:
:
:
:
00:23:04:ee:be:14
1024
00:0d:ec:a4:53:3c
1024
dc11-5548-2
dc11-5548-1
Regular (non vPC)

Etherchannel
1/4
MCEC (vPC)
Etherchannel
1/5
dc11-4948-1
dc11-4948-2
dc11-4948-2#sh lacp neighbor
<snip>
LACP port
Port
Flags
Priority Dev ID
Gi1/4
SA
32768
000d.eca4.533c
Gi1/5
SA
32768
000d.eca4.533c
Age
8s
8s
Admin
key
0x0
0x0
Oper
Key
0x1D
0x1D
Port
Number
0x108
0x108
Port
State
0x3D
0x3D
Cisco Confidential
160
Peer Keepalive Link
Peer Keepalive provides an out-of-band
heartbeat between vPC peers
Purpose is to detect and resolve roles if
a Split Brain (Dual Active) occurs
Messages sent on 1 second interval with
5 second timeout
Peer Keepalive
carried over the
OOB management
network
int mgmt 0
3 second hold timeout on peer-link loss

before triggering recovery
Should not be carried over the Peer-Link

Keepalives sourced and destined to the
mgmt0 interface
Keep-alives can be routed over L3
infrastructure
dc11-5548-1(config)# vpc domain 20
dc11-5548-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source
172.26.161.200 vrf management
Note:
--------:: Management VRF will be used as the default VRF ::--------
Cisco Confidential
161
vPC Peer Link
Peer Link carries both vPC data and
control traffic between peer switches
Carries any flooded and/or
orphan port traffic
Carries STP BPDUs, HSRP
Hellos, IGMP updates, etc.
vPC Peer
Link
Carries Cisco Fabric Services

messages (vPC control traffic)
Minimum 2 x 10GbE ports
It is not recommended to share vPC
and non-vPC traffic on the same Peer
Link
dc11-5548-1(config)# interface port-channel 20
dc11-5548-1(config-if)#
vpc peer-link
Cisco Confidential
162

vPC Roles
Role is defined under the domain
configuration
Lower priority wins if not, lower system mac
wins
Secondary
(but may be
Operational
Primary)
Role is non-preemptive so Operational Role is

what matters
Operational Role may different from the
priorities configured under the domain
vPC Role defines which of the two vPC peers

processes BPDUs
Role matters for the behavior with peer-link
failures!
Primary (but may be

Operational Secondary)
dc11-5548-3(config-vpc-domain)# role priority ?

<1-65535> Specify priority value
dc11-5548-3# sh vpc
<snip>
vPC role
: secondary, operational primary
Cisco Confidential
163
vPC Control Fabric Cisco Fabric Services

Cisco Fabric Services provides the
control plane synchronization between
vPC peers
Configuration validation/comparison
Cisco
Fabric
Services
MAC member port synchronization

vPC member port status
IGMP snooping synchronization
CFSoE
vPC status
Highly Reliable - Inherited from MDS
CFS messages are encapsulated in
standard Ethernet frames (with CoS 6)
dc11-5548-2# show CFS status
Distribution : Enabled
Distribution over IP : Disabled
IPv4 multicast address : 239.255.70.83
IPv6 multicast address : ff15::efff:4653
Distribution over Ethernet : Enabled
Cisco Confidential
164
vPC Control Plane Cisco Fabric Services

vPC supports standard 802.3ad port
channels from upstream and or
downstream devices
dca-n7k2-vdc2
Recommended to enable LACP

dc11-5548-1
dc11-5548-2
dca-n7k2-vdc2# sh run interface port-channel 201

version 4.1(5)
logging event port link-status
logging event port trunk-status
dc11-5548-1# show running int port-channel 201

version 4.1(3)N1(1)

version 4.1(3)N1(1)
vpc 201
vpc 201
Cisco Confidential
165
Virtual Port Channel - vPC

vPC Control Plane - Consistency Check
Both switches in the vPC Domain maintain
distinct control planes
CFS provides for protocol state
synchronization between both peers (MAC
Address table, IGMP state, )
System configuration must also be kept in
sync
Currently a manual process with an
automated consistency check to ensure
correct network behaviour
Two types of interface consistency checks
Type 1 Will put interfaces into suspend
state to prevent invalid forwarding of
packets
Type 2 Error messages to indicate
potential for undesired forwarding
behaviour
Cisco Confidential Internal Use Only
166
vPC Control Plane Type 1 Consistency Check

Type 1 Consistency Checks are
intended to prevent network failures
Incorrectly forwarding of traffic
Physical network incompatibilities
vPC will be suspended
dc11-5548-1# sh run int po 201
vpc 201
vpc 201
spanning-tree guard root
dc11-5548-2# show vpc brief

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
<snip>
vPC status
---------------------------------------------------------------------------id
Port
Status Consistency Reason
Active vlans
------ ----------- ------ ----------- -------------------------- ----------201
Po201
up
failed
vPC type-1 configuration
incompatible - STP
interface port guard Root or loop guard
inconsistent
167
vPC Control Plane Type 2 Consistency Check

Type 2 Consistency Checks are
intended to prevent undesired
forwarding
vPC will be modified in certain cases
(e.g. VLAN mismatch)

version 4.1(3)N1(1)

version 4.1(3)N1(1)
vpc 201
vpc 201
dc11-5548-1# show vpc brief vpc 201

vPC status
---------------------------------------------------------------------------id
Port
Active vlans
------ ----------- ------ ----------- -------------------------- ----------201
Po201
up
success
success
100-104
2009 May 17 21:56:28 dc11-5548-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface portchannel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)
168
vPC Control Plane Global Consistency Checks

Dont forget to keep global configuration
in sync
Any configuration that could cause
an error in forwarding (e.g. loop) will
disable all affected interfaces
As an example if you make a change to
an MST region you must make it on
both peers
mst region
vlans 1-5, 12
Solution: define MST region mappings

from the very beginning of the
deployment, for ALL VLANs, the ones
that exist as well as the ones that have
not yet been created
Defining a region mapping is orthogonal
to creating a VLAN
vPC
mst region
vlans 1-5, 10
vPC
vPC
This behavior equally applies to Nexus

7000 and Nexus 5500 when configured
as vPC peers
169
vPC Consistency Check Global Configuration Parameters

Global Parameters
are type 1
Global QoS
Parameters
need to be
consistent
Global
Spanning
Tree
Parameters
need to be
consistent
170
Global vs. Interface Consistency Check

Global consistency check failure for type 1 will result all vPC suspended
Interface level consistency check failure only affects the involved interfaces
n5k-1# show vpc consistency-parameters interface port-channel 200
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
------------STP Port Type
STP Port Guard
STP MST Simulate PVST
lag-id
Type
---1
1
1
1
mode
Speed
Duplex
Port Mode
Native Vlan
Allowed VLANs
1
1
1
1
1
-
Local Value
---------------------Default
None
Default
[(7f9b,
0-23-4-ee-be-64, 80c8,
0, 0), (8000,
0-1e-13-15-7-40, 1, 0,
0)]
active
10 Gb/s
full
trunk
1
1-999,1001-3967,4048-4093
Peer Value
----------------------Default
None
Default
[(7f9b,
0-23-4-ee-be-64, 80c8,
0, 0), (8000,
0-1e-13-15-7-40, 1, 0,
0)]
active
Type 2 consistency
10 Gb/s
check parameter
full
trunk
1
1-3967,4048-4093
n5k-1#
171
vPC Forwarding
172

vPC provides optimized forwarding
vPC forwards only on locally connected
members of the port channel if any exist
(same principle as VSS)
dca-n7k2-vdc2
Multiple topology choices

Square
Full Mesh
dc11-5548-1
dc11-5548-2
dca-n7k2-vdc2# sh run interface port-channel 201

version 4.1(5)
vpc 201
version 4.1(3)N1(1)

version 4.1(3)N1(1)
vpc 201
vpc 201
173
MAC_C
vPC Forwarding- Unicast Learning

5
vPC maintains layer 2 topology

synchronization via CFS
Copies of flooded frames are sent across

the vPC-Link in case any single homed
devices are attached
Frames received on the vPC-Link are not
forwarded out vPC ports
1.
2.
3.
4.
5.
6.
Host MAC_A send packet to MAC_C

FEX runs hash algorithm to select one fabric uplink
N5K-1 learns MAC_A and flood packets to all ports
(in that VLAN). A copy of the packet is sent across
the peer link
N5K-2 floods the packet to any port in the VLAN
except the vPC member ports to prevent duplicated
packets
N7K-1 and N7K-2 repeat the same forwarding logic
N5K-1 updates the the MAC address learned on the
vPC port on N5K-2 via CFS
N5K-1
CFS
N5K-2
1
MAC_A
Double Sided
vPC
174
MAC_C
vPC Forwarding- Unicast Learning

1
Traffic is forwarded if destination address is

known (both switches MAC address tables
populated)
Always forward via a locally attached
member of a vPC if it exists
1. Host MAC_C send packet to MAC_A

2. N7K-2 forwards frame based on learned
MAC address
3. N5K-2 forwards frame based on learned
MAC address
N5K-1
N5K-2
N5K-1# sh mac-address-table vlan 101

VLAN
MAC Address
Type
Age
Port
---------+-----------------+-------+---------+----101
001b.0cdd.387f
dynamic 0
Po30
101
0023.ac64.dda5
dynamic 30
Po201
Total MAC Addresses: 4

VLAN
MAC Address
Type
Age
Port
---------+-----------------+-------+---------+----101
001b.0cdd.387f
dynamic 0
Po30
101
0023.ac64.dda5
dynamic 30
Po201
3
MAC_A
175
MAC_C
vPC Forwarding- Unicast Recovery

1
On loss of all of the locally attached

members of the vPC MAC address
table is updated to forward frames
for the vPC across the vPC Peer
Link
2
N5K-1
N5K-2

VLAN
MAC Address
Type
Age
Port
---------+-----------------+-------+---------+----101
001b.0cdd.387f
dynamic 0
Po30
101
0023.ac64.dda5
dynamic 30
Po201

VLAN
MAC Address
Type
Age
Port
---------+-----------------+-------+---------+----101
001b.0cdd.387f
dynamic 0
Po20
101
0023.ac64.dda5
dynamic 30
Po201
3
MAC_A
176
vPC Failure Scenarios

on N55K
Cisco Confidential
177
vPC Failure Reaction

vPC member port failure
vPC
Primary
vPC
Secondry
Po1
When vPC member port fails,

N5k updates the MAC table for
all the address points to the
affected vPC bundle
On the right N5k, MAC_A points
to peer link Po1 after the
failure occurs
Before the failure, MAC_A

points to Po2
Po2
vPC member port status change

is updated to peer via CFS
message
MAC_A
Cisco Confidential
178
vPC Failure Reaction (FEX Straight Thru)

Peer-link failure
vPCmember port
is suspended
vPC
Primary
vPC
Secondry
When peer link fails,

secondary vpc peer switch
suspends all its vpc member
ports
vPC secondary detects
primary switch is alive
through peer keepalive link
vPCmember port
is suspended
Cisco Confidential
179
vPC Failure Reaction (FEX A/A)

Peer-link failure
vPC
Primary
vPC
Secondry
When peer link fails, secondary

vpc peer switch suspends all its
vpc member ports
FEX will be only connected to

primary switch.
FEX ports remain up
Cisco Confidential
180
vPC Failure Reaction

keepalive link failure
vPC
Primary
Dont care as long as peer link

is up
vPC
Secondry
Cisco Confidential
181
vPC Double Failure Reaction

Peer-link failure followed by keepalive link failure
vPC
Primary
vPC
Secondry
When peer link fails, secondary

vpc peer switch suspends all its
vpc member ports
Keepalive failure has no impact
Cisco Confidential
182

Peer-link failure followed by keepalive link failure
vPC
Primary
vPC
Secondry
With the failure of both peer link

and peer keepalive link, FEX will
be connected ONLY to primary
vPC switch.
Cisco Confidential
183

Keepalive link failure followed by Peer Link failure
vPC
Primary
vPC
Secondry
With the peer keepalive link

down, vPC secondary switch
doesnt know if the primary is
alive when the peer link fails
Both switch run as primary
switch
STP ensures no loop
Cisco Confidential
184
vPC Enhancements
Cisco Confidential
185
NX-OS 5.0(2)N1(1)
QoS Config Checks have been lowered to Type-2
Several features have the misconfiguration type lowered from Type 1 to Type 2
Configurations can be synched between vPC member ports by using the Config-sync
feature
tc-nexus5548-1# show vpc consistency-parameters global

Name
Type
Local Value
-------------
----
---------------------- -----------------------
QoS
([], [3], [], [], [],
([], [3], [], [], [],
[])
[])
(1538, 2240, 0, 0, 0,
(1538, 2240, 0, 0, 0,
0)
0)
Network QoS (MTU)
Peer Value
Network Qos (Pause)
(F, T, F, F, F, F)
(F, T, F, F, F, F)
Input Queuing (Bandwidth)
(50, 50, 0, 0, 0, 0)
(50, 50, 0, 0, 0, 0)
Input Queuing (Absolute
(F, F, F, F, F, F)
(F, F, F, F, F, F)
Output Queuing (Bandwidth)
(50, 50, 0, 0, 0, 0)
(50, 50, 0, 0, 0, 0)
Output Queuing (Absolute
(F, F, F, F, F, F)
(F, F, F, F, F, F)
Priority)
Cisco Confidential
186
NX-OS 5.0(2)N2(1)
vPC graceful type-1 checks
Keepalive
S1 -Primary
S2-Secondary
vPC peer-link
vPC member ports on S1 and S2 should

have identical parameters (MTU, speed,
)
Any inconsistency in such parameters
is Type1. As a consequence, all vlans on
both vpc legs are brought down in such
inconsistency
With graceful type-1 check, only
Secondary vPC members are brought
down. vPC member ports on primary
peer device remain up
Type-1
Inconsistency
vPC 1
po1
CE-1
S1(config-vpc-domain)# graceful
consistency-check
S2(config-vpc-domain)# graceful
consistency-check
Graceful Type-1 check enabled by
default.
Cisco Confidential
187
vPC Auto-Recovery
Peer Keepalive Link
When a vPC peer is missing, by

default vPC doesnt allow any
vPC member port to either flap or
for a new one to be brought
online or for existing vPC
member to go up after a reload
Auto-recovery monitors the peer
device and if the vPC peer is not
available for the reload-delay
time, it allows new ports to be
brought up even if the peer is
missing
vPC
Primary
Switch
1
Switch3
Missing vPC Peer
Switch4
Cisco Confidential
188
vPC Auto-Recovery
If enabled (default is disabled)
On switch reload, vPC listens
to switch online notification
(indicates all LCs are up)
Starts reload-delay timer

(user configurable), default 240
seconds
Secondary
S1
S2
vPC peer-link
If peer-link port comes

physically up or peer-keep
alive works, stop timer, wait for
peer adjacency to form
Normal behavior, peer
presumed alive
S4
Primary
vPC 2
vPC 1
po1
po2
Cisco Confidential
189
vPC Auto-Recovery
If enabled
C
If after reload-delay timer

expiration, no peer-keep alive or
no peer-link up received
Assume primary STP role

Assume primary LACP role
(internal role between LACP and
vPC, currently based on switch
mac comparison)
S4
Primary
S1
S2
vPC peer-link
Reinitialize vPCs
vPC 2
vPC 1
On vPC port bringup, consistency

check is bypassed for vPCs
po1
po2
Cisco Confidential
190
NX-OS 5.0(2)N2(1)
Keepalive
S1 -Primary
vPC auto-recovery
S2-Secondary
vPC peer-link
Keepalive
S1 -Primary
S2-Secondary
vPC peer-link
vPC 1
po1
vPC 1
po1
S1 -Primary
CE-1
Keepalive S2-Operational
Primary
CE-1
vPC peer-link
1. vPC peer-link goes down : vPC

secondary peer device shuts all its
vPC member ports
2. S1 goes down. S2 receive no more
messages on vPC peer-keepalive link
3. After 3 consecutive keepalive
timeouts, vPC secondary peer device
(S2) changes role and brings up its
vPC.
vPC 1
po1
S1(config-vpc-domain)# auto-recovery
S2(config-vpc-domain)# auto-recovery
CE-1
Cisco Confidential
191
Virtual Port Channel vPC
Design Considerations Orphan Ports

Orphan Ports are single homed ports
on a member of the vPC pair
In the event of loss of the vPC peer
link all vPC ports on the secondary
vPC switch are shut down to prevent
topology problems
Non vPC or orphan ports are left
active
Primary
Secondary
Potential to isolate orphan ports

Design Options
1. Connect orphan ports to the
vPC primary switch
2. Provide a secondary uplink or
switch to switch link for orphan
port VLAN (requires the use of
distinct VLAN for orphan and
vPC ports)
Orphan Ports remain active

on vPC secondary during a
failure of the peer-link
Standby link remains inactive
Cisco Confidential
192
Design Considerations Orphan Ports
Cisco Confidential
193
NX-OS 5.0(3)N2(1)
vpc orphan-port suspend new knob

Supported only on physical Ethernet
interfaces
Suspends/Disables orphan ports on
vPC secondary switch during peerlink failure
Orphan ports are re-enabled along
with vPCs on peer-link recovery
Primary
Secondary
show vpc orphan-port to display

configured orphan ports
Best practices
Eliminate orphan ports with
dual-homing when you can
If not, identify orphan ports and
use new configuration knob
1. Orphan Ports are disabled

2. Standby link takes over
Cisco Confidential
194
vPC Troubleshooting
Cisco Confidential
195
vPC troubleshooting
Basic checks
Nexus# sh vpc
...
vPC domain id
: 111
Peer status
: peer adjacency formed ok
vPC keep-alive status
: peer is alive
Configuration consistency status: success
vPC role
: primary
vPC Peer-link status

--------------------------------------------------------------------id
Port
Status Active vlans
---------- -------------------------------------------------1
Po100 up
1,34-35
vPC status
---------------------------------------------------------------------id
Port
Active vlans
---------- ----------- -------------------------- -----------1
Po1
up
success
success
34-35
Cisco Confidential
196
vPC troubleshooting
Config check (vPC default parameters not shown)
Nexus# sh run vpc
version 4.1(5)
feature vpc
vpc domain 111
peer-keepalive destination 7.7.7.77
source 7.7.7.7 vrf v1
Nexus-dg# sh run vpc

version 4.1(5)
feature vpc
vpc domain 111
source 7.7.7.77 vrf v1
vpc 1
vpc 1
vpc peer-link
vpc peer-link
Cisco Confidential
197
vPC troubleshooting
vPC peer-keepalive check
vPC timers check
Nexus# show vpc peer-keepalive
--Send status
--Last send at
--Sent on interface
--Receive status
--Last receive at
--Received on interface
--Last update from peer
:
:
:
:
:
:
:
:
peer is alive
Success
2009.06.19 00:41:15 589 ms
Eth2/35
Success
2009.06.19 00:41:14 580 ms
Eth2/35
(1) seconds, (9) msec
vPC Keep-alive parameters

--Destination
--Keepalive interval
--Keepalive timeout
--Keepalive hold timeout
--Keepalive vrf
--Keepalive udp port
--Keepalive tos
:
:
:
:
:
:
:
7.7.7.77
1000 msec
5 seconds
3 seconds
v1
3200
192
Cisco Confidential
198
vPC troubleshooting
vPC peer-keepalive statistics
Nexus# show vpc statistics peer-keepalive
: peer is alive
vPC keep-alive statistics
---------------------------------------------------peer-keepalive tx count:
9773
peer-keepalive rx count:
8985
average interval for peer rx:
991
Count of peer state changes:
159
Cisco Confidential
199
vPC troubleshooting
vPC role (primary / secondary) and system-mac
Nexus# show vpc role
vPC Role status
---------------------------------------------------vPC role
: primary
Dual Active Detection Status
: 0
vPC system-mac
: 00:23:04:ee:be:6f
vPC system-priority
: 32667
: 00:1b:54:c2:42:41
: 32667
Cisco Confidential
200
vPC troubleshooting
Global consistency parameters
Nexus# show vpc consistency-parameters global
Legend:
Name
------------STP Mode
STP Disabled
STP MST Region Name
STP MST Region Revision
STP MST Region Instance to
VLAN Mapping
STP Loopguard
STP Bridge Assurance
STP Port Type
Allowed VLANs
Type
---1
1
1
1
1
Local Value
---------------------Rapid-PVST
None
""
0
Peer Value
----------------------Rapid-PVST
None
""
0
1
1
1
1
-
Disabled
Enabled
Normal
Enabled
1,34-35,51,69-70,99,20
Disabled
Enabled
Normal
Enabled
1-2,34-35
Note currently it is user responsibility to ensure same L3 interfaces are

present and are in the same operational state on both peer devices
Cisco Confidential
201
vPC troubleshooting
Interface consistency parameters
Nexus# show vpc consistency-parameters interface port-channel 1
Legend:
Name
------------STP Port Type
STP Port Guard
lag-id
Type
---1
1
1
1
mode
Speed
Duplex
Port Mode
Native Vlan
MTU
Allowed VLANs
1
1
1
1
1
1
-
Local Value
---------------------Default
None
Default
[(7f9b,
0-23-4-ee-be-6f, 8001,
0, 0), (8000,
0-12-da-65-9e-c0, 1,
0, 0)]
active
1000 Mb/s
full
trunk
2
1500
34-35
Peer Value
----------------------Default
None
Default
[(7f9b,
0-23-4-ee-be-6f, 8001,
0, 0), (8000,
0-12-da-65-9e-c0, 1,
0, 0)]
active
1000 Mb/s
full
trunk
2
1500
34-35
Cisco Confidential
202
VLAN Err-Disabled Status On Trunk

N5K-1
N5K-2
PK
int po20
switchport trunk allowed
vlan 1,10-11,100,176,208209,3001
int po20
switchport trunk allowed
vlan 1,10-11,176,208209,3001
PL
PO10
PO20
VL100 must be in the

allowed list on both
N5K-1 and N5K-2 for
err-disabled to clear!
N5K-1# show int po20 trunk

Port
Status
Po20
Native
Vlan
1
Port
Po20
Vlans Allowed on Trunk

1,10-11,100,176,208-209,3001
Port
Po20
Vlans Err-disabled on Trunk

100
trunking
VL100 is missing on
vPC Peer Link
Port
Channel
--
VLAN shows up as err-disabled

Cisco Confidential
203
Type-1 Global Inconsistency

N5K-1
N5K-2
PK
N5K-2# spanning-tree loopguard default
PL
PO10
PO20
All vPC
Member Ports
are taken
down!
N5K-1# show port-channel sum int p20
Flags: D - Down
s - Suspended r - Module-removed
S - Switched R - Routed
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group PortType Protocol Member Ports
Channel
-------------------------------------------------------------------------------20 Po20(SD) Eth
LACP
Eth2/17(D)
N5K-1# show vpc brief

Legend:
vPC domain id
:3
Peer status
: peer is alive
Configuration consistency status: failed
Configuration consistency reason: vPC type-1 configuration
incompatible - STP global loop guard inconsistent
Type-2 consistency status
: failed
Type-2 consistency reason
: SVI type-2 configuration incompatible
vPC role
: secondary
Number of vPCs configured
:4
Peer Gateway
: Enabled
Peer gateway excluded VLANs : Dual-active excluded VLANs
:vPC Peer-link status
--------------------------------------------------------------------id Port Status Active vlans
-- ---- ------ -------------------------------------------------1 Po10 up vPC status
---------------------------------------------------------------------id Port Status Consistency Reason
Active vlans
-- ---- ------ ----------- ----------------20 Po20 down* failed
Global compat check failed -
Cisco Confidential
204
Type-1 Global Inconsistency
Cisco Confidential
205
Type-1 Interface Inconsistency

N5K-1
N5K-2
PK
N5K-1f)# show vpc brief

Legend:
PL
PO10
PO20
N5K-1# spanning-tree guard root
vPC member ports shut down until

both N5K-1 and N5K-2 configured.
Only PO20 is affected, other vPCs

remain operational.
vPC domain id
:3
Peer status
: peer is alive
Configuration consistency status: success
: failed
vPC role
: primary
:4
Peer Gateway
: Enabled
-- ---- ------ -------------------------------------------------1 Po10 up 1,10-11,176,208-209,3001
vPC status
Active vlans
-- ---- ------ ----------- ----------------20 Po20 up failed
vPC type-1 configuration incompatible - STP
interface port guard Root or loop guard
inconsistent
Cisco Confidential
206
Graceful Type-1 Recovery

N5K-1
N5K-2
PK
N5K-2# spanning-tree loopguard default
PL
PO10
PO20
Peer holding Secondary vPC role shuts down

vPC member ports
N5K-1# show port-channel sum int p20
Flags: D - Down
s - Suspended r - Module-removed
S - Switched R - Routed
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group PortType Protocol Member Ports
Channel
-------------------------------------------------------------------------------20 Po20(SD) Eth
LACP
Eth2/17(P)
N5K-1# show vpc brie

Legend:
vPC domain id
:3
Peer status
: peer is alive
Configuration consistency status: failed
Configuration consistency reason: vPC type-1 configuration
incompatible - STP global loop guard inconsistent
: failed
vPC role
: secondary
:4
Peer Gateway
: Enabled
-- ---- ------ -------------------------------------------------1 Po10 up vPC status
Active vlans
-- ---- ------ ----------- ----------------20 Po20 down* failed
Global compat check failed -
Cisco Confidential
207
Local Suspended VLAN
Common Causes:
VLAN not permitted on vPC Peer Link

VLAN doesnt exist in VL database on vPC peer
In case of global inconsistency, all VLANs suspended
Cisco Confidential
208
What Happened?
N5K-1g)# show logging level vpc
Facility
Severity
-------vpc
Default Severity
---------------2
0(emergencies)
3(errors)
6(information)
Current Session
-----------------------3
1(alerts)
2(critical)
4(warnings) 5(notifications)
7(debugging)
Default severity level

for vPC is 2.
Recommended to
change this to at least
3 to see msgs such as
below
N5K-1(config)#logging level vpc 3

N5K-1# show logging | i %VPC
2011 Aug 25 13:14:34 N5K-1 %VPC-3-GLOBAL_CONSISTENCY_FAILED: In doma
in 3, global configuration is not consistent (vPC type-1 configuration incompati
ble - STP global loop guard inconsistent)
Who Done It?

N5K-1# show accounting log | b Aug 25 13:14
Thu Aug 25 13:14:34 2011:type=update:id=10.116.186.217@pts/28:user=admin:cmd=con
figure terminal ; spanning-tree loopguard default (SUCCESS)
Cisco Confidential
209
STP and vPC

Peer link is running STP
DCN-N5K1# show spanning vlan 176
VLAN0176
Root ID
Priority
8368
Address
0023.04ee.be01
Cost
2
Port
4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec
Bridge ID
Priority
Address
Hello Time
Interface
---------------Po1
Po20
Po27
Po28
Role
---Root
Root
Desg
Desg
It is possible to see situation when

there are 2 root ports: peer-link and
vPC toward the root
This happens on vPC peer holding
the vPC secondary role
This is perfectly normal in a vPC
environment!
Forward Delay 15 sec
32944 (priority 32768 sys-id-ext 176)

000d.ecb2.2afc
2 sec Max Age 20 sec Forward Delay 15 sec
Sts
--FWD
FWD
FWD
FWD
Cost
--------1
1
1
1
Prio.Nbr
-------128.4096
128.4115
128.4122
128.4123
Type
-------------------------------(vPC peer-link) Network P2p
(vPC) P2p
(vPC) Edge P2p
(vPC) Edge P2p
Cisco Confidential
210
sh tech-support vpc
Collect for TAC/engineering to look at the issue
Collects the following
`show version`
`show module`
`show vpc brief`
`show vpc role`
`show running-config vpc`
`show system internal vpcm event-history global`
`show system internal vpcm event-history errors`
`show system internal vpcm event-history msgs`
`show system internal vpcm event-history interactions`
`show system internal vpcm mem-stats detail`
`show system internal vpcm info all`
`show system internal vpcm info global`
`show CFS internal ethernet-peer database`
`show spanning-tree`
Most often information about other components would be needed as well, so best is
to start with sh tech detail this includes in it sh tech vpc and most other
relevant outputs
Cisco Confidential
211
vPC Config Sync
Cisco Confidential
212
NX-OS 5.0(2)N2(1)
Nexus 5500 Config-Sync

Overview
Starting from NX-OS 5.0.2 release, the Nexus 5500 introduces the config-sync feature for
vPC. Config-sync allows administrators to make configuration changes on one switch and
have the system automatically synchronize to its peers. This eliminates any user prone
errors & reduces the administrative overhead of having to configure both vPC members
simultaneously.
fex associate 100
switchport mode fex-fabric
channel-group 5
Presentation_ID
PO5
Cisco Confidential
fex associate 100
channel-group 5

vPC + config sync interaction
Config-sync works in conjunction with vPC.
Configuration transport is only carried over mgmt0 interface using
CFSoIP.
Configuration Sync
Peer-Keepalive
Peer Link
interface mgmt0
ip address 10.29.170.7
interface mgmt0
ip address 10.29.170.8
vpc domain 10
PO5
vpc domain 10
It is recommended to configure the vPC peer-keepalive link to run in mgmt VRF using the mgmt
interface
Presentation_ID
Cisco Confidential

What features are supported with config sync?
Config sync is used to ensure configuration consistency between
peers who require it (i.e. vPC peers). Under the switch-profile the
following features are configurable for synchronization
VLANs
ACLSs
STP
QOS
Interface Level Configurations:
(Ethernet Interfaces)
(Port Channel Interfaces)
(vPC Interfaces)
The following are NOT automatically synchronized
Must be configured manually on each switch
Enabling the specific feature set (i.e. feature vpc, feature vlan, etc)
vPC Domain Configuration
vPC peer-keepalive configuration
FCOE configurations (not supported in a switch-profile)
Presentation_ID
Cisco Confidential

Prerequisites 3 steps required
Config sync feature is supported today on the Nexus 5500 platform running 5.0.2. In
addition, CFSoIP, Switch-profiles, and Peer-configuration must be configured on each peer
CFSoIP
Switch-profile
Peer Configuration
Transport protocol for the

configuration across peers
Used to create the config that

needs to be sync across peers
To indicate which peer will

receive the configuration
Both peers need to have

CFSoIP enabled
Both peers require identical

switch profiles
Both peers require to configure

each other as their peer
Step 1:
Step 2:
Step 3:
N5500-1# config t
N5500-1(config)# CFS ippv4 distribute
N5500-1# config sync

N5500-1(config-sync)# switch-profile
Apple

Apple
N5500-1(config-sync)# sync-peers
destination 10.29.170.8
N5500-2# config t

Apple

Apple
N5500-2(config-sync)# sync-peers
destination 10.29.170.7
Only one switch profile per switch is configurable today.

A new mode config sync, similar to config t is introduced to create switch-profiles
Presentation_ID
Cisco Confidential

Config-Sync example New Switch
This example assumes that N5Ks are new switches that will be configured for vPC. It is
assumed that only the basic vPC parameters have been enabled for vPC to operate
Enable CFSoIP
N5500-1# config t
N5500-2# config t
Configure identical
switch-profile on each
switch
Configure peer
relationship under
switch-profile

N5500-1(config-sync)# switch-profile Apple

N5500-1(config-sync)# sync-peers destination 10.29.170.8
Cisco Confidential
217

Continued
Enter all the config
under the switchprofile and VERIFY
config show switchprofile buffer
We recommend to copy smaller chunks of the profile to ensure each sync

is smooth
N5K-1(config-sync-sp)# interface Ethernet1/10
<snip>
interface Ethernet100/1/2
switchport trunk allowed vlan 5
<snip>
Once config has been

verified, issue
commit
N5K-1(config-sync-sp)# verify
Verify Successful
N5K-1(config-sync-sp)# commit
Commit Successful
Verify the
configuration was
merged sucessfully
N5K-1# sh running-config
Repeat as needed
Cisco Confidential
218

Once a configuration is applied using config-sync, that
config exists under the switch profile
No changes are allowed to the physical interface,
changes must be made within the switch profile
Deleting switch profile deletes the configuration!
DCN-N5K1(config)# interface e199/1/2
DCN-N5K1(config-if)# sw trunk allowed vlan add 200
Error: Command is not mutually exclusive
Command is denied on
physical interface. Config
must be applied under
switch-profile
DCN-N5K1(config-if)# config sync

DCN-N5K1(config-sync)# switch-profile FEX_Ports
Switch-Profile started, Profile ID is 1
DCN-N5K1(config-sync-sp)# interface e199/1/2
DCN-N5K1(config-sync-sp-if)# switchport trunk allowed vlan add 200
DCN-N5K1(config-sync-sp-if)# verify
Verification Successful
DCN-N5K1(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on
amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful
DCN-N5K1# show run int e199/1/2
DCN-N5k2# show run int e199/1/2
!Command: show running-config interface Ethernet199/1/2

!Time: Fri Aug 26 15:02:23 2011
!Command: show running-config interface Ethernet199/1/2

!Time: Fri Aug 26 13:52:47 2011
version 5.0(3)N1(1b)
version 5.0(3)N1(1b)
Cisco Confidential
219

Mutual Exclusion Check
Mutual Exclusion (Mutex) Verifies configuration between inside and outside the profile. If
there is a conflict, a verify or commit will fail. Applies to both adding and removing
configurations from inside/outside profile.
N5500-1#sh run int ether 100/1/3
int ether 100/1/3
Outside of Profile
N5500-1(config-if)# config sync

N5500-1(config-sync)# switch-profile ASwitch-Profile started, Profile ID is 1
N5500-1(config-sync-sp)# int ethernet 100/1/3
N5500-1(config-sync-sp-if)# switchport mode access
Inside of Profile
N5500-1(config-sync-sp-if)# verify
Failed: Verify Failed
N5500-1(config-sync-sp)# show switch-profile A status

Session-type: Commit
Status: Verify Failure
Error(s): Following commands failed mutual-exclusion checks:interface Ethernet100/1/3
switchportmode access
Mismatch between the outside and inside the profile results in a failure in a mutex verify
To resolve this, user needs to manually remove the configuration outside/inside profile
Inside profile includes all the configuration under a switch-profile. Outside profile includes all the
global/interface level configuration that is done outside of a switch-profile
Cisco Confidential
220

Merge Exchange Check
Merge Check occurs after peer-reachability is established in one of two scenarios.
1) Peers interacting for the first time (i.e. after a reload, or a peer being reloaded)
2) Peers interacting after an intermittent network down time. If there is a conflict between
the 2 devices, a verify and commit will fail
N5500-1#sh run switch-profile
Switch-profile Apple
sync-peers destination 10.29.170.8

Switch-profile Apple
sync-peers destination 10.29.170.7
Peer becomes unreachable due to a network outage, config sync will not occur across mgmt0.
vPC peer link is up, but vPC PKL is down due to mgm0 not reachable
Local changes on N5K-1 and N5K-2 are possible
N5500-1(config-sync-sp)# int ethernet
100/1/3
N5500-1(config-sync-sp-if)# switch mode
trunk
N5500-1(config-sync-sp-if)# commit

N5500-2(config-sync-sp)# int ethernet
100/1/3
N5500-2(config-sync-sp-if)# switch mode fexfabric
N5500-2(config-sync-sp-if)# commit
Commit Successful
Commit Successful

Cisco Confidential
221

Merge Exchange Check - continued
Once peer-reachability is established again, the Merge will fail due
to conflicting/overlapping changes. Configuration of peers remains
unchanged.

Peer becomes reachable, mgmt0 is up

N5K-1(config-sync-sp)# sh switch-profile A
status
Profile-status: Merge Failed
Status: Verify Failure
Error(s):
Following commands failed merge checks:
Mismatch both
ethernet1/10 interfaces
results in a failure in a
merge check
To resolve this, user
needs to manually
remove the configuration
outside/inside profile
Cisco Confidential
222

This example assumes that N5Ks are new switches that will be configured for vPC. It is
assumed that only the basic vPC parameters have been enabled for vPC to operate
Enable CFSoIP
N5500-1# config t
N5500-2# config t
Configure identical
switch
Configure peer
relationship under
switch-profile


Cisco Confidential
223

Continued
Enter all the config
under the switchprofile and VERIFY
config show switchprofile buffer
We recommend to copy smaller chunks of the profile to ensure each sync is smooth
<snip>
<snip>
Once config has been

reviewed, issue
commit
Verify the
configuration was
merged sucessfully
Commit Successful
Repeat as needed
Cisco Confidential
224

Config-Sync example (i.e. Dee Why Plus -> Eaglehawk)
This example assumes that N5Ks are already working in vPC, with configurations already
manually synced. User now wants to continue with config-sync
Enable CFSoIP
N5500-1# config t
N5500-1# config t
Configure identical
switch
Import config under

the switch-profile and
VERIFY running
configuration show
switch-profile buffer

Option1:
N5K-1(config-sync-sp)# import running-config
We recommend to copy smaller chunks of the profile to ensure each sync is smooth
Option2:
<snip>
<snip>
Cisco Confidential
225

Config-Sync example (i.e. Dee Why Plus -> Eaglehawk)
Continued
Once reviewed, issue
commit on BOTH
sides to import the
config locally first
Commit Successful
Commit Successful
Then, configure peers

to initiate a merge and
bring both in sync

Verify the
configuration was
merged successfully
! Any failures shall be

reported as mergefailures and need to be
manually corrected
inside/outside the
switch-profile
Repeat as needed
In this example, the peers are defined only after the configurations are put under a profile. The
reason is to eliminate any sync from occurring before user is able to review the configuration
Cisco Confidential
226

Failure Scenarios
Event
Reaction
vPC peer-link down
No impact if config-sync is over mgmt0
CFS keepalive failure
CFS issues a peer not reachable notification,

config-sync becomes non-operational with that
peer
Switch reload
Peer switches get a peer unreachable

notification from CFS and stop communicating
with this switch
Commit failure on peer
Rollback to previously taken checkpoint
Merge failure
Syslogs gets generated and user shall use 'show

switch-profile status' to determine the errors
and correct.
Cisco Confidential
227

ISSU interaction
When ISSU is in progress on a peer, then a 'verify/commit' is not permitted

on this peer
If a commit is issued from other peer, that shall fail only if the peer
undergoing ISSU was still reachable but can't accept configuration due to
ISSU, otherwise the 'commit' will become a local-operation by default
behavior.
When a verify/commit is in progress between the peers, then ISSU shall be

blocked on both peers. However, if there is no reachability then a localcommit on one peer won't be affect ISSU on the other peer.
Cisco Confidential
228

Heads up !
It is recommended to choose only one switch as the initiator. Initiator can be vPC
primary/secondary. The roles are NOT dependent one each other.
Commit should be issued on initiator. Only one session (verify/commit/merge) can be in
progress at a time. A session attempted while another session is in progress shall fail
All configuration changes are prevented when a switch-profile session is in progress i.e.
even changes through config-terminal for all supported commands (ACL, QoS etc) are
also blocked when a session is in progress.
Ensure that the specific feature is enabled on each switch (i.e. feature vpc, feature vlan,
etc
When migrating to config-sync (vPC is running with configurations already synced),
ensure you add smaller sections under the profile and commit versus doing everything
in one chunk
vPC and config sync are independent features. If peer-link is down, config-sync will still
work
Config sync is ONLY transported across mgmt0 interface
FEX pre provisioning can also be done using switch-profiles.

Cisco Confidential
229
Config Rollback
Cisco Confidential
230
Nexus 5500 Config Rollback

Overview
Starting from NX-OS 5.0(2) release, the Nexus 5500 will introduces the config rollback
feature. This feature allows the end user to take a snapshot (checkpoint) of the Cisco NXOS configuration and then reapply that configuration to the device at any point without
have to reload the device. A rollback allows any authorized admin to apply the checkpoint
configuration without requiring expert knowledge of features configured in a checkpoint
Configuration
checkpoint
Todays
configuration
-------------------
-------------------
Checkpoint
running-config
Current
running-config
-------------------
-------------------
User wants to revert back to

the original configuration
Prior to 5.0(2), the system required a reload to run another configuration file
Cisco Confidential
231

Overview
User can create a checkpoint copy of the current running configuration at any
time. Cisco NX-OS saves this checkpoint as an ASCII file that can be used to
rollback the running configuration to the checkpoint configuration at a future
time. Multiple checkpoints can be saved with different versions of the running
configuration
Todays
configuration
Configuration
Checkpoints
-------------------
-------------------------------------------------------------------------------------------
-------------------
Checkpoint_1
running-config
Checkpoint
Checkpoint
Checkpoint
Checkpoint_10
Checkpoint
running-config
running-config
running-config
running-config
running-config
Current
running-config
-------------------
-------------------------------------------------------------------------------------------
-------------------
Up to 10 Checkpoints Configuration file can be created
You can create up to 10 Checkpoint Configuration Files

All files names must be unique
Cisco Confidential
232

How to create a rollback
The global checkpoint CLI command in exec mode creates a configuration
checkpoint. The user can create a checkpoint in 2 different ways. These
checkpoints will be saved upon reboot.
In the following examples we are calling the checkpoint file Test-Config
N5K-1(config)# checkpoint Test-Config
........Done
!Checkpoint saved in /isan/etc/checkpoint director on the switch
N5K-1(config)# checkpoint file bootflash:Test-Config

......Done
!Checkpoint saved in bootflash
Cisco Confidential
233

How to verify the config captured for a rollback
The user can verify the configuration that is captured in a checkpoint before
executing a rollback.
N5K-1(config)# show checkpoint ?

<CR>
>
>>
Redirect it to a file in append mode
Test-Config
Checkpoint name
all (no abbrev)
Show default config
summary (no abbrev) Show configuration rollback checkpoints summary
system (no abbrev)
Show only system configuration rollback checkpoints
user (no abbrev)
Show only user configuration rollback checkpoints
|
Pipe command output to filter
Cisco Confidential
234

Compare current running config to rollback config file
Prior to executing a rollback, the user can compare the checkpoint configuration
and running-configuration. This way the user can verify the changes that will be
applied once a rollback is committed
N5K-1(config)# show diff rollback-patch ?
checkpoint
Use checkpoint as source configuration
file
Src Checkpoint file
running-config Use running configuration as source
startup-config Use startup configuration as source
N5K-1(config)# show diff rollback-patch checkpoint Test-Config running-config
!This will show the comparison of the configuration saved at the rollback
Test-Config versus the current running configuration
Cisco Confidential
235

How to execute a rollback
When a rollback is trigged, the Nexus 5500 only supports the atomic method. The
atomic rollback implements a rollback only if no errors occur. If an error does
occur, we go back to the last running-configuration the system was using .
N5K-1: rollback running-config checkpoint Test-Config
Note: Applying config parallelly may fail Rollback verification
Collecting Running-Config
Generating Rollback patch for switch profile
Rollback Patch is Empty
Collecting Running-Config
#Generating Rollback Patch
Rollback Patch is Empty
Rollback completed successfully.
Nexus 5500 only supports atomic rollback at FCS

Cisco Confidential
236

Heads up !
We dont support config rollback for fiber channel interface/configuration.
The CLI will get disabled if feature fcoe is enabled
The Nexus 5500 only supports atomic rollback. If an error is encountered
(i.e. a command does not go through), we will rollback to the show
running-config at the time when rollback was issued
N5K does not support auto checkpoints, only manually configured ones
If you create a configuration checkpoint and upgrade or downgrade to a

different software release, the rollback procedure is not officially supported.
However, the rollback procedure may still work depending on the
configuration changes being executed
Cisco Confidential
237
Multicast
Cisco Confidential
238
Nexus 5500 Multicast Forwarding

Fabric-Based Replication
Nexus 5500 use fabric
based egress replication
Multicast Frames
are Queued in
dedicated
multicast queues
on Ingress
Traffic is queued in the

ingress UPC for each
MCAST group
MCAST packet is
replicated in the
Fabric
Unified Crossbar
Fabric
Eth 1/8
Multicast
Scheduler
Eth 1/20
When the scheduler

permits the traffic if
forwarded into the fabric
and replicated to all
egress ports
When possible, traffic is
super-framed (multiple
packets are sent with a
single fabric scheduler
grant) to improve
throughput
Cisco Confidential
239
Nexus 5500
Multicast Fabric Replication (Animated)
Ingress Interface
Switch
Fabric
Unicast VOQ
Egress
Interface
Packet
Buffer
Mcast
Ucast
Mcast
A
B
Mcast
Multicast VOQ
Mcast
128 MCAST VOQ per port

4 Crosspoints Shared across
unicast and MCAST
8 Dedicated Egress MCAST Queues
Cisco Confidential
240

Nexus 5500 Data Plane Changes
Nexus 5500 supports 4000 IGMP snooping entries
Dedicated Unicast & Multicast Queuing and Scheduling

Resources
8 for egress queues for unicast and 8 for multicast
4 Egress cross-points (fabric buffer) per egress port
Out of 4 fabric buffer, one is used for unicast, one for

multicast and two are shared between unicast and
multicast
Two configurable Multicast scheduler modes
Overloaded mode (Proxy Queue)
Multicast
Scheduler
4 Fabric
Crosspoints
per port (10K
X-Bar buffer)
Congested egress ports are ignored
Multicast packets are sent to non-congested port only
Reliable mode
Presentation_ID
...
Packets are sent to switch fabric when all OIF ports are
ready, ie, have fabric buffer and egress buffer to accept
the multicast packets
Cisco Confidential
8 Dedicated
Egress MCAST
Queues per Port
8 Dedicated
Egress UCAST
Queues per Port
Multicast Optimization and VOQ Assignment

128 Multicast VOQ for each ingress port. Separate VOQ for multicast and
unicast traffic
One multicast VOQ per class of service without multicast optimization
Multicast optimization can be turned on for one class of service
With multicast optimization multicast traffic assigned to VOQ based on fanout

With multicast optimization
Without multicast
optimization
Multicast VOQ
Multicast VOQ
Class 1
Class 1
Class 2
Class 3
Q1
Q2
Class 2
Q3
Class 3
Class 8
Class 8
Q8
Q8
class with
multicast
optimization
Q127
Q128
Q128
Q3
Q9
Q127
Presentation_ID
Q1
Q2
Cisco Confidential
Multicast Optimization Configuration

Multicast optimization is turned on by default for class-default. It means
all multi-destination traffic will be assigned to multicast VOQ according to
their fanout
Multi-destination traffic includes:
IP multicast
Unknown unicast flooding
Broadcast traffic
L2 multicast traffic
User can choose to turn on multicast optimization for selected multidestination traffic, such as, IP multicast traffic
Multicast optimization can only be turned on for one system class.
8 multicast VOQ reserved for QoS queuing. The rest of 120 queues for
multicast optimization
Presentation_ID
Cisco Confidential
Multicast Optimization Sample Configuration

Multicast optimization can be turned on for user defined system
class.
Multicast optimization for class-default will be disabled
automatically
No change for unicast traffic
N5k(config-cmap-qos)# policy-map type qos Mcast_optimize

N5k(config-pmap-qos)# class type qos class-ip-multicast
N5k(config-pmap-c-qos)# set qos-group 2
N5k(config-pmap-c-qos)# exit
N5k(config-pmap-qos)# class type network-qos IP_mcast
N5k(config-cmap-nq)# match qos-group 2
N5k(config-cmap-nq)# policy-map type network-qos Mcast_optimize
N5k(config-pmap-nq)# class type network-qos IP_mcast
N5k(config-pmap-nq-c)# multicast-optimize
N5k(config-pmap-nq-c)# queue-limit 170000
Presentation_ID
Cisco Confidential

Nexus 5500 Data Plane Changes
Proxy queues to detect congestion at egress
One proxy queue for each hardware egress queue
Bytes are added to proxy queue when packets arrive
at egress hardware queue
Proxy queues are drained at 98% of port speed
using DWRR
When proxy queue is full egress port sends
overload message to central scheduler
Central scheduler excludes the port in multicast
scheduling calculation when overload bit is set AND
there is no fabric buffer available. Multicast packet
is sent over to non-congested port
In case of congestion there is a delay for proxy
queue to signal overload
N5k(config)#hardware multicast disable-slow-port-pruning
Presentation_ID
Cisco Confidential
...
Multicast
Scheduler
Proxy Queue sends overload

signal to scheduler when port
congested
Multicast Load-sharing Over Port-Channel

Load-sharing influenced by ingress port and VOQ number
Each interface is assigned a unique seed number for hash
calculation
1.1.1.1 224.1.1.2
Multicast optimization (turned on by default for class-default)

required for better distribution.
The Port-Channel load-sharing option configuration doesnt apply to
multicast traffic
Multicast MAC Table Lookup

OIF : 1/2 ,1/3, Po10(1/10, 1/11)
VOQ # 20
Hashing calculation
Choose 1/10 for Po10
Seed number
for eth1/1
1/10
VOQ # 20
Request to central scheduler
with OIF 1/2, 1/3 and 1/10
1/10
Source
Po10
Receivers
1/11
1/2
1/1
Switch fabric replicates
packets to 1/2 , 1/3 and 1/10
1/3
Cisco Confidential
249
Nexus 5500
Station (MAC) Table allocation
Nexus 5500 has a 32K Station table entries
4k reserved for multicast (Multicast MAC addresses)
3k assumed for hashing conflicts (very conservative)
Nexus 5500
UPC
Station Table
32k entries
25k effective Layer 2 unicast MAC address entries
4k entries for
IGMP
3k entries for potential hash collision space
25k effective MAC entries for unicast
Cisco Confidential
250
IGMP snooping
As a Layer 2 switch the N5k performs IGMP snooping
IGMP snooping constrains multicast traffic only to the
ports that need to receive it.
32k entries
MAC table
4k entries for IGMP
Cisco Confidential
251
Cisco Nexus 5500

Multicast
Cisco Confidential
252
Multicast
In Cisco NX-OS:
PIM and MSDP protocols require a LAN Enterprise Services license.
The global ip multicast-routing command does not exist in NXOS and is not required to
enable multicast forwarding/routing. (It is required in Cisco IOS Software to enable multicast
forwarding/routing)
PIM command-line interface (CLI) configuration and verification commands are not available
until you enable the PIM feature with the feature pim command.
MSDP CLI configuration and verification commands are not available until you enable the
MSDP feature with the feature msdp command.
IGMP versions 2 and 3 are supported. IGMP version 1 and Version 3 Lite are not supported.
An IGMP Snooping Querier is configured under the layer-2 VLAN with the ip igmp snooping
querier CLI command (Physical L3 interfaces cannot be configured as IGMP Snooping
Queriers). In Cisco IOS Software, an IGMP Snooping Querier is configured under the layer-3
interface.
PIM version 2 Sparse Mode is supported. Cisco NX-OS does not support PIM version 1
Sparse Mode or Dense Mode. The NX-OS cannot fallback to Dense Mode operation.
When configuring a PIM Auto-RP Candidate or BSR RP-Candidate the NX-OS requires a
configured group-list (i.e. x.x.x.x/x), whereas Cisco IOS Software defaults to 224.0.0.0/4. An
optional standard ACL can be configured to specify multicast groups in Cisco IOS Software.
When configuring PIM Auto-RP Mapping-Agent's or Candidate-RP's, Cisco NX-OS uses a

default scope of 32, whereas Cisco IOS Software requires it to be specified with the scope
option (1-255).
Cisco Confidential
253
Multicast
In Cisco NX-OS:
When configuring PIM Auto-RP, Cisco NX-OS multicast devices must be enabled to listen
and/or forward RP advertisements with the ip pim auto-rp forward listen global CLI
configuration command. Cisco IOS Software has to be configured for Sparse-Dense Mode or
Sparse Mode with the global ip pim autorp listener CLI configuration command.
When configuring PIM BSR, Cisco NX-OS multicast devices must be enabled to listen
and/or forward RP advertisements with the ip pim bsr forward listen global CLI configuration
command. Cisco IOS Software doesnt require additional configuration, but does not have
the ability to enable/disable RP forwarding and listening capabilities.
BSR-Candidate routers have a default priority of 64. Cisco IOS Software defaults to 0. The
priority value can be configured between 0 255 in both operating systems using the priority
option. A higher numeric value is preferred when comparing priorities.
BSR RP-Candidate routers have a default priority of 192. Cisco IOS Software defaults to
0. The priority value can be configured between 0 255 in both operating systems using the
priority option. The lower numeric value is preferred when comparing priorities.
When configuring a Static-RP, the NX-OS does not have an override option like Cisco IOS
Software that forces the Static-RP to be elected for its specified multicast group list. Cisco
IOS Software prefers dynamically learned RPs over Static RPs if the override option is not
configured.
When comparing PIM Static-RPs to dynamically learned RPs (Auto-RP and BSR) during
the election process: The RP with the most specific multicast group-list is elected. If the
group-lists are identical, the router with the highest RP IP address is elected.
Cisco Confidential
254
Multicast
In Cisco NX-OS:
When configuring a PIM domain border, the ip pim border interface CLI command
prevents BSR and Auto-RP packets from being sent or received on an interface. The
Cisco IOS Software command equivalent (ip pim bsr-border) only prevents BSR
packets. Cisco IOS Software requires the ip multicast boundary interface command to
prevent Auto-RP packets.
PIM neighbor authentication (IPSec ah-md5) can be enabled to authenticate directly
connected neighbors to increase security. Cisco IOS Software does not support this
functionality.
PIM neighbor logging can be enabled with the global ip pim log-neighbor-changes
CLI command. (Cisco IOS Software enables PIM neighbor logging by default)
The data in the MSDP Source-Active (SA) messages are cached by default,
whereas Cisco IOS Software requires the global ip msdp cache-sa-state and ip msdp
cache-rejected-sa CLI commands.
PIM is configured with the Source Specific Multicast (SSM) group range 232.0.0.0/8
by default (ip pim ssm range 232.0.0.0/8).
PIM does not support Bidirectional Forwarding Detection (BFD) for rapid failure
detection on the Nexus 5500 series yet, but it is being targeted for the Goldcoast
release. However, on the Nexus 7000 series, beginning with NX-OS 5.0(2a), PIM
supports BFD.
Cisco Confidential
255
Multicast
If you remove the feature pim command, all relevant PIM configuration
information is also removed.
If you remove the feature msdp command, all relevant MSDP configuration
information is also removed.
IGMP Snooping is enabled globally by default. It can be disabled globally, or
per layer-2 VLAN with the no igmp snooping command.
IGMP version 2 is enabled by default when PIM Sparse Mode is configured on
an interface.
PIM configuration is supported under IP Tunnel (GRE) interfaces in Cisco NXOS 5.2(1) and onward (PIM was previously not supported in IP Tunnels).
PIM supports three modes of operation: Any Source Multicast (ASM), Single
Source Multicast (SSM), Bidirectional Shared Tree (Bidir). The default mode is
ASM. Bidir can be configured with the bidir option when configuring a RP.
The Cisco NX-OS supports four types of PIM Rendezvous Points: Static,
Bootstrap router (BSR), Auto-RP and Anycast-RP. (Do not configure Auto-RP and
BSR in the same network)
Cisco Confidential
256
Multicast
When configuring a PIM Static-RP, the group-list defaults to 224.0.0.0/4 if one
is not specified.
The Cisco NX-OS has two different CLI syntax options when configuring BSR
and Auto RP's (New Cisco NX-OS syntax, and backwards compatible Cisco IOS
Software syntax).
The Cisco NX-OS supports multicast routing per layer-3 Virtual Routing and
Forwarding (VRF) instance.
PIM SSM and Bidir are not supported on Virtual Port-Channels (vPCs).
A topology that has a PIM router connected to a pair of Cisco Nexus 5500
Platform switches through vPC is not supported.
Configure candidate RP intervals to a minimum of 15 seconds.
A vPC peer link is a valid link for IGMP multicast forwarding.
If the vPC link on a switch is configured as an output interface (OIF) for a

multicast group or router port, the vPC link on the peer switch must also be
configured as an output interface for a multicast group or router port.
In SVI VLANs, the vPC peers must have the multicast forwarding state
configured for the vPC VLANs to forward multicast traffic directly through the
vPC link instead of the peer link.
Cisco Confidential
257
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Enabling Multicast Forwarding

ip multicast-routing
The Cisco NX-OS does not have a single

global command to enable multicast
forwarding/routing.
Enabling the PIM Feature

ability to enable or disable PIM.
feature pim
Configuring PIM Sparse Mode on an Interface

interface TenGigabitEthernet1/1
ip address 192.168.10.1 255.255.255.0
ip pim sparse-mode
ip address 192.168.10.1/24
ip pim sparse-mode
Cisco Confidential
258
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a PIM Auto-RP
interface Loopback10 I
p address 172.16.1.1 255.255.255.255
ip pim sparse-mode
ip pim send-rp-announce Loopback10
scope 32
ip pim send-rp-discovery Loopback10
scope 32
ip pim autorp listener
interface loopback10
ip address 172.16.1.1/32
ip pim sparse-mode
ip pim auto-rp rp-candidate loopback10
group-list 224.0.0.0/4
ip pim auto-rp mapping-agent loopback10
ip pim auto-rp forward listen
or
ip pim send-rp-announce loopback10
group-list 224.0.0.0/4
ip pim send-rp-discovery loopback10
ip pim auto-rp forward listen
Cisco Confidential
259
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a PIM BSR RP

ip address 172.16.1.1/32
ip pim sparse-mode
interface Loopback10
ip address 172.16.1.1 255.255.255.255
ip pim sparse-mode
ip pim bsr-candidate Loopback10
ip pim rp-candidate Loopback10
ip pim bsr bsr-candidate loopback10

ip pim bsr rp-candidate loopback10 grouplist 224.0.0.0/4
ip pim bsr forward listen
or
ip pim bsr-candidate loopback10
ip pim rp-candidate loopback10 group-list
224.0.0.0/4
Cisco Confidential
260
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a PIM Anycast-RP (BSR Example)

interface loopback0
ip address 192.168.10.1/32
ip pim sparse-mode

ability to enable the PIM Anycast RP
feature.
description Anycast-RP-Address
ip address 172.16.1.1/32
ip pim sparse-mode
ip pim bsr bsr-candidate loopback0
ip pim bsr rp-candidate loopback10 grouplist 224.0.0.0/4
ip pim anycast-rp 172.16.1.1 192.168.10.1
ip pim anycast-rp 172.16.1.1 192.168.10.2
Cisco Confidential
261
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a PIM Static-RP

ip pim rp-address 172.16.1.1
Configuring PIM Neighbor Authentication

ip address 192.168.10.1/24
ip pim sparse-mode
ability to enable neighbor authentication.
ip pim hello-authentication ah-md5 3
a667d47acc18ea6b
Cisco Confidential
262
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring a PIM BSR Border on an Interface

interface TenGigabitEthernet1/1 I
p address 192.168.10.1 255.255.255.0
ip pim bsr-border
ip pim sparse-mode
ip multicast boundary 10
access-list 10 deny 224.0.1.39
access-list 10 deny 224.0.1.40
access-list 10 permit 224.0.0.0
15.255.255.255
ip address 192.168.10.1/24
ip pim sparse-mode
ip pim border
Cisco Confidential
263
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring PIM in a Non-Default VRF Instance

ip vrf production
ip multicast-routing vrf production
ip vrf forwarding production
ip address 172.16.1.1 255.255.255.255
ip pim sparse-mode
ip vrf forwarding production
ip address 192.168.10.1 255.255.255.0
ip pim sparse-mode
vrf context production

ip pim rp-address 172.16.1.1 group-list
224.0.0.0/4
vrf member production
ip address 172.16.1.1/32
vrf member production
ip address 192.168.10.1/24
ip pim sparse-mode
ip pim vrf production rp-address 172.16.1.1
Cisco Confidential
264
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring IGMP Version 3 for an Interface

ip address 192.168.10.1 255.255.255.0
ip pim sparse-mode
ip igmp version 3
ip address 192.168.10.1/24
ip pim sparse-mode
ip igmp version 3
Configuring an IGMP Snooping Querier for a VLAN

interface Vlan10
ip address 192.168.10.1 255.255.255.0
ip igmp snooping querier
vlan 10
ip igmp snooping querier 192.168.10.1
Note: there is no subnet mask on the IP address of the nexus querier config
command.
Cisco Confidential
265
Multicast
Cisco IOS CLI
Cisco NX-OS CLI
Configuring MSDP (Anycast-RP)

interface Loopback0
description MSDP Peer Address
ip address 192.168.1.1 255.255.255.255
interface loopback0
description MSDP Peer Address
ip address 192.168.1.1/32
description PIM RP Address
ip address 1.1.1.1 255.255.255.255
description PIM RP Address
ip address 1.1.1.1/32

ip msdp peer 192.168.2.1 connect-source
Loopback0
ip msdp cache-sa-state
ip pim rp-address 1.1.1.1 group-list

224.0.0.0/4
ip msdp peer 192.168.2.1 connect-source
loopback0
Cisco Confidential
266
Multicast
Cisco IOS Software

Command Description
Interface
show ip igmp groups
show ip igmp groups
show ip igmp interface

brief
show ip igmp interface inttype
show ip igmp interface vrf
name
show ip igmp local-groups
int-type
show ip igmp local-groups
vrf name
show ip igmp route

int-type
show ip igmp vrf name
-
show ip igmp route x.x.x.x show ip igmp route int-type -
Displays all IGMP attached group

membership information
Displays IGMP information for all interfaces
Displays a one line summary status per
interface
Displays IGMP information for a specific
interface
Displays IGMP information for a specific
VRF instance
Displays IGMP local groups associated to a
specific interface
Displays IGMP local groups associated to a
specific VRF instance
Displays IGMP attached group membership
information
for a specific group
for a specific interface
Cisco Confidential
267
Multicast
Cisco IOS Software

Command Description
Interface
show ip igmp route vrf

name
show ip igmp snooping

explicit-tracking
groups
mrouter
otv
querier
statistics
vlan #
Displays IGMP attached group

membership for a specific VRF instance
Displays global and per interface IGMP
Snooping information
show ip igmp snooping Displays explicit tracking information for
explicit-tracking
IGMPv3
show mac-address-table Displays IGMP Snooping groups
multicast igmp-snooping information
Displays detected multicast routers
mrouter
Displays IGMP Snooping OTV
information
Displays IGMP Snooping querier
information
Displays packet/error counter statistics
statistics
Displays IGMP Snooping information per
specific VLAN
Cisco Confidential
268
Multicast
Cisco IOS Software

Interface
show ip msdp count

show ip msdp count
show ip msdp mesh-group show ip msdp peer
show ip msdp peer
show ip msdp peer x.x.x.x show ip msdp peer x.x.x.x
show ip msdp peer vrf
show ip msdp vrf name
name
show ip msdp peer policy show ip msdp peer route show ip msdp sa-cache
show ip msdp sa-cache
show ip msdp source
show ip msdp summary
show ip msdp summary
Command Description
Displays MSDP SA cache counters
Displays MSDP Mesh-Group members
Displays all MSDP peers
Displays a specific MSDP peer
Displays MSDP peers related to a
Displays the MSDP peer policies
Displays the MSDP route-cache
Displays the MSDP SA route-cache
Displays the MSDP learned sources
and associated statistics
Displays the MSDP peer
Cisco Confidential
269
Multicast
show ip pim df
show ip pim df x.x.x.x
show ip pim df vrf name
show ip pim group-range
show ip pim group-range
x.x.x.x
show ip pim group-range vrf
name
show ip pim interface
show ip pim interface brief
x.x.x.x
Cisco IOS
Command Description
Software Interface
show ip pim interface df Displays Bidir designated forwarders
show ip pim interface df Displays Bidir designated forwarders for
x.x.x.x
a specific RP or group
Displays Bidir designated forwarders for
a specific VRF instance
Displays the PIM group-ranges
-
Displays a specific PIM group-range
show ip pim interface int-type
show ip pim interface

int-type
show ip pim interface vrf

name
Displays the PIM group-ranges for a

Displays all PIM enabled interfaces
Displays a one line summary of all PIM
enabled interfaces
Displays information for a specific PIM
interface
Displays the PIM interfaces for a
Cisco Confidential
270
Multicast
show ip pim neighbor
show ip pim neighbor x.x.x.x
interface int-type
show ip pim neighbor vrf
name
show ip pim oif-list x.x.x.x
show ip pim policy statistics
show ip pim route
show ip pim route x.x.x.x
show ip pim route vrf name
show ip pim rp
show ip pim rp x.x.x.x
Cisco IOS
Command Description
Software Interface
x.x.x.x
int-type
Displays all PIM neighbors

Displays a specific PIM neighbor for a
specific IP address
Displays a specific PIM neighbor for a
specific interface
Displays PIM neighbors for a specific
VRF instance
Displays PIM OIF-List for a specific
multicast group address
Displays PIM statistics
Displays PIM routes
Displays a specific PIM route
Displays PIM routes for a specific VRF
instance
show ip pim rp mapping Displays PIM RP information
Displays information for a specific PIM
group address
Cisco Confidential
271
Multicast
Cisco IOS
Command Description
Software Interface
show ip pim rp vrf name
show ip pim rp-hash x.x.x.x
show ip pim statistics

show ip pim statistics vrf
name
show ip pim vrf name
show ip mroute
show ip mroute summary
show ip mroute x.x.x.x
show ip mroute vrf name
Displays information for PIM RP's in a

show ip pim rp-hash
Displays PIM RP-Hash value for a
x.x.x.x
specific group
Displays PIM packet statistics
Displays per packet statistics for a
Displays detailed PIM information per
show ip mroute
Displays the multicast routing table
show ip mroute
Displays the multicast routing table with
summary
packet counts and bit rates
show ip mroute x.x.x.x Displays a specific multicast route
show ip mroute vrf
Displays the multicast routing table for a
name
Cisco Confidential
272
Multicast
Cisco IOS Software

Interface
Command Description
Displays information for a specific PIM group

address
Displays information for PIM RP's in a specific
show ip pim rp vrf name
VRF instance
show ip pim rp-hash
Displays PIM RP-Hash value for a specific
show ip pim rp-hash x.x.x.x
x.x.x.x
group
show ip pim statistics
Displays PIM packet statistics
show ip pim statistics vrf
Displays per packet statistics for a specific
name
VRF instance
Displays detailed PIM information per specific
VRF instance
show ip mroute
show ip mroute
Displays the multicast routing table
Displays the multicast routing table with
packet counts and bit rates
Displays a specific multicast route
Displays the multicast routing table for a
Displays the Reverse Path Forwarding (RPF)
show ip route rpf
show ip rpf
table used for multicast source lookup
Cisco Confidential
273
QoS
Cisco Confidential
274
Nexus 5500 QoS

QoS Capabilities and Configuration
Nexus 5500 supports a new set of QoS capabilities designed to
provide per system class based traffic control
Lossless EthernetPriority Flow Control (IEEE 802.1Qbb)
Traffic ProtectionBandwidth Management (IEEE
802.1Qaz)
Configuration signaling to end pointsDCBX (part of IEEE

802.1Qaz)
These new capabilities are added to and managed by the
common Cisco MQC (Modular QoS CLI) which defines a threestep configuration model
Define matching criteria via a class-map
Associate action with each defined class via a policy-map
Apply policy to entire system or an interface via a servicepolicy
Nexus 5500/7000 leverage the MQC qos-group capabilities to
identify and define traffic in policy configuration
Cisco Confidential
275
Supported QoS Features

Eight class of service with eight hardware queue
Two reserved for internal control traffic
DSCP, CoS or ACL based classification at ingress

DSCP marking and CoS marking
Support no-drop class of service to achieve lossless end-toend
MTU per class of service
Queuing and bandwidth management
Strict priority queue and DWRR (Deficit Weigh Round
Robin)
Buffer tuning for drop and no-drop class
Cisco Confidential
276
DSCP Marking
Only available with Nexus 5500 platform
Configured with policy-map type qos
Independent of CoS marking
Without DSCP marking the DSCP value in the incoming packets is

preserved
ip access-list High-ACL
10 permit ip 30.30.1.0/24 any
class-map type qos match-all High-ACL
match access-group name High-ACL
policy-map type qos Policy-Classify
class High-ACL
set qos-group 2
set dscp 46
Cisco Confidential
277
Nexus 5500 QoS

QoS Policy Types
There are three QoS policy types used to define
system behavior (qos, queuing, network-qos)
There are three policy attachment points to
apply these policies to
Ingress UPC
Unified Crossbar
Fabric
Ingress interface
System as a whole (defines global behavior)
Egress UPC
Egress interface
Policy Type
Function
Attach Point
qos
Define traffic classification rules
system qos
ingress Interface
queuing
Strict Priority queue

Deficit Weight Round Robin
system qos
egress Interface
ingress Interface
network-qos
System class characteristics (drop or nodrop, MTU), Buffer size, Marking
system qos
Cisco Confidential
278
Nexus 5500 QoS

UPC (Gen 2) QoS Defaults
QoS is enabled by default (not possible to turn it off)
Three default class of services defined when system
boots up
Two for control traffic (CoS 6 & 7)
Gen 2 UPC
Default Ethernet class (class-default all others)

Cisco Nexus 5500 switch supports five user-defined
classes and the one default drop system class
FCoE queues are not pre-allocated
Unified Crossbar
Fabric
When configuring FCoE the predefined service

policies must be added to existing QoS
configurations
# Predefined FCoE service policies
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
Gen 2 UPC
Cisco Confidential
279
Nexus 5500 QoS

UPC (Gen 2) QoS Capabilities (*Not Currently Supported)
VoQs for unicast
If buffer usage crosses threshold:
Tail drop for drop class
Assert pause signal to MAC
for no-drop system class
Classify
CoS/DSCP
L2/L3/L4 ACL
MAC
Traffic
Classification
Ingress
Cos/DSCP
Marking
Ingress
Policing*
MTU
checking
Per-class
Buffer usage
Monitoring
(8 per egress port)
Central
Scheduler
128 muticast queues
Proxy Queues
Egress Queues
PAUSE ON/OFF signal
unicast
MAC
Egress
COS/DCSP
Marking
UPC Gen 2
ECN
Marking*
Egress
Policing*
Crossbar
Fabric
Truncate or drop
packets if MTU is violated
Egress
scheduling
multicast
Strict priority +
DWRR scheduling
Cisco Confidential
280
Nexus 5000 Traffic Classification

Packets are classified at ingress forwarding engine
No egress classification
Classification occurs before queuing
Classification rules share the 2K TCAM space with other features

192 CAM entries for QoS classification rules
Port ACL
VLAN ACL
SPAN
Control Traffic redirection
Matching Criteria
CoS MAC
IP, UDP/TCP port, DSCP, IP Precedence
Protocol Type
Traffic is assigned to one of 8 qos-group

Qos-group is internal to Nexus 5000
Each qos-group represents one class of service
Queueing and network-qos policy are applied to qos-group after classification
Cisco Confidential
281
Scheduling and Bandwidth Sharing

Each qos-group is mapped to one egress queue
Scheduler controls how bandwidth is shared among 8 egress
queues
Control traffic is mapped to strict priority queue

One qos-group can be mapped to strict priority queue
Non-strict priority queues share bandwidth using Deficit Weight
Round Robin (DWRR)
Schedule the queue
Schedule the queue
N
Is control traffic
SP queue empty
N
Y
Is user
SP queue empty
Schedule non-SP queue

Using DWRR
Cisco Confidential
282
Nexus 5500 QoS

UPC (Gen 2) Buffering
640KB dedicated packet buffer per one 10GE port
Buffer is shared between ingress and egress with majority of buffer
being allocated for ingress
Ingress buffering model
Buffer is allocated per system class
Egress buffer only for in flight packet absorption
Buffer size of ingress queues for drop class can be adjusted using
network-qos policy
Class of Service
Ingress Buffer(KB)
Egress Buffer(KB)
78
18.0 & 18.0
19
9.6 & 9.6
User defined no-drop class of service

with MTU<2240
78
19
User defined no-drop class of service

with MTU>2240
88
19
User defined tail drop class of service

with MTU<2240
User defined tail drop class of service
with MTU>2240
Class-default
22
19
29
19
All remaining buffer
19
Class-fcoe
Sup-Hi & Sup-Lo
Default
Classes
Cisco Confidential
283
Nexus 5500 QoS

Priority Flow Control and No-Drop Queues
Nexus 5000 supports a number of new QoS concepts

and capabilities
Priority Flow Control is an extension of standard 802.3x
pause frames
No-drop queues provide the ability to support loss-less
Ethernet using PFC as a per queue congestion control
signaling mechanism
Cisco Confidential
284
Nexus 5500 QoS

Actions when congestion occurs depending on
policy configuration
PAUSE upstream transmitter for lossless
traffic
Tail drop for regular traffic when buffer is
exhausted
Priority Flow Control (PFC) or 802.3X PAUSE
can be deployed to ensure lossless for
application that cant tolerate packet loss
Buffer management module monitors buffer
usage for no-drop class of service. It signals
MAC to generate PFC (or link level PAUSE)
when the buffer usage crosses threshold
FCoE traffic is assigned to class-fcoe, which is
a no-drop system class
Other class of service by default have normal
drop behavior (tail drop) but can be configured
as no-drop
SFP SFP SFP SFP
1. Congestion
or Flow
Control on
Egress Port
Egress
UPC
2. Egress
UPC does not
allow Fabric
Grants
Unified
Crossbar
Fabric
3. Traffic is
Queued on
Ingress
ingress
UPC
SFP SFP SFP SFP
4. If queue is
marked as nodrop or flow
control then
Pause is sent
Cisco Confidential
285
Nexus 5500 QoS

Tuning of the lossless queues to support a
variety of use cases
Extended switch to switch no drop traffic lanes
Support for 3km with Nexus 5500
Support for 3 km no
drop switch to
switch links
Inter Building DCB
FCoE links
Increased number of no drop services

lanes (4) for RDMA and other multi-queue
HPC and compute applications
Gen 2 UPC
Configs for
3000m no-drop
class
Buffer size
N5020
N5548
Pause Threshold
(XOFF)
Resume
Threshold (XON)
143680 bytes
58860 bytes
38400 bytes
152000 bytes
103360 bytes
83520 bytes
Unified Crossbar
Fabric
Gen 2 UPC
5548-FCoE(config)# policy-map type network-qos 3km-FCoE

5548-FCoE(config-pmap-nq)# class type network-qos 3km-FCoE
5548-FCoE(config-pmap-nq-c)# pause no-drop buffer-size 152000 pause-threshold 103360
resume-threshold 83520
Cisco Confidential
286
Nexus 5500 QoS

MTU per Class of Service (CoS Queue)
MTU can be configured for each class of service (no interface level MTU)
No fragmentation since Nexus 5000 is a L2 switch
When forwarded using cut-through, frames are truncated if they are larger
than MTU
When forwarded using store-and-forward, frames are dropped if they are
larger than MTU
class-map type qos iSCSI
match cos 2
class-map type queuing iSCSI
match qos-group 2
policy-map type qos iSCSI
class iSCSI
set qos-group 2
class-map type network-qos iSCSI
match qos-group 2
policy-map type network-qos iSCSI
class type network-qos iSCSI
mtu 9216
system qos
service-policy type qos input iSCSI
service-policy type network-qos iSCSI
Each CoS queue on the

Nexus 5000 supports a
unique MTU
Cisco Confidential
287
QoS Configuration MQC

MQC(Modular QoS CLI) defines three-step configuration
model
Define matching criteria
class-map
Associate action with each defined class

policy-map
Apply policy to entire system or an interface

service-policy
Cisco Confidential
288
Policy Types
Policy Type
Function
Attach Point
qos
Define traffic classification rules
System qos
Ingress Interface
queuing
Strict Priority queue

Deficit Weight Round Robin
System qos
Egress Interface
Ingress Interface*
System class type(drop or no-drop)

MTU per class of service
Buffer size
Marking
network-qos
System qos
Prefer service policy attached under interface when same type of

service policy is attached at both system qos and interface
Qos and network-qos policy-map are required to create new system
classes
*Queuing policy applied under ingress interface is advertised to server using DCBX protocol
Cisco Confidential
289
Some key commands to remember

class-map and policy-map type qos
Mostly used for classification and marking (for
DSCP)
class-map and policy-map type network-qos
Mostly used for network properties such as queuesize, drop vs no drop / MTU, multicast optimize and
marking (for CoS)
class-map and policy-map type queueing
Mostly used for bandwidth allocation (in egress) and
assigning the priority
Or to communicate the bandwidth allocation to a
CNA (in ingress)
Presentation_ID
Cisco Confidential
Classification Options type qos

Remember the qos-group concept
untagged CoS:
Specifies CoS for untagged frames received on an interface
switch(config)# interface ethernet 1/1
switch(config-if)# untagged cos 5
Or via policy-map type qos:

policy-map type qos classify-5548-global
class voice-global
set qos-group 5
class video-signal-global
set qos-group 4
class critical-global
set qos-group 3
class scavenger-global
set qos-group 2
Presentation_ID
Cisco Confidential
This could be ACL based
Classification Options - type qos

Example of Classification
Order matters
class-map type qos match-any
cfy-video
policy-map type qos classify
match cos 4
match dscp 34
match access-group
class-map type qos match-any

cfy-transact
match cos 2
match dscp 18
match access-group
Presentation_ID
Cisco Confidential
class cfy-video
set qos-group 4
set dscp 34
class cfy-transact
set qos-group 3
set dscp 18
Setting Network Properties type network-qos

Drop/no Drop, MTU, multicast optimize etc
Class-map type network just
matches the qos-group (you
cannot match anything else
class-map type network-qos video
class-map type network-qos nfs
queue-limit <Bytes>
mtu 9216
set cos 2
match qos-group 2
You can set:

MTU
Drop/No Drop
Multicast Optimize
Queue size
CoS (notice DSCP is in type qos)
class type network-qos video
class type network-qos nfs
match qos-group 4
Presentation_ID
policy-map type network-qos

<name>
Cisco Confidential
Setting Scheduling type queueing

Bandwidth Allocation
Class-map type queuing just
matches the qos-group (you
cannot match anything else
class-map type queuing video
match qos-group 4
class-map type queuing nfs
match qos-group 2
You can set:

Bandwidth allocation
Priority scheduling
Presentation_ID
Cisco Confidential
policy-map type queuing <name>
class type queuing video
bandwidth percent 40
class type queuing nfs
bandwidth percent 10
priority
Policy Attach Point

System qos configuration context
Apply service policy to whole system, i.e., all

interfaces
All three types of policy can be applied under
system qos
Ingress Interface
Policy-type qos for classification rules

Policy-type queuing for strict priority and DWRR.
Input queuing policy defines egress queuing policy
for device connected to Nexus 5000, such as CNA
or FEX
Egress Interface
Output queuing policy for strict priority and DWRR

Cisco Confidential
295
Set Jumbo MTU

Nexus 5000 supports different MTU for each system class
MTU is defined in network-qos policy-map
No interface level MTU support on Nexus 5000
Following example configures jumbo MTU for all interfaces
N5k(config)# policy-map type network-qos policy-MTU

N5k(config-pmap-uf)# class type network-qos class-default
N5k(config-pmap-uf-c)# mtu 9216
N5k(config-pmap-uf-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-MTU
N5k(config-sys-qos)#
Cisco Confidential
296
Adjust N5k Ingress Buffer Size

Step 1 Define qos class-map
Step 4 Define network-qos Class-Map
N5k(config)# ip access-list acl-1

N5k(config-acl)# permit ip 100.1.1.0/24 any
N5k(config-acl)# exit
N5k(config)# class-map type qos class-1
N5k(config-cmap-qos)# match access-group name acl-1
N5k(config-cmap-qos)# class-map type qos class-2
N5k(config-cmap-qos)#
N5k(config)# class-map type network-qos class-1

N5k(config-cmap-nq)# class-map type network-qos class-2
Step 2 Define qos policy-map

N5k(config)# policy-map type qos policy-qos
N5k(config-pmap-qos)# class type qos class-1
N5k(config-pmap-c-qos)# class type qos class-2
Step 3 Apply qos policy-map under

system qos
N5k(config)# system qos
N5k(config-sys-qos)# service-policy type qos input policy-qos
Step 5 Set ingress buffer size for

class-1 in network-qos policy-map
N5k(config)# policy-map type network-qos policy-nq
N5k(config-pmap-nq)# class type network-qos class-1
N5k(config-pmap-nq-c) queue-limit 81920 bytes
N5k(config-pmap-nq-c)# class type network-qos class-2
Step 6 Apply network-qos policy-map

under system qos context
N5k(config-pmap-nq-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos
policy-nq
Step 7 Configure bandwidth allocation

using queuing policy-map
Cisco Confidential
297
Configure no-drop system class

N5k(config)# class-map type qos class-nodrop
N5k(config-cmap-qos)# match cos 4

N5k(config-pmap-qos)# class type qos class-nodrop

system qos

Step 5 Configure class-nodrop as nodrop class in network-qos policymap

N5k(config-pmap-nq)# class type network-qos class-nodrop
N5k(config-pmap-nq-c) pause no-drop

policy-nq

using queuing policy-map
Cisco Confidential
298
Configure CoS Marking


N5k(config-acl)# exit
N5k(config)# class-map type qos class-1


N5k(config-pmap-qos)# class type qos class-1

system qos
Step 5 Enable CoS marking for class-1

in network-qos policy-map
N5k(config-pmap-nq)# class type network-qos class-1
N5k(config-pmap-nq-c) set cos 4

policy-nq

for new system class using queuing
policy-map
Cisco Confidential
299
DSCP/IP Precedence Marking on 5548

On the N5548 dscp or ip precedence marking can be
configured in type qos input policy (attached at
system qos or interface)
Switch-6(config-cmap-qos)# policy-map type qos cos1dscp-IF
Switch-6(config-pmap-qos)# class type qos class-1
Switch-6(config-pmap-c-qos)# set dscp ef
Switch-6(config-pmap-c-qos)# set qos-group 2
Switch-6(config-cmap-qos)# policy-map type qos cos1precedence
Switch-6(config-pmap-qos)# class type qos class-1
Switch-6(config-pmap-c-qos)# set precedence 2
Switch-6(config-pmap-c-qos)# set qos-group 2
Cisco Confidential
300
Revert QoS policy to default configuration
Display service policy under system qos

context
N5k#
sh run | begin "system qos"
system qos
service-policy type qos input policy-qos
service-policy type network-qos policy-nq
service-policy type queuing output policy-BW
Display default policy-map name with show policy-map
Name of the default policy-map starts with default

Default qos policy-map: default-in-policy
Default network-qos policy-map: default-nq-policy
Default egress queuing policy-map: default-in-policy
Revert QoS service policy to default

policy by applying default policy-map
under system qos
no service-policy command doesnt
exist under system qos
Interface level service policy can be
removed with no service-policy
command

N5k(config-sys-qos)# service-policy type qos input default-inpolicy
N5k(config-sys-qos)# service-policy type network-qos default-nqpolicy
N5k(config-sys-qos)# service-policy type queuing output defaultout-policy
N5k(config-sys-qos)#interface e1/1
N5k(config-if)# no service-policy type qos input policy-qos
Cisco Confidential
301
Nexus 5500 QoS
Mapping the Switch Architecture to show queuing

dc11-5548-4# sh queuing int eth 1/39
SFP SFP SFP SFP
Interface Ethernet1/39 TX Queuing

qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
Egress (Tx) Queuing

Configuration
UPC
Interface Ethernet1/39 RX Queuing

qos-group 0
q-size: 243200, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port
: 85257
Ucast pkts sent to the cross-bar
: 930
Unified
Mcast pkts sent to the cross-bar
: 84327
Crossbar
Ucast pkts received from the cross-bar : 249
Fabric
Pkts sent to the port
: 133878
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Inactive), Tx (Inactive)
<snip other classes repeated>
Total Multicast crossbar statistics:

Mcast pkts received from the cross-bar
: 283558
Packets Arriving on this port

but dropped from ingress
queue due to congestion on
egress port
Cisco Confidential
302
Troubleshooting
Cisco Confidential
303
SPAN
Cisco Confidential
304
Nexus 5500 SPAN Features

4 active SPAN sessions
Protects data traffic when experiencing congestion
with SPAN
ACL based SPAN to monitor selected flows (Future)
For ingress SPAN, replicate packets before the packets
are rewritten. For egress SPAN replicate packets after
packets are rewritten
Support ERSPAN. Accurately timestamp packets by
including IEEE 1588 timestamp in ERSPAN header
Option to truncate SPAN packets to reduce
bandwidth (Future)
Support FEX ports as SPAN destination port (Future)
Cisco Confidential
305
Ingress SPAN Packet Flow

Data is replicated at ingress port ASIC-Unified Port
Controller(UPC)
SPAN packets is queued at the SPAN destination port VOQ
Each port has 12Gbps connection to switch fabric. Data packets and
SPAN packets share the 12Gbps fabric connection at SPAN source.
Egress Interface
Ingress interface (rx SPAN source)

data
Packet
Buffer
data
12Gbps
span
Multicast VOQ
span
Unified Fabric Controller
Unicast VOQ
12Gbps
SPAN
Destination
12Gbps
Cisco Confidential
306
Egress SPAN Packet Flow

SPAN copy is made at egress pipe of the TX SPAN source port.
SPAN packets are looped back to ingress pipe of UPC and
sent to switch fabric
SPAN and data share the 12Gbps fabric link
Egress Interface
(tx SPAN source)
Ingress Interface
Packet
Buffer
12Gbps
data
Multicast VOQ
Unicast VOQ
data
12Gbps
data
span
Unicast VOQ
12Gbps
span
SPAN
Destination
Cisco Confidential
307
Protecting Data Traffic

RX SPAN
Ingress interface measures the

fabric link(connection between
10GE port and switch fabric)
utilization at SPAN source port
SPAN policing kicks in when

incoming data traffic rate is close to
6Gbps for RX SPAN source. For
small frame size, policing kicks in at
5Gbps due to internal header
SPAN policing regulates the allowed

bandwidth for SPAN traffic.
Production data traffic always get
fabric bandwidth
SPAN and data traffic are stored in

separate packet buffer pools.
SPAN traffic wont affect data

traffic when SPAN destination port
is congested
Ingress Interface
(rx SPAN source)
Packet Buffer
data
Traffic meter
12Gbps
span
SPAN Policing
Cisco Confidential
308
Protecting Data Traffic

TX SPAN
TX SPAN source interface measures the

received traffic rate
SPAN policing is enabled ONLY when RX

traffic rate is higher than 6Gbps for TX
SPAN source port. For small frame
policing kicks in with 5Gbps RX traffic
Separate buffer pool for SPAN and data
Egress Interface
(tx SPAN source)
SPAN Policing
span
12Gbps
Ingress Interface
TX data
TX data
12Gbps
RX data
RX data
Traffic meter
span
12Gbps
SPAN
Destination
Cisco Confidential
309
Expected SPAN Performance for Each SPAN

Source
12
10
8
Received traffic rate
6
Data throughput
SPAN throughput per source
0
1
10
This charts assume the SPAN policing kicks in at 5.5Gbps traffic and policing
rate for SPAN traffic is set to 0.75Gbps per SPAN source interface.
Cisco Confidential
310
SPAN Performance
Scenario 1: No oversubscription
5Gbps
eth1/1
Monitor session 1
source interface eth1/1 rx
destination interface eth1/12
5Gbps
Eth1/2
Unified Port Controller

eth1/5

Eth1/10
5Gbps
Eth1/11
5Gbps
Eth1/12
10Gbps
Two rx SPAN source interfaces each

carries 5Gbps traffic
Total traffic need to be monitored is
10Gbps
No congestion point. All data and SPAN
traffic are received at egress
Sniffer
Cisco Confidential
311
SPAN Performance
Scenario 2: SPAN Destination Oversubscription
4Gbps
eth1/1
4Gbps
Eth1/2
Monitor session 1
4Gbps
Eth1/3

eth1/5

Eth1/10
4Gbps
Eth1/11
Eth1/12
8Gbps 10Gbps
Sniffer
Three SPAN source interfaces each carries

4Gbps traffic
Total SPAN traffic exceed SPAN
destination port speed. 10Gbps SPAN
traffic delivered over SPAN destination port
SPAN destination port back pressure to
SPAN source.
SPAN traffic dropped at SPAN source port
when buffer threshold is reached
Data traffic is not affected due to the fact
that SPAN traffic uses separate buffer
Cisco Confidential
312
SPAN Performance
Scenario 3-Fabric Link Oversubscription
8Gbps
eth1/1
Monitor session 1
8Gbps
Eth1/2

eth1/5

Eth1/10
8Gbps
Eth1/11
Eth1/12
8Gbps 1.5Gbps
Sniffer
SPAN source interface carries 8Gbps

Fabric link between SPAN source port and
switch fabric is congestion point
SPAN policing kicks in and rate limits the
SPAN traffic
Data traffic is not affected. SPAN
throughput for each SPAN source will be
the pre-configured poling rate( Assume
policing is configured as 0.75Gbps in this
example)
Cisco Confidential
313
SPAN Configuration
A SPAN destination port needs to be configured as a switchport monitor port for
the session to become active.
Configure the Destination SPAN Port:
n5000(config)# interface ethernet 2/14
n5000(config-if)# switchport
n5000(config-if)# switchport monitor
Configure destination monitor port
Configure the Monitor (SPAN) Session:

n5000(config)# monitor
n5000(config-monitor)#
session 1
description Inbound(rx) SPAN on Eth 2/13
source interface ethernet 2/13 rx
destination interface ethernet 2/14
no shut
Sessions must be activated
Monitor (SPAN) Options:

n5000(config-monitor)# ?
description Session description (max 32 characters)
destination Destination configuration
exit
Exit from command interpreter
filter
Filter configuration
no
Negate a command or set its defaults
shut
Shut a monitor session
source
Source configuration
Presentation_ID
Cisco Confidential
VLAN Filter for 802.1q tagged trunks
Port = ethernet, port-channel, or sup-eth

Traffic = rx, tx, or both
314
SPAN Verification
Verifying the Destination Port Type:
n5500# show interface ethernet 2/14
Ethernet2/14 is up
Hardware is 10/100/1000 Ethernet, address is 001b.54c0.fedd (bia 001b.54c0.fedd)
Encapsulation ARPA
Port mode is access
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Switchport mode
Switchport monitor is on
Last clearing of "show interface" counters never
Verifying the SPAN Session:

n5500# show monitor session 1
session 1
--------------description
: Inbound(rx) SPAN on Eth 2/13
type
: local
state
: up
source intf
:
rx
: Eth2/13
tx
:
both
:
Source Interface
source VLANs
:
rx
:
tx
:
both
:
filter VLANs
: filter not specified
destination ports
: Eth2/14
Presentation_ID
Cisco Confidential
Operational monitor session = up

Other options:
= down (Session admin shut)
= down (No hardware resource)
= rx
Destination interface
315
Ethanalyzer
Presentation_ID
Cisco Confidential
316
Ethanalyzer (Control Plane Traffic)

Ethanalyzer is an internal CLI based protocol analyzer that captures packets on
the CPU control plane (ingress or egress). Ethanalyzer is useful when
troubleshooting CPU and/or control plane related issues.
The packets can be viewed using the CLI or exported to a Wireshark protocol
analyzer on an external host for GUI analysis.
Ethanalyzer Guidelines:
Configured in user-exec mode
Three interface options can be specified - inbound-hi, inbound-low, mgmt

10 packet capture limit by default Configurable up to 2.1 billion packets
Packet contents scroll on the console by default
Packet capture can be redirected to a destination file - Recommended

Brief or Detailed analysis available (Brief is enabled by default)
User configurable Frame-Size, with Capture and Display Filter options
Presentation_ID
Cisco Confidential
317

Control Plane Elements
Monitoring of in-band traffic via NX-OS
built-in ethanalyzer (sniffer)
Eth3 is equivalent to inbound-lo
Eth4 is equivalent to inbound-hi
N5k-2# ethanalyzer local interface ?
inbound-hi
Inbound(high priority) interface
inbound-low Inbound(low priority) interface
mgmt
CLI view of in-band control plane data

DCN-N5K1# show hardware internal cpu-mac inband counters
eth3
Link encap:Ethernet HWaddr 00:0D:EC:B2:2A:C3
RX bytes:682915556 (651.2 MiB) TX bytes:5638322004 (5.2 GiB)
Base address:0x6020 Memory:fa4a0000-fa4c0000
eth4
Presentation_ID
CPU
Intel LV Xeon
1.66 GHz
South
Bridge
NIC
eth3
eth4
Unified Port
Controller
Link encap:Ethernet HWaddr 00:0D:EC:B2:2A:C4

RX bytes:24429668210 (22.7 GiB) TX bytes:4141361337 (3.8 GiB)
Base address:0x6000 Memory:fa440000-fa460000
Cisco Confidential
318
Ethanalyzer Configuration
Create a Capture:
n5500# ethanalyzer local interface
inbound-hi
inbound-hi/Outband interface
mgmt
Capture using Defaults and Write to a File on Bootflash:

n5500# ethanalyzer local interface inbound-hi write bootflash:ethanalyzer-data
Capturing on inbound-hi
10
Real-Time counter
Additional Capture Options:

n5500# ethanalyzer local
<CR>
>
>>
capture-filter
decode-internal
detailed-dissection
display-filter
dump-pkt
limit-captured-frames
limit-frame-size
write
interface inbound-hi ?
Redirect it to a file in append mode
Filter on ethanalyzer capture
Include internal system header decoding
Display detailed protocol information
Display filter on frames captured
Hex/Ascii dump the packet with possibly one line summary
Maximum number of frames to be captured (default is 10)
Capture only a subset of a frame
Filename to save capture to
Applies a capture-filter to limit data
Writes to a file instead of the console
Limit Captured Frame Size:

n5500# ethanalyzer local interface inbound-hi limit-frame-size ?
<64-65536>
Size in bytes
Presentation_ID
Cisco Confidential
Slice packets for headers only

319
Ethanalyzer Capture-Filter Configuration

Capture filters can be used to reduce the amount of data collected when
troubleshooting. The following CLI illustrates some basic examples.
The capture filter syntax is the same as tcpdump (also same as Wireshark).
n5500# ethanalyzer local interface inbound-hi capture-filter "icmp"

n5500# ethanalyzer local interface inbound-hi capture-filter "tcp"
n5500# ethanalyzer local interface inbound-hi capture-filter "udp"
n5500# ethanalyzer local interface inbound-hi capture-filter "ip proto ospf"
n5500# ethanalyzer local interface inbound-hi capture-filter "ip proto eigrp"
n5500# ethanalyzer local interface inbound-hi capture-filter "src net 192.168.204.2"
n5500# ethanalyzer local interface inbound-hi capture-filter "dst net 224.0.0.2"
n5500#
n5500#
n5500#
n5500#
ethanalyzer
ethanalyzer
ethanalyzer
ethanalyzer
local
local
local
local
interface
interface
interface
interface
inbound-hi
inbound-hi
inbound-hi
inbound-hi
capture-filter
capture-filter
capture-filter
capture-filter
"tcp
"tcp
"udp
"udp
dst
src
dst
src
port
port
port
port
23"
23"
23"
23"
n5500# ethanalyzer local interface inbound-hi capture-filter "src net 10.20.0.190 and tcp dst port 23"
n5500# ethanalyzer local interface inbound-hi capture-filter "dst net 224.0.0.2 and udp dst port 1985"
Presentation_ID
Cisco Confidential
320
Ethanalyzer Brief Output (Console)

The Ethanalyzer output defaults to brief mode for collecting an initial snapshot of
packets on the CPU control plane. If more information is needed, perform a
detailed capture and specify a capture-filter for a more specific match.
Packets will scroll on the screen to the specified capture limit. (Default is 10)
n5500# ethanalyzer local interface inbound-hi

2008-06-02 20:44:40.327808
2008-06-02 20:44:41.480658
2008-06-02 20:44:41.730633
2008-06-02 20:44:41.730638
2008-06-02 20:44:42.480586
2008-06-02 20:44:43.480513
2008-06-02 20:44:45.480499
2008-06-02 20:44:45.480506
2008-06-02 20:44:46.308177
2008-06-02 20:44:46.974771
192.168.20.1
192.168.20.2
192.168.20.2
192.168.20.2
192.168.20.2
192.168.20.2
192.168.20.2
192.168.20.2
192.168.10.1
192.168.10.2
->
->
->
->
->
->
->
->
->
->
224.0.0.5
OSPF Hello Packet
207.68.169.104 DNS Standard query
65.54.238.85 DNS Standard query A
224.0.0.5
OSPF Hello Packet
224.0.0.5
OSPF Hello Packet
A print.cisco.com
A print.cisco.com
print.cisco.com
print.cisco.com
A print.cisco.com
A print.cisco.com
print.cisco.com
The output can also be copied to a local flash (i.e. bootflash, logflash, usb1, usb2)
Presentation_ID
Cisco Confidential
321
Ethanalyzer Detailed Output (Console)

Use the detail option to capture detailed packet information.
Packets will scroll on the screen to the specified capture limit. (The default is 10)
n5500# ethanalyzer local interface inbound-hi detail
Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Nov 2, 2009 22:07:57.150394000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 60 bytes
Capture Length: 60 bytes
[Frame is marked: False]
[Protocols in frame: eth:llc:stp]
IEEE 802.3 Ethernet
Destination: 01:80:c2:00:00:00 (01:80:c2:00:00:00)
Address: 01:80:c2:00:00:00 (01:80:c2:00:00:00)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: 00:0d:ec:6d:96:6f (00:0d:ec:6d:96:6f)
Address: 00:0d:ec:6d:96:6f (00:0d:ec:6d:96:6f)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Length: 39
Trailer: 00000000000000
<Text Omitted>
Presentation_ID
Cisco Confidential
322
Reading Ethanalyzer Output Locally

You dont need to specify an output option when writing a capture to a local
destination. Use the detail option if you want to see the packet details.
Brief:
n5500# ethanalyzer local read bootflash:ethanalyzer-data
00:0d:ec:6d:96:6f -> 01:00:0c:cc:cc:cc CDP
00:1b:54:c1:0a:69 -> 01:00:0c:cc:cc:cd STP
00:1b:54:c1:0a:69 -> 01:80:c2:00:00:00 STP
00:1b:54:c1:0a:69 -> 01:00:0c:cc:cc:cd STP
192.168.1.2 -> 224.0.0.10
EIGRP Hello
Device ID: MSDC-N5K-01(FLC12100023) Port ID:

RST. Root = 32788/00:18:ba:d8:58:25 Cost = 2
RST. Root = 32769/00:18:ba:d8:58:25 Cost = 2
RST. Root = 32769/00:18:ba:d8:58:25 Cost = 2
Ethernet1/40
Port = 0x9009
Port = 0x9009
Port = 0x9009
Note: Timestamps Omitted
Reading detailed output from local bootflash:
Detailed:
n5500# ethanalyzer local read bootflash:ethanalyzer-data detail

Frame 1 (268 bytes on wire, 268 bytes captured)
Arrival Time: Nov 2, 2009 21:50:18.794493000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 268 bytes
Capture Length: 268 bytes
[Frame is marked: False]
[Protocols in frame: eth:llc:cdp:data]
IEEE 802.3 Ethernet
Destination: 01:00:0c:cc:cc:cc (01:00:0c:cc:cc:cc)
Address: 01:00:0c:cc:cc:cc (01:00:0c:cc:cc:cc)
Presentation_ID
Cisco Confidential
323
Core Files & Logging
Presentation_ID
Cisco Confidential
324
System Crash
Determine the reset reason and how long since last
reset:
DCN-N5K1# show system reset-reason
----- reset reason for Supervisor-module 1 (from Supervisor
in slot 1) --1) At 574259 usecs after Thu Jul 21 18:59:24 2011
Reason: Reset Requested by CLI command reload
Service:
Version: 5.0(3)N1(1b)
2) At 605182 usecs after Tue Apr 19 20:53:24 2011
Reason: Disruptive upgrade
Service:
Version: 4.2(1)N2(1a)
Reason: Reset by installer
Service:
Version: 4.1(3)N2(1)
Reason: Reset Requested by CLI command reload
Service:
Version: 4.1(3)N2(1)
DCN-N5K1# show system uptime

System start time:
Thu Jul 21 19:04:28 2011
System uptime:
34 days, 6 hours, 41 minutes, 30 seconds
Kernel uptime:
Active supervisor uptime: 34 days, 6 hours, 41 minutes, 30 seconds
Cisco Confidential
325
Process Crash
Investigate syslog file for errors:
switch# show log logfile | include error
Run the show processes command. State of ER

indicates process should be running but is not.
Check the process log for a stack trace or core dump:
DCN-N5K1# show process log
Process
--------------installer
installer
installer
PID Normal-exit Stack Core Log-create-time

------ ----------- ----- ----- --------------24484
N
N
N Wed Jun 23 16:26:47 2010
24493
N
N
N Wed Jun 23 16:27:18 2010
24508
N
N
N Wed Jun 23 16:28:14 2010
Cisco Confidential
326
Core Files & Logging

Show cores:
switch# show cores
Module-num
Process-name
PID
Core-create-time
----------
------------
---
----------------
fwm
2834
Aug 13 16:3
Copy to a remote server:

switch# copy core:?
core:
Enter URL "core://<module-number>/<process-id>"
switch# copy core://1/2834 ftp://128.107.65.217/ vrf management
Enter username: anonymous

Password:
***** Transfer of file Completed Successfully *****
OBFL Logging:
N5K-S003-LAB# sh logg onboard exception-log
---------------------------OBFL Data for
Module:
----------------------------
N5K-S003-LAB# sh logg last 20

Cisco Confidential
327
Grab a show tech-support

Or not
Sometimes too general
Large file, time consuming

If time permits, use targeted outputs or a specific
show tech
If there is no time, use tac-pac and copy off

Much quicker than transmitting to terminal
Zips entire output to file in volatile:
Copy file off of switch for analysis

N5k-1# tac-pac
N5k-1# dir volatile:
180242
Jan 28 4:37:26 2011
show_tech_out.gz
Cisco Confidential
328
Which show tech?

As of 5.0(3), There Are 68
N5k-1# show tech-support ?
aaa
Display aaa information
aclmgr
ACL commands
adjmgr
Display Adjmgr information
arp
Display ARP information
ascii-cfg
Show ascii-cfg information for technical support personnel
assoc_mgr
Gather detailed information for assoc_mgr troubleshooting
bcm-usd
Gather detailed information for BCM USD troubleshooting
bootvar
Gather detailed information for bootvar troubleshooting
brief
Display the switch summary
btcm
Gather detailed information for BTCM component
callhome
Callhome troubleshooting information
cdp
Gather information for CDP trouble shooting
...
session-mgr
Gather information for troubleshooting session manager
snmp
Gather info related to snmp
sockets
Display sockets status and configuration
spm
Service Policy Manager
stp
Gather detailed information for STP troubleshooting
sysmgr
Gather detailed information for sysmgr troubleshooting
time-optimized Gather tech-support faster, requires more memory & disk space
track
Show track tech-support information
vdc
Gather detailed information for VDC troubleshooting
vpc
Gather detailed information for VPC troubleshooting
vtp
Gather detailed information for vtp troubleshooting
xml
Gather information for xml trouble shooting
Cisco Confidential
329
Logging
Often Overlooked, but very Important
show logging logfile
Basis for tracing events chronologically
Try using start-time or last
N5k-1# show logging logfile start-time 2011 Mar 9 20:00:00
2011 Mar 9 20:17:18 esc-n5548-1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/1 is down (None)
2011 Mar 9 20:17:18 esc-n5548-1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/3 is down (None)
N5k-1# show logging last ?
<1-9999> Enter number of lines to display
show accounting log

Basis for tracing configuration changes
terminal log-all to also log show commands
All commands end with (SUCCESS) or (FAILURE)

Cisco Confidential
330
Other System Logs

show logging nvram
Survives reloads helpful for crash or reload issues
show process log details

Process failure or exit reason
Onboard Failure Logging

show logging onboard
obfl-logs
obfl-history
exception log
kernel-trace
show system reset-reason
Cisco Confidential
331
Hardware Issues
Cisco Confidential
332
POST and OHMS (Online Health Monitoring System)

Types Of Errors
Types of Reaction
Failures causing NXOS

not be able to come
up properly
Console continuous print error messages every 30 seconds.

System LED sets to Flashing Amber. Example of such failure:
DRAM, backplane SPROM checksum failure, PCIe enumeration
failure
Failures not fatal and

NXOS can boot up
System comes all the way up. Syslog, OBFL and callhome
initiated to indicate failure. Example of such failure: OBFL flash,
CTS keystore.
Failure causing port

failures
System comes all the way up. Syslog, OBFL and callhome
initiated to indicate failure. Example of such failure: ASIC ECC
error found during POST or OHMS
N5K-C5548P-L11-01# sh platform nohms errors

1) Event:E_DEBUG, length:79, at 806296 usecs after Sun Apr 18
09:57:02 2010
[102] nohms_process_lc_online(350): FEX-100 On-line (Serial
Number JAF1307BHCD)
2) Event:E_DEBUG, length:57, at 498025 usecs after Sun Apr 18

09:57:00 2010
[102] nohms_handle_lc_inserted(191): n_errs 0 n_notices 0
Cisco Confidential
333
NOHM (Online Health Monitoring) logging

switch# show logging |grep NOHMS
2008 Apr 18 23:00:01 switch %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime
diag detected major event: Port failure: Ethernet1/1
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1
temperature sensor 1 failed.
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: Module 1
temperature sensor 2 failed.
2008 Apr 19 01:45:25 swor35p %NOHMS-2-NOHMS_ENV_ERROR: System major
temperature alarm on Module 1. Sensor 9 Temperature 42 Major
Threshold 0
Cisco Confidential
334
Environmental Monitoring
switch# show environment
Displays following status:
Fan
Temperature
Power Supply
Power Usage Summary
Cisco Confidential
335
Diagnostic Result
switch# show diagnostic result module 1
Current bootup diagnostic level: complete
Module 1: 40x10GE/Supervisor
15) TestFabricPort :
SerialNo : JAB1208005T
Eth
9 10 11 12 13 14 15 16 17 18 19 20
Port ------------------------------------------------------------
Overall Diagnostic Result for Module 1 : PASS
Diagnostic level at card bootup: complete

Eth
Test results: (. = Pass, F = Fail, I = Incomplete,
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
Port ------------------------------------------------------------
U = Untested, A = Abort)
9 10 11 12 13 14 15 16 17 18 19 20
1) TestUSBFlash ------------------------> .
2) TestSPROM ---------------------------> .
16) TestForwardingEngine :
3) TestPCIe ----------------------------> .
4) TestLED -----------------------------> .
Eth
5) TestOBFL ----------------------------> .
Port ------------------------------------------------------------
6) TestNVRAM ---------------------------> .
7) TestPowerSupply ---------------------> F
8) TestTemperatureSensor ---------------> .
Eth
9) TestFan -----------------------------> .
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
Port ------------------------------------------------------------
10) TestVoltage -------------------------> .
11) TestGPIO ----------------------------> .
12) TestSupervisorPort ------------------> .

13) TestMemory --------------------------> .
17) TestForwardingEnginePort :
14) TestFabricEngine :
Eth
9 10 11 12 13 14 15 16 17 18 19 20
Port -----------------------------------------------------------.
Eth
Port -----------------------------------------------------------.
9 10 11 12 13 14 15 16 17 18 19 20
Port -----------------------------------------------------------.
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
.
Eth
Eth
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
Port -----------------------------------------------------------.
Cisco Confidential
336
Show tech
Capture to terminal emulator buffer or log file:
switch# terminal length 0
switch# show tech-support details
`show switchname`
switch
`show system uptime`
System start time:
Mon Aug 11 15:33:17 2008
System uptime:
.
.
.
Or
Capture
to file in volatile:
switch# tac-pac
switch# dir volatile:

66860
Aug 13 16:23:03 2008 show_tech_out.gz
switch# copy volatile:show_tech_out.gz sftp://[email protected]/ vrf management
Cisco Confidential
337
Port Issues
Cisco Confidential
338
Ethernet Interface Counters

switch# show interface eth1/21
Ethernet1/21 is up
Hardware is 10000 Ethernet, address is 000d.ec6d.84dc
(bia 000d.ec6d.84dc)
Encapsulation ARPA
Port mode is access
Input flow-control is off, output flow-control is off
5 minute input rate 22203 bytes/sec, 346 packets/sec
5 minute output rate 640597 bytes/sec, 10000 packets/sec
Rx
16501327 Input Packets 9 Unicast Packets 16500923
Multicast Packets
395 Broadcast Packets 0 Jumbo Packets 0 Storm
Suppression Packets
1056159080 Bytes
0 No buffer 0 runt 0 crc 0 ecc
0 Overrun
0 Underrun 0 Ignored 0 Bad etype drop
0 Bad proto drop 0 If down drop 0 Collision

0 Late collision 0 Lost carrier 0 No carrier
0 Babble
Tx
433943286 Output Packets 26171 Multicast Packets
0 Broadcast Packets 0 Jumbo Packets
27772499094 Bytes
0 Ouput errors
16499333 Rx pause 0 Tx pause 0 Reset
Cisco Confidential
339
Ethernet Interface Counters

switch# sh interface ethernet 1/17 counters detailed all
64 bit counters:
0.
rxHCTotalPkts = 475168
1.
txHCTotalPks = 3445907
2.
rxHCUnicastPkts = 1390
3.
txHCUnicastPkts = 2053
4.
rxHCMulticastPkts = 191780
5.
txHCMulticastPkts = 473324
6.
rxHCBroadcastPkts = 281998
7.
txHCBroadcastPkts = 2970530
14.
rxTxHCpkts512to1023Octets = 195759
15.
16.
All Port Counters:

0.
InPackets = 475168
27.
ShortFrames = 0
28.
Collisions = 0
29.
SingleCol = 0
30.
MultiCol = 0
31.
LateCol = 0
32.
ExcessiveCol = 0
33.
LostCarrier = 0
34.
NoCarrier = 0
35.
Runts = 0
36.
Giants = 0
Cisco Confidential
340
Interface Error Counters

N5K# show interface E1/13 counters errors
-------------------------------------------------------------------------------Port
Align-Err
FCS-Err
Xmit-Err
Rcv-Err
UnderSize OutDiscards
-------------------------------------------------------------------------------Eth1/13
0
0
0
0
0
0
-------------------------------------------------------------------------------Port
Single-Col
Multi-Col
Late-Col
Exces-Col
Carri-Sen
Runts
-------------------------------------------------------------------------------Eth1/13
0
0
0
0
0
0
-------------------------------------------------------------------------------Port
Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
-------------------------------------------------------------------------------Eth1/13
0
-0
0
0
0
N5K# show interface e1/13 flowcontrol

-------------------------------------------------------------------------------Port
Send FlowControl Receive FlowControl RxPause TxPause
admin
oper
admin
oper
-------------------------------------------------------------------------------Eth1/13
off
off
off
off
0
0
N5K# show interface e1/13 priority-flow-control
============================================================
Port
Mode Oper(VL bmap) RxPPP
TxPPP
============================================================
Ethernet1/13
Auto Off
Cisco Confidential
341
QoS Counters
d14-switch-1# show policy-map interface ethernet 3/1
Ethernet3/1
Service-policy system: global
class-map: class-fcoe
Statistics:
: 0
: 0
Ucast pkts received from the cross-bar
: 0
: 0
: 0
class-map: class-default
Statistics:
: 761951066
: 429740044
Ucast pkts received from the cross-bar
: 3127717414
: 3308485758
: 9038
Multicast crossbar statistics:

: 140042101
: 357560270
Cisco Confidential
342
QoS Counters
DCN-N5K1(config-if)# show queuing interface e1/1
Ethernet1/1 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
RX Queuing
qos-group 0
drop-type: drop, xon: 0, xoff: 1520
Statistics:
: 6330629
: 5580600
: 750029
: 10598898
:0
qos-group 1
drop-type: no-drop, xon: 128, xoff: 240
Statistics:
:0
:0
:0
:1
:0
Total Multicast crossbar statistics:
: 2905930
Cisco Confidential
343
Monitoring PAUSE frame counters

switch# show int ethernet 1/5 priority-flow-control
------------------------------------------------------------------------------Port
Mode Oper RxPPP
TxPPP
-------------------------------------------------------------------------------
Eth1/5
auto on
2967222
switch# show interface ethernet 1/6 flowcontrol

------------------------------------------------------------------------------Port
Send FlowControl
Receive FlowControl
admin
admin
oper
RxPause TxPause
oper
------------------------------------------------------------------------------Eth1/5
off
off
off
off
3127212 0
Cisco Confidential
344
Interface Transceiver Details

N5K# show interface e1/13 transceiver details
Ethernet1/13
sfp is present
name is CISCO-AVAGO
part number is SFBR-7700SDZ
revision is B4
serial number is AGD121321JF
nominal bitrate is 10300 MBits/sec
Link length supported for 50/125um fiber is 82 m(s)
Link length supported for 62.5/125um fiber is 26 m(s)
cisco id is -cisco extended id number is 4
SFP Detail Diagnostics Information (internal calibration)
---------------------------------------------------------------------------Alarms
Warnings
High
Low
High
Low
---------------------------------------------------------------------------Temperature
35.87 C
75.00 C
-5.00 C
70.00 C
0.00 C
Voltage
3.26 V
3.59 V
3.00 V
3.46 V
3.13 V
Current
6.43 mA
10.50 mA
2.50 mA
10.50 mA
2.50 mA
Tx Power
-2.46 dBm
1.49 dBm -11.30 dBm
-1.50 dBm
-7.30 dBm
Rx Power
-2.63 dBm
1.99 dBm -13.97 dBm
-1.00 dBm
-9.91 dBm
---------------------------------------------------------------------------Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Cisco Confidential
345
Troubleshooting sfpInvalid Status

DCN-N5K1(config-if)# show int e1/1
Ethernet1/1 is down (SFP validation failed)
switch# show logging | grep 1/7
2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for
interface Ethernet1/7 is not supported
2005 Jul 1 16:07:41 switch %ETHPORT-3-IF_UNSUPPORTED_TRANSCEIVER_VENDOR:
Transceiver vendor for interface Ethernet1/7 is not supported
switch#
switch# show system internal ethpm event-history errors | grep 1/7
0x0
[102] Ifindex (Ethernet1/7)0x2006000, SFP security check: CRC failed, rcvd CRC
calculated CRC 0xe9777080
Most Common Reason for sfpInvalid:

speed 1000 missing from a 1Gig SFP
Cisco Confidential
346
Error Disabled Interface

switch# show interface e1/14
e1/7 is down (errDisabled)
View internal state transition info:

switch# show system internal ethpm event-history interface e1/7
>>>>FSM: <e1/7> has 86 logged transitions<<<<<
1) FSM:<e1/7> Transition at 647054 usecs after Tue Jan 1 22:44..
Previous state: [ETH_PORT_FSM_ST_NOT_INIT]
Triggered event: [ETH_PORT_FSM_EV_MODULE_INIT_DONE]
Next state: [ETH_PORT_FSM_ST_IF_INIT_EVAL]
2) FSM:<e1/7> Transition at 647114 usecs after Tue Jan 1 22:43..
Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
Triggered event: [ETH_PORT_FSM_EV_IE_ERR_DISABLED_CAP_MISMATCH]
Next state: [ETH_PORT_FSM_ST_IF_DOWN_STATE]
Examine the log file for port state transitions:

switch# show logging logfile
...
Jan 4 06:54:04 switch %PORT_CHANNEL-5-CREATED: port-channel 7 created
Jan 4 06:54:24 switch %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface portchannel 7 is down (No operational members)
Jan 4 06:54:40 switch %PORT_CHANNEL-5-PORT_ADDED: e1/8 added to port-channel 7
Jan 4 06:54:56 switch %PORT-5-IF_DOWN_ADMIN_DOWN: Interface e1/7 is down
(Admnistratively down)
Jan 4 06:54:59 switch %PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: speed is not compatible
Jan 4 06:55:56 switch%PORT_CHANNEL-5-PORT_ADDED: e1/7 added to port-channel 7
Cisco Confidential
347
Port Channel Link Selection

N5K# splf interface port-channel 200 dst-mac ffff.ffff.ffff
Load-balance Algorithm: source-ip
crc8_hash: 0
Outgoing port id: Ethernet1/33
N5K# splf int port-ch 200 src-mac 0050.5646.3e72 dst-mac ffff.ffff.ffff
N5K# splf interface port-channel 200 src-mac 0050.5646.3e72 dst-mac 0050.5646.582b
N5K# show port-channel load-balance forwarding-path interface po200 src-ip 14.17.104.32
crc8_hash: 19
Outgoing port id: Ethernet1/37
N5K# show platform fwm info pc port-channel 200 | grep hash
Po200: hash params - l2_da 0 l2_sa 1 l3_da 0 l3_sa 1
Po200: hash params - l4_da 0 l4_sa 0 xor_sa_da 1 hash_elect 1
N5K# show port-channel load-balance
System: source-ip
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-mac
IP: source-ip source-mac
Note: To fit the output onto

the slide splf is used for
show port-channel load
forwarding-path
Cisco Confidential
348
LACP Not Coming Up?

DCN-N5K1# show lacp interface e1/18
Interface Ethernet1/18 is up
Channel group is 20 port channel is Po20
PDUs sent: 94993
PDUs rcvd: 95702
Are PDUs being received? If not, LACP
Markers sent: 0
configured on neighbor?
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Are there any Unknown or Illegal packets
Lag Id: [ [(7f9b, 0-23-4-ee-be-1, 8014, 8000, 204), (7f9b, 0-23received? If so, get a sniffer capture of the
4-ee-be-2, 8014,
packets on the wire and open a TAC case.
8000, 112)] ]
Operational as aggregated link since Wed Jul 27 17:47:49
2011
Cisco Confidential
349
Common LACP Misconfiguration
N5K sends the packets

untagged, whereas the host is
expecting them tagged with
VL100.
N5K can see LACP PDUs

from the host on VLAN 100
Server configured to tag

dot1Q VL100
To remediate, either change the switch port to

a trunk or do not tag at the server
Cisco Confidential
350
Feature Comparisons
Cisco Confidential
351
Nexus 5000 to 5500 Comparison

Features
Nexus 5000
Next Generation N55K
512 (128 per port)
4K (flexible allocation)
480 KB
640 KB
Number of unicas VoQ per ingress port
416
1024 (800 with sunnyvale)
Number of mulicast VoQ per ingress por
128
Number of Egress queues
16 (8 for unicat and 8 for multicast)
COS marking
Ingress
Ingress & Egress
DSCP marking
NO
Ingress & Egress
ECN marking
NO
YES
ACL based buffering and queuing
YES
YES
Station Table (MAC table)
16K
32K
VLAN Table
1K
4K
Number of active VLAN
512
4K
Mulicast index Table
4K
8K
Number of IGMP entries
1K
4K
Numer of ports per ASIC (Gatos / Carmel)

Numer of LIF per ASIC (Gatos / Carmel)
Buffer per port
The items marked in RED will NOT be available in Eagle Hawk release
Cisco Confidential
352
Nexus 5000 to 5500 Comparison (cont)

Features
Nexus 5000
Nexus 5500
Multiple egress SPAN source
NO
YES (up to 4)
Port Channel can be egress SPAN source
NO
YES
VLAN can be egress SPAN source
NO
YES
ERSPAN
YES
YES
ERSPAN v3
NO
YES
FEX port as destination SPAN
NO
YES
3.2 us
2 us
IEEE 1518
No
Yes
Number of Port channel per box
16
48
Number of port in a port channel
16
16
L2/L3/L4 SA/DA
L2/L3/L4 SA/DA , VLAN
Port Channel Load balancing for multicast flow

destination
NO
YES
LID multipathing
NO
YES
Superframing
YES
YES
Flexible output buffer selection between unicast

and multicast
NO
YES
Proxy queue mulicast overload
NO
YES
Latency
Port Channel load balancing
The items marked in RED will NOT be available

in Eagle Hawk releaseCisco Confidential
353
Nexus 5000 to 5500 Comparison (contd)

Features
Nexus 5000
Nexus 5500
TCAM size
2K
4K
FC Forwarding
YES
YES
FCoE Forwarding
YES
YES
FCF lookup table
4K
8K
DCE Forwarding
NO
YES
DCE lookup table
N/A
8K
TRILL Forwarding
NO
YES
TRILL lookup table
N/A
8K
L3 binding table
2K
4K
FC zoning table
2K
4K
RBAC table
2K
2K
Policers
256
512
Dedicated buffer allocated for SPAN
NO
YES
Multiple ingress SPAN source
YES
YES
Number of acive SPAN session
The items marked in RED will NOT be available in Eagle Hawk release
Cisco Confidential
354
Nexus Layer 3 Functional Comparison

7000 vs 5500
L3 Functional Areas
Nexus 7000 / M1 Modules
Nexus 5500 + L3 Module
Routing Protocols
OSPF, EIGRP, RIPv2, BGP, IS-IS, PIM,

IGMP, BiDir PIM
Base Enterprise: Static, OSPF*, EIGRP Stub,

RIPv2, PIM, IGMP
LAN Enterprise: BGP, OSPF, EIGRP
IPv6
Dual Stack, OSPFv3, EIGRP, HSRPv6
For Management
L3 Segmentation
Base: VRF Lite, VRF Aware Features, VRF

Import/Export
MPLS License: MPLS VPNs
Base Enterprise: VRF (Management)

LAN Enterprise: VRF-Lite
High Availability
ISSU, NSF, Graceful Restart, Multicast NSF,

IGP NSR
ISSU Edge L2 Only
Fast Convergence
BFD, Next Hop Tracking, BGP PIC, MPLS-TE
No
Monitoring
Flexible Netflow, Sampled Netflow, MPLS

OAM, ERSPAN
ERSPAN**
L2 over L3
Overlay Transport Virtualization (OTV)
No
Traffic Steering
Policy-Based Routing, VRF Select, WCCPv2,

Static Multicast MAC
No
Tunneling / Mobility
Unicast over GRE, LISP
No
* 256 dynamically Learned routes

** CY11 Roadmap. Not available in existing release
Cisco Confidential
355
Nexus Layer 3 Scale Comparison

7000 vs 5500
L3 Functional Areas
L3 Interfaces
4K
4K
IPv4 Unicast FIB
1M
8K*
IPv4 Multicast FIB
M1/XL: 32K
4K
L3 ECMP
16 Way
16 Way
ARP
50K
8K
Routing Adjacency
128K
8K
FHRP
4K HSRP Groups
1K HSRP Groups
L3 ACLs
128K
Ingress: 2K
Egress: 1K
Segmentation
1K VRFs
1K VRFs
* With Enterprise LAN License
Cisco Confidential
356
Nexus Layer 3 System Comparison

7000 vs 5500
L3 Functional Areas
Redundant Route
Processors
Yes
No
Control Plane Protection
Extensive CoPP Granularity
Single Rate Limiter, Basic CoPP**
Distributed Processing
Yes, Distributed Multicast replication

and BFD
No for L3
FEX Routed Port
Yes
No
FEX Scale L3
32
ISSU
Yes L2 or L3
Edge L2 Only
Stateful Process Restart
Yes OSPF, IS-IS
No
L3 over VPC
No
No
** CY11 Roadmap. Not available in existing release
Cisco Confidential
357
Cisco Confidential
358

TRM N55K L2only-Config Tshoot Jdinkin2 2hr 20120208

Uploaded by

Copyright:

Available Formats

TRM N55K L2only-Config Tshoot Jdinkin2 2hr 20120208

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

TRM N55K L2only-Config Tshoot Jdinkin2 2hr 20120208

Uploaded by

Copyright:

Available Formats

What are the main hardware components of the Nexus 5500 series switches?

What are the main hardware components of the Nexus 5500 series switches?

What expansion module options are available for the Nexus 5500 series switches?

What expansion module options are available for the Nexus 5500 series switches?

Cisco Advanced Services

Cisco Nexus 5500 Series

Instructor: Joel Dinkin ([email protected])

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Series

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Hardware

32 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

48 Fixed Ports 1/10G Ethernet or 1/2/4/8 FC

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Hardware

32 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Out of Band Mgmt

Nexus 5500 Hardware

48 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Out of Band Mgmt

Nexus 5500 Hardware

Contain forwarding ASIC (UPC-2)

2008 Cisco Systems, Inc. All rights reserved.

1G Support on all ports

Requires the use of a standard 1G SFP

Supports for all features at 1G speed other than Unified I/O

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Layer 3 Options

Nexus 5500 Hardware

Note: 5548UP and 5596UP ONLY, not 5548P

2008 Cisco Systems, Inc. All rights reserved.

Reverse Air Flow - CLI

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Internals

Unified Crossbar Fabric

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Hardware Overview

Multimode Media access controllers

Packet buffering and queuing

2008 Cisco Systems, Inc. All rights reserved.

Ethernet (Layer 2 and FabricPath) and

640 KB of buffering per port

Nexus 5500 Hardware Overview

Program Store - 2 GB of eUSB flash for base

On-Board Fault Log (OBFL) - 64 MB of flash to

RS-232 console port: console0

Boot/BIOS Flash - 8 MB to store upgradable

NVRAM - 6 MB of SRAM to store Syslog and

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Hardware Overview

In-band traffic is identified by the

IGMP, CDP, TCP/UDP/IP/ARP (for

Eth4 handles Rx and Tx of high

2008 Cisco Systems, Inc. All rights reserved.

Nexus 5500 Hardware Overview

CPU queuing structure provides strict

Each of the two in-band ports has 8 queues

CLASS 6 is configured for DRR scheduling

Default classes (0 to 5) are configured for DRR

Additionally each of the two in-band