#CLMEL

How To Setup An
ACI Fabric From
Scratch

Ramses Smeyers, Principal Engineer Services

BRKACI-2004

#CLMEL
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-2004

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda

• Prerequisites
• Hardware inspection
• Hardware installation
• Build fabric topology
• Configure the fabric
• Upgrade the fabric

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Prerequisites
• Before starting, you should have:
• At least 6 routable IP addresses for APIC OOB mgmt and APIC CIMC
• Functional NTP server
• Serial number of all leafs and spines
• Optionally but recommended:
• 1 IP per leaf and spine for OOB
• SCP / FTP / HTTP server
• Console / serial server
• Infrastructure VLAN / VTEP pool
• vCentre IP address and credentials

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Hardware inspection / installation

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Configure APIC CIMC

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Our setup for the day

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Physical Layout
N3K
Spine2 SAL1925H0JK
Spine1 SAL1925H0HV
Leaf 2 ACI SAL1951VHXH
Leaf 1 ACI SAL19079J47
UCS R3
UCS R2
UCS R1
APIC
APIC
APIC
UCS Mini

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Consoles
Device bdsol-2901- Device bdsol-2901-51
51

bdsol-aci12-ucs-A 2011 bdsol-aci12-spine1 2015

bdsol-aci12-ucs-B 2012 bdsol-aci12-spine2 2016

bdsol-aci12-leaf1 2013 bdsol-aci12-n3k 2017

bdsol-aci12-leaf12 2014

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
IPs
Device IP

bdsol-aci12-apic1 10.48.22.69

bdsol-aci12-apic2 10.48.22.70

bdsol-aci12-apic3 10.48.22.71

leaf1 10.48.22.77

leaf2 10.48.22.78

spine1 10.48.22.75

spine2 10.48.22.76

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
UCS Mini FI A FI B
Details

• FI A: 10.48.22.80 ESXi-1
vmk 10.48.22.66
• FI B: 10.48.22.81

• Cluster: 10.48.22.82
ESXi-2
vmk 10.48.22.67

1/1 1/1

41 1/2 1/2 42

ACI Leaf 1 ACI Leaf 2

1/11 1/12
42 41
5548

2011 2022

VLAN 22
#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ACI - topology

Spine 1 Spine 2
1/33 10.48.22.75 1/35 1/33 10.48.22.76 1/35

1/49 Leaf 1 1/51 1/49 Leaf 2 1/51

10.48.22.77 10.48.22.78
1/33 1/33 1
9 10 11 9 11
0
po2

1 N3K 2
10.48.22.79
1 2 1 2 1 2
BM01 LNX BM02 LNX ESXi 03
10.48.22.63 10.48.22.64 10.48.22.65

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Build fabric topology
• APIC initial configuration (APIC #1) [only the 1st one for now]
• 1st leaf discovery
• spine discovery
• leaf discovery
• Remaining APIC 2 and 3 configuration
• Verification
• OOB mgmt. IP’s for leafs and spines
• NTP configuration
• Route reflector

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Live demo #1
Configure the fabric
• Bringing workloads into the fabric
• VMware vCentre integration with UCS-B / UCS-C / vPC
• Bare metal integration

• Tenant / EPG’s
• External connectivity through OSPF

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Live demo #2
 bdsol-aci18-spine1 bdsol-aci18-spine2 bdsol-aci18-spine3 bdsol-aci18-spine4
Mgmt: 10.48.22.157 Mgmt: 10.48.22.158 Mgmt: 10.48.22.159 Mgmt: 10.48.22.160

Upgrade the fabric

1/1 1/2 1/35 1/36 1/1 1/2 1/35 1/36 1/1 1/2 1/35 1/36 1/1 1/2 1/35 1/36

• Download APIC software and

leaf/spine software to the fabric
• Verify critical hosts are dual bdsol-aci18-leaf1
Mgmt: 10.48.22.161 2/1 2/2 2/3 2/4 2/1 2/2 2/3 2/4
bdsol-aci18-leaf2
Mgmt: 10.48.22.162
2/1 2/2 2/3 2/4 2/1 2/2 2/3 2/4

connected bdsol-aci18-leaf3 bdsol-aci18-leaf4

Mgmt: 10.48.22.163 Mgmt: 10.48.22.164
1/47 1/48 1/33 1/34 1/33 1/34
1/47 1/48 1/47 1/48 1/25 1/47 1/48 1/25

• Separate in 2 maintenance groups

• Upgrade APIC’s
• Upgrade maintenance group 1
• Upgrade maintenance group 2
1/13 1/14 1/15 1/16 1/13 1/14 1/15 1/16

bdsol-aci-6506-01 bdsol-aci-6506-02
10.48.22.140 10.48.22.141
10G-p1 10G-p2
10G-p1 10G-p2
10G-p1 10G-p2 vmnicX vmnicX
vmnicX vmnicX
vmnicX vmnicX
M ï¡ï¡ï

M ï¡ï¡ï M ï¡ï¡ï
1/M 2

1/M 2 1/M 2

bdsol-aci18-esx1 - 10.48.22.147 bdsol-aci18-esx2 - 10.48.22.148 bdsol-aci18-esx3 - 10.48.22.149

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Live demo #3
Special considerations
Special considerations
• # 1 TEP pool / infrastructure VLAN
• #2 UCS-B connectivity with VMware

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
# 1 TEP pool / infrastructure VLAN
• TEP Pool
• Tunnel endpoint address pool
• IP addresses from this pool are assigned to all fabric elements to communicate
over the infrastructure VLAN
• “This subnet should not overlap with any other routed subnets in your network. If
this subnet does overlap with another subnet, change this subnet to a different /16
subnet. The recommended minimum mask is /19.
• Infrastructure VLAN:
• In-band VLAN
• Allows APIC to communicate with leafs and spines
• Hypervisor integrations, MultiPOD  Infrastructure VLAN and TEP Pool will be
extended
• OpenStack, Kubernetes, …

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
# 1 VTEP pool / infrastructure VLAN

OpenStack compute server

0.0.0.0/0 
10.142.35.1/24
NTP Server
10.0.0.35/24
Eth0
10.142.35.67/24

ACI Leaf Eth1.3912

Infra 10.0.0.95/16
VLAN

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 #2 UCS-B connectivity with VMware
ACI Leaf ACI Leaf
 Each Fabric Interconnect has a port-channel
towards the ACI Leafs
 Fabric interconnects are connected for clustering
 no data traffic is on the link
vPC vPC

L1 L1
 The hypervisor running on a blade has 2
Fabric Interconnect
L2 L2
Fabric Interconnect
independent connections  no switch
dependent protocols can be used
 Using IP-hash algorithms will cause MAC flaps on
the UCS FI’s and N5K’s
UCS 2104XP UCS 2104XP

UCS 5108

!
SLOT SLOT
1 2

1 1

SLOT SLOT
2 3 4 2

3 3

4 4

SLOT SLOT
5 6

SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

VMware

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 #2 UCS-B connectivity with Vmware (cont.)
ACI Leaf ACI Leaf

PC Interface Policy Group

vPC vPC Port Channel Policy  LACP
L1 L1

L2 L2
Fabric Interconnect Fabric Interconnect

UCS 2104XP UCS 2104XP

VSwitch Policy
1

2
UCS 5108

SLOT
1

SLOT
3
SLOT
2

SLOT
4
!

2
Port Channel Policy  MAC Pinning
3 3

4 4

SLOT SLOT
5 6

SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

VMware

#CLMEL BRKACI-2004 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Q&A

#CLMEL
Continue
your Cisco
Demos in
Labs Meet The
Expert
Related
sessions
education the World
of
Solutions

#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Complete Your Online Session Evaluation

• Give us your feedback and receive a

complimentary Cisco Live 2019 Power
Bank after completing the overall event
evaluation and 5 session evaluations.
• All evaluations can be completed via
the Cisco Live Melbourne Mobile App.
• Don’t forget: Cisco Live sessions will be
available for viewing on demand after
the event at:
https://ciscolive.cisco.com/on-demand-library/

#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you

#CLMEL
#CLMEL