IBM Spectrum Virtualize - BP Zoning 101-V9.2
IBM Spectrum Virtualize - BP Zoning 101-V9.2
IBM Spectrum Virtualize - BP Zoning 101-V9.2
Chuck Laing
Senior Technical Staff Member
IBM GTS SO/IS Storage Delivery
© Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
Session objectives
• The top 10 IBM Spectrum Virtualize (SVC) Best Practices (BP) zoning tips
• How to follow the BP rules of configuration to achieve :
• Proper dual core zoning for back-end devices
• Proper multiport (4 of more) for hosts and clusters
• Proper zoning for AIX LPM configurations
• Proper zoning for Enhanced Stretch Cluster
Action/s:
• The best practices will leverage Hursley supported configurations and tested
environments
– Varying from these recommendations may result in unexpected consequences. In addition, configuring as recommended
above will ensure the ability to replace nodes non-disruptively in the future.
– Do not zone the same ports:
• Host/Storage ports to ports designated for Inter-node use or Replication use in the 8/12/16 port configurations
• Inter-node and replication ports.
This is to minimize any B2B credit exhaustion situations, due to long distance latencies introduced by replication from
tying up buffers needed by hosts, storage or inter-node communications.
– With 12 or more ports per node we highly recommend 4 ports to be dedicated for inter-node traffic, especially when high
write data rates are expected (3Gb and greater) as all writes are mirrored over these ports within an I/O Group.
– GTS does not support dedicating certain ports specifically for hosts and other ports for storage as it negates the
full duplex capability of the fibre channel HBA ports.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 3
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
DH8 Port Layout 7.6.x– 16 Gb Capable speeds for New Builds
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 1213 14 15 16 1 2 34 5 6 7 8 9 10 11 1213 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA I/O Group 0 I/O Group 0
Action/s:
• The best practices will leverage Hursley supported configurations and tested
environments
– Recommendation - no mixed FC port speeds whenever possible - either all 8Gbps or all 16Gbps … else
– Use of mixed FC port speeds* should be limited to configurations with same speed ports dedicated (through zoning) to
either host/storage or inter-node/replication: Do not use mixed speeds for the same purpose (single host multipath with
mixed speeds is officially unsupported configuration).
– Same rules apply to another mixed port speeds configuration at 4x 8Gbps and 4x 16Gbps, again with dedicated ports
utilization enforced by zoning
– Mixed FC-iSCSI configurations should limit iSCSI ports oversubscription, to avoid turning SVC into a "NAS-like" appliance
– use the same as FC ports.
– For existing account environments where 2 port HBA cards at 16Gbs ports are used, you should ensure that the 8Gbs
HBAs remain in slots 1 and 2
– (*) One such configuration is shown is the next slide
Physical 1 2 1 2 2 2 2 2 3 3 5 5 1 2 1 2 2 2 2 2 3 3 5 5
port 1 2 2 2 1 2 3 4 1 2 1 3 1 2 2 2 1 2 3 4 1 2 1 3 Logical
number port with
1 2 34 56 78 9 10 11 12 1 2 34 56 78 9 10 11 12 wwpn #
Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5 embedded
16G HBA
Node 1 Node 2
8G HBA
I/O Group 0
• Regarding the calculation of the message login limit, here is how it works:
• This limit applies to any single node X, as seen by another node Y.
• Only 16 port logins are allowed from one node to any other node in an SAN fabric
• New error code 888, alert 801 will be generated if 16 exceeded
• Must correct zoning or masking or a combination of both zoning and masking to clear this issue
• To check whether the limit is exceeded, count the number of distinct ways by which a port on node X
can log into a port on node Y. This must not exceed 16.
• Only ports that are permitted to carry cluster message traffic need be included in the count.
• Any given two Nodes can see each other for messaging in 8 different ways (4 on each switch)
• Only the ports that can "see" each other (based upon switch zoning) need be included.
• Note : Port logins associated with host/storage ports, strictly speaking,
can be disregarded since they don't carry message traffic. For separate
reasons however, Hursley development recommends that the number of
Spectrum Virtualize ports in any given single zone be limited to avoid the
overheads associated with a large numbers of logins.
• There are three(3) types of Port masking to be aware of with the new
code and DH8
• Host port masking (mkhost)
• Default port masking, with 64 bits, used to be "1111" now it is "...111111111111". reading from right to
left...1 for each of the 12 ports. Leaving the default of 1 enables the flow of IO to pass through the node
port, "0" means to block IO on the port.
• Benefits
• zoning changes
• Less complexity for Vdisk IOgrp moves,
• Changes introduced in the SAN, or the hosts, with less complexity
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical
1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
port Logical
number port with
1 2 3 4 5 6 7 8 9 10 11 1213 14 15 16 1 2 34 5 6 7 8 9 10 11 1213 14 15 16
Slot 3 Slot 3 wwpn #
Slot 1 Slot 2 Slot 5 Slot 1 Slot 2 Slot 5
embedded
16G HBA Node 1 Node 2
8G HBA
I/O Group 0 I/O Group 0
STG
Zone-3
STG Host
Zone-1 Zone-2
Host
Zone-4
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
2 SVC Node
Target ports
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
1 SVC Node
Target ports
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
Host
A1 B1
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
A1 B1 A2 B2 A3 B3 A4 B4
Fabric_A Zone Definitions Fabric_A Zone Definitions
Fabric_B Zone Definitions A4=10:00:00:00:c9:42:80:16
Fabric_B Zone Definitions A3=10:00:00:00:c9:42:89:ac
B1=10:00:00:00:c9:3f:74:16
Fabric_A Zone Definitions B2=10:00:00:00:c9:3f:f7:2f
A1=10:00:00:00:c9:3f:75:37
Fabric_B Zone Definitions
Fabric_B Zone Definitions
Fabric_A Zone Definitions B4=10:00:00:00:c9:42:b6:18
B3=10:00:00:00:c9:42:b4:4b
A2=10:00:00:00:c9:3f:80:54
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
V7000
DS8K-1 DS8K-2
CAUTION 1 2 1 2 3 4 1 2 3 4 CAUTION
XIV
3 4 1 2
1 2 1 2
3 4 3 4
Disconnect all Disconnect all
supply power for supply power for
complete isolation 1 2 complete isolation
1 2
2 1
2 1
complete isolation complete isolation
supply power for supply power for
Disconnect all Disconnect all
4 3 4 3
2 1 2 1
2 1 4 3
4 3 2 1 4 3 2 1 2 1
CAUTION CAUTION
V7000
DS8K-1 DS8K-2
CAUTION 1 2 1 2 3 4 1 2 3 4 CAUTION
XIV
3 4 1 2
1 2 1 2
3 4 3 4
Disconnect all Disconnect all
supply power for supply power for
complete isolation 1 2 complete isolation
1 2
2 1
2 1
complete isolation complete isolation
supply power for supply power for
Disconnect all Disconnect all
4 3 4 3
2 1 2 1
2 1 4 3
4 3 2 1 4 3 2 1 2 1
CAUTION CAUTION
Storage Storage
Zone-1
Fabric1 Core1 Fabric 2 Core2
Zone-2
XIV Storage
Module1 Module2 Module3 Module4 Module5 Module6
HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2
P P P P P P P P P P P P P P P P P P P P P P P P
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
XIV Storage
Module1 Module2 Module3 Module4 Module5 Module6
HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2 HBA1 HBA2
P P P P P P P P P P P P P P P P P P P P P P P P
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
Same SAN
SAN 1
SAN 2
SAN 1
IBM.
BASIC BEST PRACTICE RULE # 9
Back-end Storage
Host X
HBA1 HBA2
A1 B1
P P P P
1 2 1 2
STG Host Host
STG Zone-2 Zone-1
Zone-1 Zone-2
IO request 1 ISL Hop1
1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5 1 2 1 2 2 2 2 2 3 3 3 3 5 5 5 5
Physical 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 1 2 3 4 1 2 3 4 1 2 3 4
Logical
port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16
port with
number Slot 1 Slot 2 Slot 3 Slot 5 Slot 1 Slot 2 Slot 3 Slot 5
wwpn #
16G HBA Node 1 Node 2 embedded
8G HBA
I/O Group 0
• Break down
Host_1a Host_1b Host_2a
Host_ Host_2b
A1
A1 B1 A1 B1 A1 B1 A1 B1
Fabric A Fabric B
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
• Break down
Host_1a Host_1b Host_2a
Host_ Host_2b
A1
A1 B1 A1 B1 B1 A1 B1
Fabric A Fabric B
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
• Break down
Host_1a Host_1b Host_2a
Host_ Host_2b
A1
A1 B1 A1 B1 B1 A1 B1
Fabric A Fabric B
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
• Break down
Fabric A Fabric B
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
• Create Pseudo host Spectrum Virtualize host definitions containing only two virtual wwpns, one from
each fabric
• Follow the rules shown in the next few slides
• Map LUNs to the virtual host FC HBA port wwpns not the physical host FCA Adapter wwpn
• Exception to map only boot from SAN LUNs to the physical host FC HBA wwpn
• Round Robin the Vdisks to the LPAR Pseudo and both the active and inactive LPARs to enable LPM
• Note: Following these rules will result in not having a degraded path error on the Spectrum Virtualize, and keep the 4 paths per Vdisk rule
Fame1 Hypervisor
P1 P3
Avoiding this
P2 P4
. SAN VP1.1i VFCA Active
•VFCA
VFCA
LPM Pseudo1 Client
Logical
P3 … P64 VP7.1i VFCA
I Could go to Partition
Fame2 VP5.1i VFCA
(Pseudo
Frame2 or LPAR1b)
Hypervisor Pseudo2 During LPM
Frame3 VFCA
VP3.1i
P1 P3
P2 P4 During LPM the number of paths double from 4 to 8
. SAN
•VFCA
Starting with 8 paths per vdisk will render an unsupported 16
VFCA
P3 … P64 I paths during this time - could lead to IO interruption
Fame3
Hypervisor
IBM Systems Technical Events | ibm.com/training/events
© Copyright IBM Corporation 2016. Technical University/Symposia materials
may not be reproduced in whole or in part without the prior written permission of
IBM.
51
VIO Server1 P1 P5 VIO Server2
FCA1 FCA3 VP1.1aVFCA
P2 P6 Active
Pseudo1 Client
P3 P7 VP7.1aVFCA Logical
FCA2 FCA4
P4 SAN P8 VP5.1aVFCA Partition
(LPAR1)
Pseudo2
VP1.1i VP1.1a VP5.1a VP5.1i VP3.1aVFCA
VP2.2i VP2.2a VP6.2a VP6.2a
SVC VP2.2aVFCA
VP3.1i VP3.1a VFCA4.1VP7.1i
VP7.1a Pseudo1 Active
VP8.2aVFCA Client
VP4.2i VP4.2a VP8.2a VP8.2i
I VFCA
VP6.2a Logical
Pseudo 2 Partition
(LPAR2)
VP4.2aVFCA
inactive & active LPM
vWWPN pairs
Fame1 Hypervisor
P1 P3
P2 P4
. SAN VP1.1i VFCA Active
•VFCA
VFCA
LPM Pseudo1 Client
Logical
P3 … P64 VP7.1i VFCA
I Could go to Partition
Fame2 VP5.1i VFCA
(Pseudo
Frame2 or LPAR1b)
Hypervisor Pseudo2 During LPM
Frame3 VFCA
VP3.1i
P1 P3
P2 P4 During LPM the number of paths double from 4 to 8
. SAN
•VFCA
Starting with 8 paths per vdisk will render an unsupported 16
VFCA
P3 … P64 I paths during this time - could lead to IO interruption
Fame3
Hypervisor
IBM Systems Technical Events | ibm.com/training/events
© Copyright IBM Corporation 2016. Technical University/Symposia materials
may not be reproduced in whole or in part without the prior written permission of
IBM.
52
Dual VIOS to Multiple LPARs
Is it resilient? - One VIOS Failure
x
VFCA
VP4.2a
Fame1 Hypervisor
© Copyright IBM Corporation 2016. Technical University/Symposia materials 53
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
Dual VIOS to Multiple LPARs
Is it resilient? – One SAN Fabric Failure
Fame1 Hypervisor
Clustered Servers
where applicable
SVC Cluster 3
SVC Cluster 0
SVC Global Mirror Node 1 Node 2
Site-1 Site-2 ASYNC
DR Site
• Quorum
Storage Pool 1 Storage Pool 2
Storage Pool 3
B COPY
C COPY
Mission Critical Block for Applications
HA Storage: can sustain .3-1 ms latency hit) DR, Dev, Test, QA .. etc.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 55
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
Spectrum Virtualize (SVC) ESC – Site-1 & Site-2 current building block example
2X
• A Core Private Cisco VSAN is configured for Node to
2X
Node Communication port connections
2X 2X
Node to
node
Node to
node
• 12 Port Nodes are used with 4 ports each at 8Gb port
2X
2X speeds
Public
9710 Core • A Core Public enterprise class Cisco VSAN is
VSAN Public configured for Host/Storage port connections
3x / 3x / VSAN
Private nod nod
e Private
VSAN e VSAN • A Core Private Cisco VSAN is configured for Node
to Node Communication port connections
SVC DH8
2
X 2
2
X • 12 Port Nodes are used with 4 ports each at 8Gb
X 2
X
port speeds
2
2
X
X
2 2 • Edge Switches per port count requirements are
X X
2 2
used for a true BP Core-Edge design to allow future
X
2
X growth scalability
2
X
X
2 2
X
2
X
2 • Storage building blocks of Flash, XIV and V7000 are
X
X used for a 3 tier or mixed tier approach
4x/Flash, 4x/Flash, • Separate Storage zones are created for each device
6x/XIV, 6x/XIV,
2x/V7K 2x/V7K providing isolated zones from each device type
8x ISL / Edge
8x ISL / Edge
• Host Servers are connected to the Edge Switches
Site1 Site2
FCIP GM
4X
4X
4X
4X
2X @ 16Gb/s CWDM LW SFP
DR Site
1 2 3 4
Submit four or more session
evaluations by 5:30pm Wednesday
to be eligible for drawings!
*Winners will be notified Thursday morning. Prizes must be picked up at
registration desk, during operating hours, by the conclusion of the event.
ibm.com/training
provides a comprehensive
portfolio of skills and career
accelerators that are designed
to meet all your training needs.
If you can’t find the training that is right for you with our
Global Training Providers, we can help.
Backup Slides
• All Spectrum Virtualize nodes must see same set of LUNs from disk
controller
– Otherwise degraded mode on controller and/or MDisks
• Create two cluster zones (do not include MM/GM and intra-cluster traffic
ports)
• Never span zones to include more than one Backend storage device!
• Never put Host OS ports, Spectrum Virtualize ports and Backend Storage
ports together in the same zone
– Instead
• Create zones with Host ports and Spectrum Virtualize ports
• Create zones with Backend and Spectrum Virtualize ports
– Never use the same DS8K ports or any native back-end port for
connectivity to Spectrum Virtualize and an attached host
• If Spectrum Virtualize is attached to the DS8K or other native back-end
devices and the DS8K or other back-end device is using native GM (not
Spectrum Virtualize GM) then dedicate appropriate back-end ports
specifically for GM, not to be used for attaching any other device, whether
Host Server, Spectrum Virtualize or other connectivity relationships.