Storage solutions

Topics

FC SAN (Fiber Channel Storage Area Network) iSCSI (Internet Small Computer System Interface) NAS (Network Attached Storage) VMFS (Virtual Machine File System)

07/04/2013

Fibre Channel SAN Storage

Components of FC SAN

07/04/2013

Basic concepts of storage

Storage System This is the hardware itself, which manages disks into logical units (RAID). LUN Logical Unit Number A part of logical unit. One logical unit can consist of multiple LUNs (and usually does) RAID, JBOD (Just a Bunch of Disks)*, Hot Spare, Parity** SP Storage Processor WWN (World-Wide Name) 64 Bit number, to which we can attach a label Fabric part of SAN storage, components which are part of one access path (serverport, switch, storage port) HBA Host Bus Adapter the card which connects a host to the Fiber Channel network Switch) A non-RAID (SAN disk architecture
**A parity bit is a bit that is added to ensure that the number of bits with the value one in a set of bits is even or odd. Parity bits are used as the simplest form of error detecting code.

07/04/2013

Basic concepts of storage

Storage security Which ESX host should see which LUN?

FC / SAN Switch The most important component of the FC network, the HBAs can be connected to the SAN switch

SAN Zoning A method, which makes it possible to create logical groups containing WWN nodes over the physical Fabric Channel network; it is applied on the FC switch level; it is similar to VLANs used in Ethernet networks LUN Masking A method, which makes it possible to hide certain Luns from certain hosts; usually implemented on HBA level

07/04/2013

SAN Zones

07/04/2013

WWN, Zoning, Masking

07/04/2013

Zoning

07/04/2013

Storage multipath

Storage multipath is a method which is vital in order to achive and maintain high availablity Active/Active array (rarely used) Active/Passive array (frequently used) There are two types of multipathing policies in an ESX evironment:

This is automatically set by the ESX, but can be changed.

Fixed The ESX only uses the preferred path, path failover policy will only be used in case of failure. MRU (Most Recently Used) : The path used most recently will be used in case of a failure, this is set by the ESX automatically. RoundRobin Experimental only

07/04/2013

10

Addressing SAN LUNs in VMkernel

The VMkernel is addressing the LUNs and disk partitions the following way:

Examples: LUN addresses vmhba0:0:11 vmhba1:1:12

Partition addresses vmhba0:0:11:3 vmhba1:1:12:1

11

07/04/2013

Example for addressing in practice

07/04/2013

12

iSCSI storage

Topics

Components and addressing of iSCSI iSCSI hardware and software initiators Configuration of iSCSI softwareinitiated storage

07/04/2013

14

What is iSCSI

SCSI transport protocol, creating connection to storage devices over stanard TCP/IP protocol It maps SCSI block-based storage over TCP/IP The Initiators, for example an HBA in the ESX, send SCSI commands towards the targets within the iSCSI Storage System
The task itself does not differ from Fiber Storage connection

IP
07/04/2013 15

Types of iSCSI connections

Booting ESX server from iSCSI Storage

Only possible with hardware initiator

VMFS on iSCSI LUN

Usable just like on an FC SAN Storage

VMs access to raw iSCSI LUNs VMotion service for VMs which are on iSCSI LUNs
16

07/04/2013

Components of iSCSI SAN

IP Network

* Software implementation
07/04/2013 17

Addressing of IP SAN
iSCSI target name iqn.1992-08.com.netapp:stor1
iSCSI alias stor1 IP address 192.168.36.101

iSCSI initiator name iqn.1998-01.com.vmware:train1 iSCSI alias train1 IP address 192.168.36.88

IP Network

07/04/2013

18

How iSCSI LUNs can be reached

Two discovery methods are supported: Static configuration SendTargets iSCSI device sends target info on request

192.168.36.101:3260

IP Network SendTargets response

SendTargets request

07/04/2013

iSCSI target 192.168.36.101:3260

19

How iSCSI storage authenticates the ESX Server

CHAP (Challenge-Handshake Authentication Protocol) Using this authentication method makes it possible to authenticate through the network without actually sending it.

CHAP password: K

CHAP password: K

Log into target (only request) Computes using formula against K using C and sends result R

C R

Create random hash/computation value C send value back to ESX Server; perform local computation against K using C to come up with R. Compares local R against returned R; If match, then ESX Server is authenticated.

07/04/2013

20

iSCSI software and hardware initiator

ESX Server 3 provides full support to software initiators
Software Initiator Hardware initiator

07/04/2013

21

Set up networking for iSCSI software initiator

Both Service Console and VMKernel has to be able to reach the iSCSI Storage There are two possibilities for this: 1. Service Console and VMKernel port share a virtual switch, in the same subnet.

2. Local routing, so that Service Console and VMKernel both reach the iSCSI storage

07/04/2013

22

Software iSCSI client has to be enabled in the firewall

07/04/2013

23

Configure iSCSI software adapter

07/04/2013

24

Configure software initiator: General properties Enable iSCSI initiator

07/04/2013

25

Configure software initiator: General properties (2)

The iSCSI name and alias will be filled automatically after activation of the initiator

07/04/2013

26

Configure software initiator: Dynamic discovery

the IP address of all target can be given on Dynamic Discovery tab All pinging targets will appear on Static Discovery tab
27

07/04/2013

Configure software initiator: CHAP Authentication

CHAP is turned off by default After enabling it a username and password can be supplied
28

07/04/2013

Discover iSCSI LUNs

A rescan is needed after any modification in iSCSI configuration

07/04/2013

29

Multipathing with iSCSI

SendTargets advertises multiple routes

Routing done via IP network For the software initiator

It reports different IP addresses to allow different paths to the iSCSI LUNs

IP Network

Counts as one network interface NIC teaming and multiple SPs allow for multiple paths
30

07/04/2013

Managing multiple paths

07/04/2013

31

VMFS datastore

Topics

VMFS-3 filesystem Creating VMFS partition VMFS partition align VMFS Metadata VMFS UUID VMFS Extending Distributed Lock Handling Importance of SCSI ID, LUN syncronization, DR site scenario Raw Device Mapping
33

07/04/2013

What is VMFS?

A filesystem optimized for virtualization, capable of clustering In case of shared storage solution, capability of SAN interrupts In service console /vmfs/volumes contains the VMFS volumes
Each virtual machines files are located in its own subdirectory

Storing further files

Templates, ISO images

07/04/2013

34

Creating a VMFS

Choose device location (iSCSI or FC LUN)

Choose datastore name

Maximum file size and Capacity can be modified

07/04/2013

35

Creating a VMFS

Recommended from GUI (partition align) Choosing the block size:

1MB 2MB 4MB 8MB block block block block size size size size 256GB maximum file size 512GB maximum file size 1024GB maximum file size 2048GB maximum file size

The size of the smallest files will be the block size

07/04/2013

36

VMFS Partiton align

Misaligned partitons cause performance problems (I/O) Formatting from VI will result in automatic alignment but the recommendations of the manufacturer should be taken into consideration VM Guest OS align (the file system of the guest OS has to be aligned as well)

07/04/2013

37

VMFS partition align

Misaligned VMFS and Guest OS partitions

07/04/2013

38

VMFS partition align

Aligned VMFS and Guest OS partitions

07/04/2013

39

VMFS properties

What is to be found in VMFS metadata

Block size Number of extents Volume size Version of VMFS Label VMFS UUID

Vmfstools P h <Label> (e.g: vmkfstools P h LUN_5)

-P -h read metadata set unit of volume size

07/04/2013

40

VMFS properties

UUID: Universal Unique ID This identifies the VMFS volume Similar to SID (AD), but in this case not the RID master FSMO creates them, but the LVM (Logical Volume Manager) Structure of UUID: 123abc98-12abc456bd12-000569ab56bf

123abc98 Time of VMFS creation or resigniture 12abc456 CPU timestamp counter value Bd12 Random number 000569ab56bf MAC address of service console IF (which created or resignatured the UUID)
41

07/04/2013

Distributed Lock Handling by VMFS-3

ESX1 and ESX2 mount a VMFS volume The host IDs appear in the VMFS heartbeat The host IDs are updated regularly ESX1 locks the file, for identification the host ID is used In case ESX1 stops or loses connection to the volume, the lock will be out of date (the Distributed Lock Manager DLM sees that a dead host owns a file) e.g. HA failover ESX2 tries to lock the file previously locked by ESX1 ESX2 looks up the host ID record of ESX1 in its VMFS heartbeat If the host ID record is not updated by ESX1, ESX2 takes over the lock The other hosts dont try to lock the file agains ESX2 Tries to rollback the VMFS file journal DLM asks the journaling module to recover the filesystem (if necessary) The lock is at the DLM which uses the out-of-date lock The old lock is unregistered, and ESX2 gets to lock the file
42

07/04/2013

VMFS properties

07/04/2013

43

Extend VMFS

The size of a VMFS is dynamically extendable Possible to add more space to a VMFS online Possible to create filesystems bigger than 2 TB
Before spanning After spanning

vmhba0:0:3:1

vmhba0:0:6:1

vmhba0:0:3:1

vmhba0:0:6:1 Extent

07/04/2013

44

Datastore

Datastores consist of VMFS partitions Max. 32 extent (similar to windows) Ideal for creating big datastores Sometimes it might be necessary (the customer is increasing the datastore slowly) If the master is lost, data from all extents are also lost All ESX has to see all extents LUN, otherwise the whole datastore falls apart Because of these the master VMFS can not be on local storage because then not all VM can see the datastore, or if it is so, all VMFS has to be on local storage, and then only one VM can see it
45

07/04/2013

Adding new extent to VMFS

The partiton can not be VMFS to which you want to extend If you choose one which contains data, the program (VI) will notify you that the data will be lost

07/04/2013

46

Adding new extent to VMFS

If one LUN can be maximum 2 TB of size, how is it possible to create a max 64 TB file system? Solution: extending: 32 x 2 TB = 64 TB

07/04/2013

47

VMFS extent list updated

07/04/2013

48

SCSI ID

SCSI ID formats known by ESX

Network Address Authority (frequent) Serial Number (IBM Shark) Symm6 (Symmetrix 6 and pre)

07/04/2013

49

Importance of SCSI IDs

When a VMFS-3 volume is created on a LUN, the SCSI ID comes from the SAN storage (unique) and is stored in the header of the LVM When an ESX scans the LUNs of the storage and finds a VMFS-3 file system, it compares the SCSI ID information of the storage and the one stored in the VMFS LVM metadata This is why it does not mount a replicated LUN, because the SCSI ID is not recognized It can not be mounted because the UUID is the same as the originals and a VM can not mount VMFS volumes which have the same UUID
50

07/04/2013

EnableResignature vs. DisallowSnapShotLUN

LVM.EnableResignature LVM.DisallowSnapShotLUN When a LUN is cloned (LUN1 replicated to LUN2) to another storage, the ESX hosts do not find the cloned LUN when rescanning DisallowSnapShotLUN ESX B cannot see LUN1 EnableResignature If ESX B can see LUN1 resignature is necessary, otherwise there will be a conflict because of the UUID This also means that the VMs need to be reregistered under the VI because the UUID is also stored in the vmx files
51

07/04/2013

EnableResignature vs. DisallowSnapShotLUN

07/04/2013

52

RDM

RDM Raw Device Mapping Direct link to a raw LUN on a storage Its slightly faster the VMFS (e.g. database) but it depends on the application Physical and virtual compatibility mode Physical: the most direct access for a VM However we lose some options like template creating, snapshoting (VCB backup) migrations which cause moving of disks; these are prerequisites for some services, such as Microsoft Clustering under VMWare Virtual: We have the opportunity to fully use VMWare functions (snapshot, cloning, migrating)
53

07/04/2013

NAS Storage and NFS datastores

Topics

NAS storage NAS components, addressing NAS datastore configuration

07/04/2013

55

What kind of NAS is supported by ESX?

The VMkernel only supports NFS

Actually it is NFSv3 over TCP

NFS volumes are the same in FC and iSCSI They are capable of storing the VMs disks
ISO templates VM templates

When the VMs disks are on a NAS, they can be vMotion-ed in case: CPU compatibility All necessary network and storage are reachable on the target as well
56

07/04/2013

What is NAS and NFS

NAS
Network Attached Storage Storage shared over the network at a filesystem level

Why use NAS?

A low cost, moderate-performance option

There are two key NAS protocols

NFS (the Network File System) SMB (Windows networking, also known as CIFS)

Major NAS applicances support both NFS and SMB

Notably those from the Network Appliance and EMC

Server OS support both

57

07/04/2013

NAS device or server

Volume which we share with ESX over the network

ESX server with a physical NIC attached to a virtual switch

VMkernel port on the virtual switch

07/04/2013

58

Addressing and access control with NFS

07/04/2013

59

Before you begin using NAS/NFS

Create a port on a virtual switch

A seperate IP is necessary for using NAS, different than the Service Console IP

07/04/2013

60

Configure an NFS datastore

Describe the NFS share

07/04/2013

61

Configure an NFS datastore (cont.)

Verify that the NFS datastore has been added

07/04/2013

62

Summary

An NFS datastore can be used to hold templates, ISO images and the files that make up a VM ESX Server supports NFS version 3 over TCP A VMkernel port must be configured in order for the VMkernel to access the IP-based storage
63

07/04/2013