Troubleshooting vSphere Storage

Table of Contents

Troubleshooting vSphere Storage

Credits

About the Author

Acknowledgment

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Instant Updates on New Packt Books

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Errata

Piracy

Questions

1. Understanding vSphere Storage Concepts and Methodologies

Storage virtualization

Supported filesystems

VMFS

NFS

Raw disk

Storage naming

Viewing device identifiers

Within the vSphere Client

Within ESXi Shell

The many ways vSphere identifies storage

Friendly names

Identifiers

NAA identifiers

T10 identifiers

IQN identifiers

MPX identifiers

Runtime names

The vSphere Pluggable Storage Architecture

Pluggable Storage Architecture (PSA) roles and commands

Multipathing Plugin – the VMware Native Multipathing Plugin roles and commands

Storage Array Type Plugin roles and commands

Path Selection Plugin roles and commands

An I/O request – from start to finish

Summary

2. Commonly Used Tools for Troubleshooting Storage

The vSphere troubleshooting methodology

vCenter Storage Views (Reports and Maps)

Reports

Maps

Using esxtop/resxtop

Switching displays

Field customization

Refresh interval

Logfiles used in vSphere Storage troubleshooting

ESXi logging

Viewing ESXi logs

vCenter Server logging

Virtual machine logging

Summary

3. Troubleshooting Storage Visibility

Common storage visibility issues across block transport types

ESXi claim rules and LUN masking

Troubleshooting paths and path selection

vCenter Server Storage filters

Disk resignaturing

LUN numbering

Troubleshooting Fibre Channel storage visibility

Registered State Change Notification (RSCN)

Identifying and interpreting Fibre Channel connection issues in the logs

Troubleshooting IP storage visibility

Verifying network connectivity to our iSCSI and NFS array

iSCSI and NFS port requirements

CHAP authentication

Identifying and interpreting iSCSI storage connection issues in the logs

Troubleshooting NFS storage

NFS ACLs

NFS permissions

Identifying and interpreting NFS storage connection issues in the logs

Summary

4. Troubleshooting Storage Contention

Identifying storage contention and performance issues

Disk adapter latency statistics

Disk device latency statistics

Virtual machine latency statistics

Using vCenter alarms to monitor latency

Planning – how to properly scale your storage

Calculating IOPs requirements

Calculating the number of disks required

How do we know when we need more IOPs?

vSphere features to help with latency, contention, placement, and compliance

Profile-Driven Storage

Storage DRS for performance

SCSI reservation conflicts

Monitoring SCSI reservation conflicts with esxtop

Resolving excessive SCSI resolutions

Storage queuing in vSphere

OS queue

Adapter queue

Per-LUN queue

Viewing queues in ESXi

vSphere Storage I/O Control

Configuring Storage I/O Control

Troubleshooting NAS/iSCSI storage performance

iSCSI port binding and Round Robin path policy

Jumbo Frames and MTU size

Summary

5. Troubleshooting Storage Capacity and Overcommitment

Monitoring datastore usage

Thin Provisioning

Array thin provisioning

Hypervisor (VMDK) thin provisioning

Monitoring Thin Provisioning

Snapshots

Understanding vSphere Snapshots

Monitoring snapshot disk usage

Datastore file management

Virtual machine swap files

Increasing the size of our datastores

Growing a VMFS datastore

Adding an extent to a VMFS datastore

Balancing capacity with Storage DRS

Summary

A. Troubleshooting Steps

You cannot connect to your iSCSI storage

Have all the network requirements for your iSCSI connection been met?

Various things to check on ESXi

Have all CHAP requirements been met?

Has there been any advanced settings dealing with iSCSI incorrectly configured?

Further check the logs for iSCSI-related errors

You cannot connect to your NFS storage

Have all the network requirements for you NFS connection been met?

Check the settings on your NFS array

Has there been any advanced settings dealing with NFS incorrectly configured?

Further check the logs for NFS related errors

You cannot connect to your Fibre Channel Storage

Fibre Channel switch zoning

Check paths to storage array/PSP

Various things to check on ESXi

Check the storage array

Further check the logs for Fibre Channel related errors

My storage/virtual machine is slow

Inspect latency values to further pinpoint where performance degradation is occurring

Ensure that your storage array is handling the demand

B. Statistics of esxtop

Overview of esxtop's interactive commands

Activating different displays

Field selection

Field order

Filtering and sorting the displayed objects

Other useful information

esxtop statistics

C. iSCSI Error Codes

Interpreting software iSCSI error codes

Index

Troubleshooting vSphere Storage

Troubleshooting vSphere Storage

Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: November 2013

Production Reference: 1131113

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78217-206-2

www.packtpub.com

Cover Image by Aniket Sawant (<[email protected]>)

Credits

Author

Mike Preston

Reviewers

Jason Langer

Mario Russo

Aravind Sivaraman

Eric Wright

Acquisition Editor

Owen Roberts

Commissioning Editor

Shaon Basu

Technical Editors

Kanhucharan Panda

Vivek Pillai

Project Coordinator

Aboli Ambardekar

Proofreader

Jenny Blake

Indexer

Mariammal Chettiyar

Graphics

Yuvraj Mannari

Production Coordinator

Arvindkumar Gupta

Cover Work

Arvindkumar Gupta

About the Author

Mike Preston is an IT professional and an overall tech enthusiast living in Ontario, Canada. He has held all sorts of IT titles over the last 15 years including Network Technician, Systems Administrator, Programmer Analyst, Web Developer, and Systems Engineer in all sorts of different verticals, from sales to consulting. Currently, he is working as a Systems Analyst supporting the education market near his home in Belleville, Ontario.

Mike has always had an intense passion for sharing his skills, solutions, and work with various online communities, most recently focusing on the virtualization communities. He is an avid blogger at blog.mwpreston.net and participates in many discussions on Twitter (@mwpreston). It's his passion for sharing within the virtualization community which has led to Mike receiving the vExpert award for 2012 and 2013.

Mike has presented at VMworld, VMUGs, and various virtualization conferences on numerous times both as a customer and an overall evangelist and has published different whitepapers and articles for various tech websites. His commitment to giving back to the community has resulted in his most recent venture of becoming a Toronto VMUG co-leader. He is a VMware Certified Professional in Datacenter Virtualization on both Version 4 and 5 of vSphere and is currently pursuing his VCAP5-DCA, which he hopes to accomplish by 2014.

Acknowledgment

Firstly, I would like to thank my wife Alicia for putting up with all my late night and early morning writing sessions. I would also like to thank my two children, Hayden and Clara. Without their support, the completion of this book would have been nearly impossible.

Secondly, I would like to thank my technical reviewers Jason Langer, Eric Wright, and Angelo Luciani. Having those powerful brains comb over my work is one of the best decisions I ever made.

Lastly, I would like to thank the virtualization community. Without the many hours donated by the mentors, vRockstars, and others involved in the community, I would certainly not have been able to accomplish a project of this size. It's definitely a welcoming and helpful community and I'm humbled and happy to be a part of it.

About the Reviewers

Jason Langer works as a Solutions Architect for a VMware partner in the Pacific Northwest region helping customers achieve their datacenter virtualization goals. Jason has obtained multiple levels of certification both from Microsoft and VMware and brings 15 years of IT experience to the table. When not working his day job, Jason is active in the VMware community as a member of the Seattle VMUG Steering Committee and generating content for his blog, virtuallanger.com.

He is also currently working as a technical reviewer for VMware ESXi 5.1 Cookbook, Mohammed Raffic Kajamoideen, Packt Publishing.

Mario Russo has worked as an IT Architect, a Senior Technical VMware Trainer, and in the pre-sales department. He has also worked on VMware technology since 2004.

In 2005, he worked for IBM on the First Large Project Consolidation for Telecom Italia on the Virtual VMware ESX 2.5.1 platform in Italy with Physical to Virtual (P2V) tool.

In 2007, he conducted a drafting course and training for BancoPosta, Italy, and project disaster and recovery (DR Open) for IBM and EMC.

In 2008, he worked for the Project Speed Up Consolidation BNP and the migration P2V on VI3 infrastructure at BNP Cardif Insurance.

He is a VMware Certified Instructor (VCI) and is certified in VCAP5-DCA.

He is also the owner of Business to Virtual, which specializes in virtualization solutions.

He was also the technical reviewer of the book, Implementing VMware Horizon View 5.2, Jason Ventresco, Packt Publishing.

I would like to thank my wife Lina and my daughter Gaia. They're my strength.

Aravind Sivaraman is a Virtualization Engineer with more than seven years of experience in the IT industry and for the past five years, he has been focused on virtualization solutions especially on VMware products. He has been awarded with the VMware vExpert title for the year 2013. He is a VMware Technology Network (VMTN) contributor and maintains his personal blog at http://aravindsivaraman.wordpress.com/. He can also be followed on Twitter (@ss_aravind).

I would like to thank my wife and my family members for supporting me towards the contribution of this book.

Eric Wright is a Systems Architect and VMware vExpert with a background in virtualization, business continuity, PowerShell scripting, and systems automation in many industries including financial services, health services, and engineering firms. As the author behind www.DiscoPosse.com, a technology and virtualization blog, Eric is also a regular contributor to community driven technology groups such as the VMUG organization in Toronto, Canada. You can connect with Eric at www.twitter.com/DiscoPosse.

When Eric is not working in technology, you may find him with a guitar in his hand or riding a local bike race or climbing over the obstacles on a Tough Mudder course. Eric also commits time regularly to charity bike rides and running events to help raise awareness and funding for cancer research through a number of organizations.

I wish I could thank everyone personally, but let me say thank you to my family, friends, and the very special people who've inspired me to be involved with technology. Thank you to the amazing and very accepting technology community who have helped me to be able to share my knowledge and to learn from the amazing minds that drive this incredible community.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt