Enabling Remote Access (SSH) to the ESX/ESXi Host

Every command in this article is run from the host, therefore you'll have to enable remote SSH
access to the host. Assuming that you don't have direct physical access to the host system, you'll
have to enable remote access via the vSphere Client; here's how:

1. Open and connect to your vSphere vCenter Server using your vSphere Client.

2. Select a host system. In the right pane, select the Configuration tab.

3. Select Security Profile in the Software Section.

4. Click the Properties link in the upper right of your vSphere Client.

5. Select Remote Tech Support (SSH) and click the Options button.

6. Select the 'Start and stop with host option,' click the Start button, and then click OK to finish.

7. Click OK to close the Properties window.

At this point, you might receive a prompt to reboot the ESX/ESXi host. If so, allow the host to reboot.

If your system required a reboot, allow it to reconnect to your vCenter Server and then open
a connection to it using an SSH client, such as PuTTY (Windows) or command line SSH. Login to
the host using the host's root account or using your assigned host credentials.

All of the tools described in this article are located in /sbin along with standard Linux utilities.
Be aware that an ESXi host only maintains a relatively small subset (~300) of the entire complement
of standard Linux commands.

The default and only available shell on ESXi systems is ash. This lightweight shell is part of
the BusyBox system. There are some missing utilities in BusyBox that the esxcfg utilities replace.
For example, route and ifconfig are both missing, however esxcfg-route, esxcfg-nics, and esxcfg-
vmknic adequately substitute for them.
Using the esxcfg Utilities

On ESXi 4.1 update 3 hosts, there are 20 utilities that all share the common esxcfg- prefix. To look
at usage syntax for these commands, enter the following at a host prompt:

# esxcfg-<utility> - - help

The <utility> is one of the 20 unique utility identifiers. The esxcfg utilities include:

1. esxcfg-advcfg

2. esxcfg-dumppart

3. esxcfg-hwiscsi

4. esxcfg-info

5. esxcfg-init

6. esxcfg-ipsec

7. esxcfg-module

8. esxcfg-mpath

9. esxcfg-nas

10. esxcfg-nics

11. esxcfg-pciid

12. esxcfg-rescan

13. esxcfg-resgrp

14. esxcfg-route

15. esxcfg-scsidevs

16. esxcfg-secpolicy

17. esxcfg-swiscsi
18. esxcfg-vmknic

19. esxcfg-volume

20. esxcfg-vswitch

Most of these utilities can be used with the –l switch, which lists information about your host system.
Gathering a baseline can help you troubleshoot your system when future problems arise. To that
end, once you've installed your ESX or ESXi system and have established your datastores,
networks, and switches; you should capture your system information to files and then transfer them
to a central repository.

For example, execute the following three commands on your system to gather route and NIC
information.

# esxcfg-route –l > balvenie-route.txt

Figure 1: Checking the default route configuration on an ESXi host.

# esxcfg-nics –l > balvenie-nics.txt

Figure 2: Displaying physical NIC information.

# esxcfg-vmknic –l > balvenie-vmknic.txt

Figure 3: Displaying ESXi host virtual network adapter information.

# esxcfg-vswitch –l > balvenie-vswitch.txt

Figure 4: Gathering virtual switch information.

Collect all the information from your host and transfer it to your workstation or network
drive for future reference. If you experience a failure or have to reinstall a host system, you can refer
to these documents to rebuild the new system with the same configuration as the old one.

Once you've gathered all of your host system's information, you find that you want to
change a particular parameter. For example, you find from running the esxcfg-info utility that the
NFS MaxVolumes setting currently configured on your system is eight, which is the default. The
maximum possible setting is 64.
See Figure 5.

Figure 5: Examining esxcfg-info NFS MaxVolumes configuration information.

You want to change this host's setting to 16. To change values observed from esxcfg-info, you must
use the esxcfg-advcfg utility.
# esxcfg-advcfg -s 16 /NFS/MaxVolumes Value of MaxVolumes is 16

This command sets the MaxVolumes setting to 16.

The esxcfg-info command gives you the setting value and the parameter name that you'll need in
order to make changes to your system using the esxcfg-advcfg utility. If you know the parameter
name, you can query it directly using the esxcfg-advcfg utility.

# esxcfg-advcfg –g /NFS/MaxVolumes Value of MaxVolumes is 16

You can issue the esxcfg-advcfg –g (get) command at any time to have a look at the current value
of a setting.

One word of caution before we leave the discussion of the esxcfg utilities: Use extreme discretion when
issuing esxcfg-init commands. Generally speaking, you should only issue these commands under
direction either of an advanced VMware Administrator or of a VMware Support Technician.

Using the esxcli Utility

The general syntax of the esxcli utility is:

esxcli [dispatcher options] <namesspace> <object> <command> [options]

The esxcli utility might remind you of the very powerful Windows utility "netsh" in that they both use
namespaces or contexts within you can view or alter various settings and configurations. The six
namespaces that you can use with esxcli are:

 corestorage -- VMware core storage commands

 network -- Networking commands

 nmp -- Native Multipath Plugin (NMP)

 swiscsi -- iSCSI commands

 vaai -- VAAI Namespace containing vaai code

 vms -- Limited virtual machine operations

As I stated earlier, there are a few missing native utilities from a basic Linux installation in BusyBox.
One of those missing utilities is "netstat." The netstat utility can diagnose a number of network
problems including "hung" daemons, network connectivity issues, and ports that aren't listening. You
can also detect certain network "attacks" using netstat. However, the esxcli utility replaces netstat.

To understand the statement that esxcli replaces netstat issue the following command:

# esxcli network connection list

You will see a list of tcp and udp listening ports, established ports, some TIME_WAIT,
CLOSE_WAIT, and FIN_WAIT statuses.

Another interesting use of esxcli is to kill virtual machines that are "stuck" and not responding to
normal stop operations. This situation occurs when a host has experienced some fault and can't be
placed into maintenance mode without some intervention. It's not recommended to use the "kill"
functionality under normal circumstances.

To use esxcli kill, first list the host's virtual machines that are still powered on:

# esxcli vms vm list

Figure 6: Listing the powered on virtual machines.

To kill a virtual machine, you have to supply the type of kill: soft, hard, or force. You also have to
supply the virtual machine's World ID. Requiring these explicit parameters avoids accidentally killing
virtual machines by using the virtual machine name only.
# esxcli vms vm kill –t soft –w 29156592

You should attempt to kill an errant virtual machine with soft, hard, and force in that order so that
critical data isn't lost in the process.
If the "kill" command is successful, another "esxcli vms vm list" will return no data.

Exploring the 'vm' Commands

Of the list of 25 "vm" commands under /sbin, you'll find a few useful utilities that you can use for
troubleshooting. Three of the utilities are extended variants of the "lite" versions contained in
BusyBox. They are: vmkping, vmtar, and vmcp. These versions function like the open source
versions that you're used to having access to on your Linux systems. Use them instead of the
standard system ping, tar, and cp. Note that the system ping command is a symbolic link to
the /sbin/vmkping executable.

The "vmware" utility displays your version information. Have this information ready to give to a
VMware support technician or to add to your incident ticket on the VMware incident support website.

# vmware –l

VMware ESXi 4.1.0 Update 3

# vmware –v

VMware ESXi 4.1.0 build-800380

For major troubleshooting, VMware support will likely prompt you to run the "vm-support" utility.
This utility thoroughly checks your system and may take several minutes to complete. While running,
you'll see a status checker and updates:

# vm-support –l

Preparing files: /

When complete, the system gives you instructions that include:

 The name of the collected support file.

 The incident support website.

 How to view the collected files.

Your VMware support professional might ask you to use different switches when gathering system
information. You'll have to transfer the collected files and send them to the VMware support tech via
email or upload to a support site via FTP.

The vmdumper utility is a unique one for troubleshooting. Using it, you can successfully BSOD a
Windows system or kernel panic a Linux system. To do so, issue the following command to list any
running virtual machines:

#vmdumper –l

wid=31885225 pid=-1 cfgFile="/vmfs/volumes/531f2d35-24323028-8423-

0019b9b8fe1b/Automat/Automat.vmx" uuid="42 11 ad 85 88 38 ab d7-b4 d1 0d 35 ce 6c c0
b0" displayName="Automat" vmxCartelID=31881128

The important pieces of information from this response are the displayName and the World ID
(wid). You need the displayName to verify that you are crashing the correct system and the World ID
is required for the vmdumper command.

# vmdumper [World ID] nmi

# vmdumper 31885225 nmi

Sending NMI to guest...

Figure 7 below shows the screen of a Linux virtual machine (CentOS 6.5). The system received the
NMI, but didn't fail.
Figure 7: A Linux VM displaying the results of an NMI.

Unfortunately, a fully patched Windows 2008 R2 Server virtual machine wasn't as resilient. See
Figure 8.
Figure 8: Crashing a Windows virtual machine with an NMI signal.

VMware ESXi hosts provide the savvy administrator with 50 or so command line utilities for
advanced data gathering and troubleshooting. ESX hosts and virtual management appliances
(vMAs) boast a few more plus the full array of system utilities. You should familiarize yourself with
connecting directly to your host systems via SSH in case you ever have to contact VMware
Technical Support