Performance and Troubleshooting with esxtop
Contents
1. 975 SUIT VMWA RD ee Fields O VMs O vCPUs CPU load average 0 03 1 1 idle ero er o 131290 131290 esxtop 152217 1 2 76 25 644 Oo 97 20 a O 0e ial 5 helper TTY 0 54 0 54 Oo 7700 00 0 48 7O6 706 hostd 20e1 14 0 53 0 55 0 1396 62 a 0 84 958 956 wpxa 3015 16 0 36 0 35 Oo 1799 04 a 0 58 1003 10035 sh 3069 1 0 33 0 37 Oo 96 60 1 06 1054 1054 openwamand 3120 3 0 04 0 04 0 00 299 61 0 25 a 2 aystern 3 0 03 O 02 0 00 9300 00 0 03 mieie 566 debd z905 1 0 03 0 03 0 00 100 00 O 01 3 3 drivers 11 0 02 O 02 0 00 1100 00 0 02 aoe 992 Wrware usharbit 2 0 02 0 02 0 00 200 00 0 02 1265 12635 stfch ProviderMa 10 O 02 O 02 O 00 1000 00 0 00 7O9 709 wprohed 2744 a 0 01 O 01 O 00 300 00 0 00 671 671 storageRM e7O05 a 0 01 O 01 O 00 200 00 O 01 130605 130605 sshd 151411 1 o 00 O 06 0 00 99 95 O 01 640 640 husyhox 2674 1 0 dg 0 0 0 00 100 00 0 0d 641 641 busybox 26T5 1 0 0 00 0 00 100 00 0 1156 1156 deui 3226 1 0 dd 0 dd 0 00 100 00 0 0d 25303 55303 sh 99625 1 0 dd 0 dd 0 00 100 00 o 00 Figure 1 Esxtop outlining main statistics and showing location of fields Screen Views with the esxtop Utility The default view when esxtop is launched is going to show information for the CPU You can change the screen view by simply typing in a corresponding letter for the view that you are interested in inspecting Here is the list of views that you can switch to by simply typing in the let2. 099 78 7 0 02 1263 1263 efch ProviderMa 10 0 02 0 02 0 00 999 40 7 0 02 392 992 wmware usbarbit Z 0 02 0 02 0 00 199 88 7 0 00 2 2 syetem 0 02 0 02 0 00 899 81 7 0 05 564 564 debodez906 i 0 02 0 02 0 00 99 93 7 o o1 709 709 vprobed 2744 3 0 01 0 01 0 00 299 85 z 0 00 GTI 671 storageRM 2705 2 o o1 o o1 0 00 199 93 7 0 00 1261 1261 efeh ProviderMa Z 0 00 o oo 0 00 799 60 z 0 00 4262 1262 efch ProviderMa 7 0 00 0 00 0 00 699 57 7 0 00 1362 1362 sfcb ProviderMa 7 0 00 0 00 0 00 699 64 7 0 00 640 640 bueyhox 2674 i 0 00 0 00 o oo 99 98 7 0 00 641 641 busybox 2675 i 0 00 0 00 0 00 99 98 z 0 00 1033 1033 she3og9g 1 0 00 0 00 0 00 99 98 7 0 00 Figure 4 Displays the CPU screen with VMs running Mn a O E UL M M eee Copyright 2013 Global Knowledge Training LLC All rights reserved 6 Figure 4 shows CPU activity for the ESXi host and there are two VMs running on the system named second and w2k3vm In order to create contention on the CPU both VMs have a CPU affinity set for CPU 1 and are running a math application in a loop which is generating 99 busy If you look at the USED for both VMs they are both running at a little more than 49 since they are competing equally for the same PCPU Another interesting field that is used for performance monitoring when it comes to CPU issues is the RDY field The RDY field is the percentage of time that the world was ready to run but was waiting for its turn In the example above the two VMs sec
3. CPU Ready is a Delta which is the change from the previous interval As an example some counters are calculated as the delta between two successive snapshots or intervals The Used is a good example of a Delta Used Total CPU used time at the second snapshot Total CPU used time at the first snapshot time elapsed between snapshots To help understand the esxtop output it helps to define fields and counters that you are viewing World Is a schedulable entity ID World Identifier GID World Group Identifier NWLD Number of Worlds for an entity CPU Load Average is the mean of CPU loads in 1 minute 5 minute and 15 minutes base on 6 second samples Interpreting CPU Activity using Esxtop Utility Two VMs each VM has one vCPU CPU 1 99 busy 11 36 13pm up 2 50 254 worlds 2 VMs 2 yvCPUS CPU load average 1 04 PCPU USED 5 7 99 AVG 52 by POPU UTIL i 6 4 100 AVG 53 USED NAME NWLD a WAIT idle 2 O 94 23 96 35 0 00 0 00 103 61 T 30 second 5 49 54 49 93 0 02 396 88 0 00 52 97 wo VMs B y2 k3vm 5 49 43 49 54 0 02 396 80 0 00 53 08 running O esxtop 9044 1 2 15 2 41 0 00 97 38 0 16 named 6 hostd 2821 14 0 33 0 34 0 00 1399 22 0 21 secondand fs helper 73 0 31 0 32 0 00 7298 46 0 31 w k3vm 5 vpra 3015 18 0 20 0 20 0 12 1798 70 7 0 18 sahd 3040 i 0 12 0 14 o oo 99 79 7 o o1 openwsmand 3110 3 0 03 0 03 0 00 299 90 7 0 02 g 9 drivers ii 0 02 0 02 0 00 1
4. G Global Knowledge Expert Reference Series of White Papers Performance and Troubleshooting with esxtop 1 800 COURSES www globalknowledge com Performance and Troubleshooting with esxtop Steve Baca VCP VCI VCAP Global Knowledge Instructor Introduction This paper introduces and gives examples of how the esxtop utility can help address performance issues First we will discuss the history of esxtop and show several different methods that can be utilized to start the moni toring tool Next we will discuss how to use esxtop by using interactive commands that can be typed in while esxtop is running Finally we will look at how to use esxtop is given by looking at how to interpret CPU data utilizing the esxtop utility History The esxtop command is a tool based upon the old UNIX command line tool called top that continuously up dates every five seconds displaying a snapshot of the processes running on an ESXi host The top program has been around since the mid 1980s and has been ported to many different versions of UNIX and Linux Originally VMware ported a version of the UNIX top program and customized it to gather statistics for the ESX host the Standard top program was included in the service console as well When VMware changed the direction of its hypervisor and removed the service console esxtop continued to be a useable command line utility within the ESXi hypervisor which runs a proprietary version of UNIX
5. VMware also modified esxtop to run remotely and called it resxtop The remote resxtop runs within the vCLI and allows the user to remotely connect to an ESXi host and run esxtop esxtop resxtop The resxtop command is used when you want to run esxtop remotely from the vSphere command line inter face CLI using vCLI usually within the vMA The resxtop utility is referred to as remote esxtop and offers a secure method to run scripts across multiple ESXi hosts and virtual machines This paper concentrates on how to use esxtop since once resxtop is started all of the counters and fields are the same Copyright 2013 Global Knowledge Training LLC All rights reserved 2 Using esxtop in Batch Mode The esxtop command can also be run in batch mode which allows statistics to be collected and saved into a file then played back at a later point in time The data can be read using the Windows Perfmon utility or Microsoft Excel To start running esxtop in batch mode use the following syntax esxtop a b gt outputfile csv a show all of the statistics b stands for batch mode gt outputtile csv redirect the output to the file and the file must end with csv To stop processing in batch mode do Ctrl C Using esxtop in Interactive Mode By default esxtop runs in interactive mode which initially begins by typing in esxtop at the command line Depending on what system you are running on you might have to set the terminfo database to xte
6. d 1156 dcui z 3 48 1 05 0 0d 1 05 1 05 0 00 0 0d 611 vmkevrentd 2640 3 01 1 16 o 00 1 16 1 16 o 00 o 00 le6be stch Providerta a td 0 67 0 0d 0 67 0 67 0 00 0 00 671 atoragekM 2705 aa ta 1 36 0 00 1 36 1 36 0 0d 0 00 709 wprobed a744 2 56 0 74 0 0d 0 74 0 74 0 0d 0 0d 55383 sh 99625 Zagi 0 1z 0 00 0 1z Ode o 00 o 00 Figure 2 Default esxtop screen when first started Help Screen To learn more about other options you can choose type in h to get the help view for esxtop ri esHO1 netapp local PuTTY Iof x al Space update display hor help show this text cf quit Interactive commands are EF Add or remove fields oo Change the arder of displayed fields 3 set the delay in seconds between updates i set the number of instances to display W Write configuration file esxtops5ore k Fill a world Expand Rollup Cpu Statistics WV View only VWM instances L Change the length of the MAME field T Limit display to a aingle group oort hy 0 3 USED R sRDY W GIn ovitch display cicpu i interrupt ri i IWeMoLr y ninetwork d disk adapter u disk device vidisk VHM Pi power mgmt Hit any key to continue Figure 3 Displays the help screen interactive commands Copyright 2013 Global Knowledge Training LLC All rights reserved 5 Calculating Performance Counters The performance counters are calculated in different ways The counters or statistics types can be a Rate Del ta or Absolute value
7. is high the VM cannot run because of the CPU limit setting SYS The percentage of time spent on the ESXi VMKernel running process interrupts and other system ser vices on behalf of the world IDLE The percentage of time the vCPU world is in an idle loop CSTP The percentage of time the vCPUs of a VM are spent in the co stopped state waiting to be co started Copyright 2013 Global Knowledge Training LLC All rights reserved 7 SWPWT The percentage of time the world is waiting for the ESXi s VMKernel to swap memory If SW PWT is high then the VM is swapping memory RUN The percentage of total scheduled time for the world to run If RUN of a VM is high the VM is using lots of CPU resources but does not necessarily mean the VM is under resource constraint WAIT The percentage of time the world spent in the wait or idle state This WAIT is the total wait time the world is waiting for some VMKernel resource The WAIT time can be high because there are many worlds waiting for events to happen and the total wait time can be high dude to the large number of worlds waiting on events Summary The esxtop utility provides detailed performance data for an ESXi host This real time data gives the system administrator information that aids in detecting performance issues To better interpret esxtop data it helps to understand how to setup the esxtop view with the appropriate fields When dealing with CPU perfo
8. ond and w2k3vm have a RDY time a little greater than 50 which is extremely high Nor mally become concerned if see a steady value greater than 10 If the RDY is greater than 10 would look to see if YMLMTD is high as well If YMLMTD is high it would signify that a CPU Limit has been set on the VM and needs to be investigated In addition there is a field called WAIT that shows wait and idling time together CPU Statistics PCPU USED CPU utilization per physical CPU includes logical CPUs USED CPU Utilitzation The percentage physical CPU time accounted to the world The formula is USED RUN SYS OVERLP It is possible that the USED of a world can be greater than 100 if the system service runs on a different PCPU for this world If the USED of a VM is high that means the VM is using lots of CPU resources which can be normal RDY The percentage of time the world was ready to run but was not provided the CPU resources A world in a run queue is waiting for the CPU scheduler to let it run on a PCPU If RDY of a VM is high it means the VM is possibly under resource contention Check MLMTD as well If Y MLMTD is high you may raise the CPU Limit setting for the VM If RDY MLMTD is high the VM is under CPU contention MLMTD The percentage of time the world was ready to run but deliberately was not scheduled because that would violate the CPU Limit setting What does It mean if MTMLD of a VM
9. rm export TERM xterm esxtop Once you launch esxtop you will see a default screen Figure 1 included callout descriptions to some of the main host statistics and fields The esxtop output can show more information than you will need for the per formance or troubleshooting problem that you are addressing There are also interactive commands that can be issued to customize the display which will be shown in Figure 3 Figure 1 is an example of the output generated from esxtop or resxtop There are several screens that can be viewed The default screen is always the CPU view as shown in the screen shot Figure 1 and the screen refreshes every five seconds by default The esxtop displays statistics based on worlds A world can be defined as schedulable entity and other operating systems would call it a process Each virtual machine will have multiple worlds running based on several factors There will be one world for each of the vCPUs running on the VM There will be a world for the VM s MKS and a world for the virtual machine monitor VMM of the world Copyright 2013 Global Knowledge Training LLC All rights reserved 3 fero VMs CPU load over last one five and fifteen minutes uptime of rre ar time on ESXi tan BSH etapp TOPA FUTIT pn 05 30am up 2 days 17 125 250 worl 250 worlds O VM PCPU USED 9 9 6 2 AVG 8 0 BPCPUSUTIL S 10 7 0 AVG 8 8 _ Statistics Th GID MAME ULE STSED RUN
10. rmance problems for a VM one of the first fields to observe is RDY If this field is larger than 10 it could mean that you have more requests for CPU processing than resources available Thus RDY time is the best indicator of possible CPU performance issues Learn More To learn more about how you can improve productivity enhance efficiency and sharpen your competitive edge Global Knowledge suggests the following courses VMware vSphere Fast Track V5 1 VMware vSphere Optimize and Scale V5 1 Visit www globalknowledge com or call 1 800 COURSES 1 800 268 7737 to speak with a Global Knowledge training advisor About the Author Steve Baca has been working in the Information Technology field for more than 15 years after graduating from the University of Nebraska with a Bachelors degree in Computer Science and Mathematics After spending time programming and doing Systems Administration Steve has been doing technical training for VMware Netapp Sun Microsystems and Symantec Copyright 2013 Global Knowledge Training LLC All rights reserved 8
11. ter associated with the view e c CPU view which is the default view e m Memory view e n Network view e d Disk adapter view e u Disk device view e v Disk VM view J i Interrupts e p Power management For example if you want to switch from looking at the CPU view information to looking at the memory view simply type in the letter m to make the switch Figure 2 shows the memory view Copyright 2013 Global Knowledge Training LLC All rights reserved 4 Fi esxO1 netapp local PuTTY Oy x 2 06 14am up 3 days 16 26 249 worlds O VNs O vCPUs MEM overcommit avg 0 00 0 00 0 00 al PMEM MB 6143 total BOS wink 103 other 5234 free VMEMEM ME 6115 managed 326 mintiree 2112 ravd 4002 ursvd high state PSHARE MB 16 shared 16 common 0 saving SWAP MB E Curr D relmtgt 0 00 r s 0 00 ws ZIP MB O zipped o saved MEMNCTL MB Oo Curr O target O max MEMS Zz GRANT ae lisT TCHD TLHD W SCOR a TT 7O6 hostd lt 621 54 77 44 41 0 00 44 41 trti 0 00 0 00 O56 wpxa 3015 a l4 14 49 0 00 14 49 14 49 0 00 0 00 le61 afchb ProwiderMa 14 50 12 39 0 00 12 39 12 39 0 00 0 00 le635 stch Providerta 10 2 0 6 79 o 00 6 79 6 79 0 00 0 00 332 wobd 2975 5 34 1 23 0 0d 1 23 Teza 0 00 0 0d 604 wraeyslogd 2620 6 66 4 53 0 0d 4 53 4 53 0 0d 0 0d 603 wmreyslogd 2619 4 62 Jael 0 0d Jarl Jal 0 0d 0 0d 1364 afchb ProwiderMa a 4 1 50 o 00 1 40 1 50 0 00 o 00 1257 stch Providerta a 60 er ge o 00 er ge Ieva 0 00 0 0
Download Pdf Manuals
Related Search