1. VMware Performance Troubleshooting Presented by Chris Kranz

2. Topics Covered Introduction Root Cause Analysis Performance Characteristics CPU Networking Memory Disk Virtual Machine optimisation ESXTop vm-support Service Console Resource Groups Design Guidelines Capacity Planner limitations and cautions Conclusion Reference Articles

3. Introduction Multiple layers of virtualisation are used to increase service levels, availability and manageability However, multiple layers of virtualisation often mask performance and configuration issues making it more of a challenge to troubleshoot and correct The worst out come is that performance issues after a virtualisation project lead to the perception that VMware results in reduced performance and future confidence in VMware can be affected

4. Virtual Machine Resources CPU Memory Disk Networking Performance Basics

5. Resource Maximums http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

6. Typical Host Typical 3 VMs per core, 24VM’s per Host Each has 2GB of RAM = 48GB of RAM

7. Root Cause Analysis http://www.vmware.com/resources/techresources/10066

8. Root Cause ...

9. Do not rely on guest tools, but Can show high CPU, & Memory Utilisation Measurement of Latency & throughput of Disk & Network Interfaces Use the virtualisation layer, to diagnose cause: Guest is unaware of virtualisation workload The way in which guest OS’s account time is different No visibility of available resources Monitoring Performance

10. esxtop (service console only) resxtop (remote command line utilities) Performance graphs in vCentre Performance Analysis Tools

11. esxtop can be run: Interactively Batch (eg. esxtop -a -b > analysis.csv) Load batch into windows perfmon or MS Excel Two keys to remember H : help F : fields to display esxtop

12. esxtop basics Host Resources Name of Resource Pool, Virtual Machine or World Number of Worlds

13. Performance Characteristics CPU Networking Memory Disk Slow Processing High CPU Wait Packet Loss Slow Network Slow Processing Disk Swapping Log Stalls Disk Queue Slow Application Performance Reduced User Experience Data Loss and Corruption

14. CPU ESX Scheduler Basic World States Read / Run / Wait CPU States Ready / Usage / Wait Service Console Virtual Machine Limits / Shares / Reservations

16. %USED: Utilization

17. %RDY: Ready Time

18. %RUN: Run Time

20. CPU Further Investigation %MLMTD shows this VM has been limited

21. CPU Further Investigation High ready time caused by CPU resource limit

23. VMware Tools Balloon Driver to force the VM to swap to disk

25. Ballooning can be disabled (0 value) or controlled on a per Virtual Machine basis using: sched.mem.maxmemctl Default is set to 65%, can be controlled at host level. Only is an issue in resource contention scenarios. (or VM’s with low latency eg Citrix) Memory

27. Ballooned Memory

28. Shared Memory

30. Memory – Guest Overhead

31. Memory Virtual Machine Memory Metrics – VI Client

32. Memory Host Memory Metrics – VI Client

33. Memory PMEM: Total physical memory breakdown VMKMEM: Memory managed by vmkernel COSMEM: Service Console memory breakdown PSHARE: Page sharing statistics SWAP: Swap statistics MEMCTL: Balloon driver data esxtop

34. Memory esxtop / VI Client metrics : Virtual Machines

35. Memory esxtop / VI Client metrics : Host Usage

36. Memory VI Client memory usage graph

37. Memory Troubleshooting Memory usage issues

39. VLAN Trunking

40. Flow Control (full)

41. Speed & Duplex (1000Mb / Full)

42. Port Fast

43. BPDU Disabled

44. STP Disabled

45. Link State Tracking

46. Jumbo FramesNetwork configuration is more likely to blame than resource contention

47. Networking esxtop Transmit and Receive in Mb/s Transmit and Receive in Packets

48. Networking esxtop Dropped Packets Transmit Drop Packets Received

50. Disk subsystem configuration (SAN, NAS, iSCSI, local disk)

51. Disk caching

53. Queuing within the kernelK: Kernel D: Device G: Guest

55. Disk usage: sum of read BW and write BW (KB/s)

56. Disk read / write requests (per 20s interval)

57. Bus resets / Command aborts (per 20s interval)

60. Device Average / command (DAVG/cmd)

62. LUN Queue Length (LQLEN)

63. VMKernel (QUED)

64. Active Queue (ACTV)

66. Disk Desired Latency Calculations Desired Larency in msec <= (Outstanding IOs * Block size in KB) / Throughput per host Example: Number of Hosts: 16 Effective Link Bandwidth: 90 MBps Throughput per host: 90 / 16 = 5.6 MBps Desired Latency: (32 * 32) / (5.6) = 182.86 msec

67. Disk SAN Cache enabled High throughput VI Client SAN Cache disabled Poor throughput

68. Disk esxtop Latency is quite high After enabling cache, Latency is reduced

70. The disks MUST be block aligned to the storage (even when using NFS and SAN)

71. Where possible, always separate data disks from OS disks

72. Windows performance settings should be optimised for application performance

73. Guest operating system timeouts should be set as defined by the SAN vendor

74. Pagefileshould be separated where appropriate (this can impact VMware SRM however)

75. Unused Windows services should be disabled (wireless config, print spooler, audio, etc.)

76. Last access update time should be disabled (unless where required)

77. Logging of the VM should be disabled (only enabled for troubleshooting)

78. Remove any unused virtual hardware (floppy drives, USB, etc.)

79. Disable screen savers and power saving features, including logon screen saver

80. Enable Remote Desktop, avoid using the VI Client for remote administration

81. Install standard applications into template (bginfo, AntiVirus, any host agents, etc)

83. esxtop Command Options when inside esxtop

84. esxtop Command Line Options from the console

85. esxtop Expand the default window size for your session to get all statistics

87. contains the grub configuration

88. etc

89. contains the Console OS configuration files (cron, tcpwrappers, syslog, etc)

90. proc

91. contains much of the hardware configuration modules and variables

92. tmp

93. contains a lot of the ESX specific configuration output

94. var

95. contains log files and any core dumps

96. vmfs

97. contains the structure of the VMFS datastores

98. esx3-installation (where appropriate)

101. Increased Service Console memory – upto 800MB

102. Use host agents supplied by your vendors

103. Make storage recommended tweaks such as HBA Queue Depth and IO timeouts

104. Minimal use of the VI Client console – RDP or SSH instead

107. Standard configuration across all aspects (ESX, Storage, Networking, etc.)

108. Standard naming conventions

109. Learn from others mistakes

110. Follow guidelines from vendors best-practices

112. Back-end storage system performance

113. P2V machines will require block-aligning to the storage

115. Performance tools help diagnose issues and help rule out non-issues

116. Performance tools are useful in different contexts, not always either/or

117. Real-time data and troubleshooting: esxtop

118. Historical data: VI Client

119. Coarse resource / cluster usage: VI Client

120. Detailed resource usage: esxtop

121. Combine information from various tools to get a complete picture