Tuesday, September 24, 2013

vSphere Flash Read Cache (vFRC)

Flash Reads Cash...
Pretty awesome alterations and additions are implemented in VMware's latest vSphere release, vSphere 5.5. Amongst the additions are items like a newly revised and much easier to deploy version of SSO, VMware vSAN, improved vCloud services and a lot of other features. A couple stood out to me and one in particular, vSphere Flash Read Cache, was an awesome and much needed addition. I spent some time researching and got into the nitty gritty details of how it works and what someone using vFRC could expect.

vFRC is actually a hypervisor-like layer that sits on top of physical caching devices installed in ESXi hosts, devices like SSDs or PCI caching cards like FusionIO. Whether you have one caching device or mulitple, vFRC aggregates these devices and creates a cache pool that then can be allocated to Virtual Machines on the ESXi hosts (devices are not required to be placed in the cache pool). This in essence presents a situation similar to a pass-thru device and passes thru the caching capabilities to the VMs VMDKs (virtual hard disks).

The vFRC pooled resources are assigned on a per VMDK basis and each assignment is granular to the point of assigning a cache size space and cache block size per VMDK, this allows for portional assignment of cache resources per VMDK if you have limited caching hardware in your hosts. Some items to keep in mind:
  • Space allocated in the cache pool for VMDKs is thick provisioned and reserved, so allocate what you know your VM/VMDK will actively use.
  • Sizing your cache block size is extremely important, its obvious to say but size your cache blocks to match your workload block size. For instance if you were allocating a portion of the cache pool to a SQL Server you possibly be setting your cache block size to 64K.
  • vFRC is best used and the most performance increase is gained with data that is high re-read (i.e. databases) 
  • Know your workload, some performance monitoring and evaluation will go a long ways when trying to fairly and accurately size the cache amount allocated and block size.
  • Your cache block size affects your system memory too, vFRC keeps cache metadata in memory for quick access. The smaller the cache block size, the more metadata is read in and kept in memory.
  • By default VMs that have vFRC assigned to any of their VMDKs are set to vMotion cache with the system memory too. This will increase vMotion times but will keep the cache "warm" instead of evicting all cached data to simply rebuild once it moves. During warm ups, performance will potentially be degraded compared to a full warm cache.

Sizing your cache and cache block size is also important to acheive more hits (reads from cache), less misses (reads that had to go to disk because the data wasnt in cache) and less evictions (when data is pushed from cache to allow for other data to be read into cache). If you have your block size configured improperly, large blocks of data stored in 64K cache blocks could potentially be evicted so smaller 4 or 8K blocks blocks of data can take that space in cache. When a smaller data block fills a larger cache block the remainder of the space not used in the cache block by data is marked as invalid and cannot be used until the smaller amount of data is eventually evicted. This type fo scenario can kill performance and cause misses (bypassing cache to go to disk for data). During every of data if it is not found in cache, a miss occurs, and the data is then asyncronously read into cache then delivered to the requesting party.

Two points that are really cool: vFRC can (and most likely will) improve performance for VMs that are not actually utilizing vFRC, this occurs because vFRC is alleviating I/O requests from the storage system where other VMS share the storage, which allows them to find more "time" getting I/O requests filled. The other cool thing is the algorithm used to to evict data from cache. Some caching systems use a round robin effect and evict data on a systematic "You're next because your next in line" system, vFRC is intelligent enough (as are some other caching systems) to instead evict data based on a least used model, so data in cache that was accessed the least is evicted when space is needed.

Finishing up here, I recommend reading up on types of SSDs and Caching systems, SLC (Single Level-Cell) is faster than MLC (Multi Level-Cell), and PCIe based caching systems are normally faster than drive based SSDs. VMware recommends PCIe based caching cards with SLC flash and a deep queue depth, 256 or above. This is an awesome new feature to add to vSphere, I am all sorts of excited about it! Good caching all!

Friday, August 9, 2013

Veeam - Cannot Perform Threshold Check

I'll tell you one thing I hate and that's any kind of warning, alert, exclamation point, red triangle danger zone alert that shows up in the environments I work in. I get a little crazy and I HAVE to know what is causing it and get rid of them ASAP. Just like most admins, you want to see a clean, alert free, properly working environment so when I was helping a client relocate their Veeam backups to a new repository (off of a test location, and over to their permanent home) and start building out Veeam backups for their entire environment, I wanted it to be clean, done right and working properly. Everything seemed awesome, we had access to the backup share that became the new Veeam Backup repository and jobs were running successfully, UNTIL THE END! I instantly rent my clothes and smashed my laptop.....O.K. rewind, only one of those things happened, and it didn't result in a smashed laptop. I duct taped my clothes back together and gained some composure. 
Composed, I turned my attention to the report of the jobs. The jobs actually completed successfully, but because Veeam couldn't get a poll on the remaining space on the backup repository it was completing the jobs in a warning state, which drives me NUTS. Here is the warning I was getting: "Could not perform threshold check for backup location \\Backuplocation\Veeam due to space info retrievement fail!". 

The mice got on their wheels and I started to think about the what the possible issue could be, I instantly thought of permissions on the share, but that seemed like an odd conclusion because the share on the Backup Repository was actually wide open for admins. Turns out this is actually a known Veeam bug for CIF shares which is what I was using. Check this KB from Veeam for the details. The fix turned out to be a super simple task. Basically, this issue has been resolved in Veeam 6.5 patch 1, now this install of Veeam is only 40 days old and no patches have been applied, at the time of this writing patch 3 for Veeam 6.5 was available. Veeam states that all patches are cumulative, therefore I skipped 1 and 2 and jumped straight to patching Veeam with patch 3 (side note, patch 2 is mysteriously missing from the interwebs, conspiracy....for sure. I bet Veeam 6.5 patch 2 had an embedded version of Napster, the 1999 version. To bad I can't find patch 2 now). You can download patch 3 from here, look at the bottom of the KB for the download link, keep in mind you need a Veeam account to download it.

Installing the patch is super easy, download the patch, extract it from its archive and double click the executable. Make sure the Veeam console is closed or the install will fail. The install takes 30 seconds:

















After the install is complete, when you start the Veeam console you will be prompted to update the Veeam components installed on all proxy machines.















Simply select the servers you want to push the update out to and let Veeam do its thing. You will see the progress of the install as it pushes out the updates, unpacks them, installs them and then reports back. When all is done, you should see a screen nearly identical to this, although I superimposed (or did I....) the star to indicate what a good job we all did, so, good job!:



Sunday, July 28, 2013

Jumbo Frame MTU on vSphere Software iSCSI Adapters

In the book "Storage Implementation in vSphere 5.0" (SIIV5) from VMware Press, I ran across a really cool piece of information. It is regarding the MTU on port groups bound to Software iSCSI Adapters in vSphere environments. It has been a discussion I have had with a handful of people previously but in a slightly different sense. Previously when discussing this topic it was a conversation which focused on setting Jumbo frames (when using iSCSI software adapter) for a 1Gb network or leaving it at the standard 1500 MTU. I have generally heard that it does not improve network performance to use Jumbo Frames on a 1Gb connection, and if it does improve performance it is negligible and not worth it. What I found while reading SIIV5 is that myself and those whom I have had this conversation with before were possibly attributing the performance gain to the wrong portion of the environment. Where we were looking for networking performance improvements, SIIV5 actually pin points the performance gains on the ESXi host CPU level. As a review for some people and maybe new information for others here are a couple bullet points to quickly summarize the different categories of iSCSI initiators you can use with ESXi, this information is key in our discussion:
  1. Independent Hardware Initiator -  This initiator functions and operates on its own outside of any need to interface with ESXi. They offload all iSCSI and network processing from the host and onto the controller. They can be managed  through their firmware and in some cases through the vSphere UI.
  2. Dependent Hardware Initiator -  This initiator also can offload iSCSI and network processing from the host onto the controller, but dependent initiators depend on the ESXi host for  the network stack, configuration of the initiator and management through the commandline or the vSphere UI. Because it does has dependencies on the host, its offload capabilities are not an assumed function but is made possible through the use of TCP Offload Engine to move the processing of iSCSI and networking to the controller. Wiki Article about TOE. Requires that a vmkernel port group be created and bound to vmnic.
  3. Software Initiator - ESXi provides the software initiator as a component of the vmkernel. It requires ESXi to operate and can only be configured from either the commandline or the vSphere UI. Requires that a vmkernel port group be created and bound to vmnic.
It goes without saying that because the Software iSCSI adapter has no dedicated hardware, unlike an Independent or Dependent iSCSI adapter, they lack the off loading capabilities of non-software iSCSI adapters. This can potentially put more stress on the ESXi host's CPU as it has the need to handle every datagram, fragment payloads, and recompile payloads. Because of no dedicated hardware controller backing, VMware in SIIV5 recommends to always set the MTU on Software iSCSI port groups to 9000 (Jumbo Frames), this will improve performance by minimizing the load on the ESXi hosts. This improvement manifests itself as the CPU will no longer need to process so many datagrams as it begins working with larger network payloads. As stated in SIIV5:
"To compensate for lack of offloading capabilities of the iSCSI SW initiator, enabling Jumbo Frame can significantly I/O throughput." SIIV5 pg.140, "Configuring SW Initiator with Jumbo Frames"
The question about using Jumbo Frames for 1Gb networking of not no longer exists in my mind. When using a Software iSCSI Initiator in vSphere, Jumbo Frames is always the WAY-TO-GO! Now although I do not profess to be a storage or networking Guru, I hoped to pass on this information so that others who may be wondering about these topics have a good place to start learning!