vSphere Flash Read Cache (vFRC)

Flash Reads Cash...

Pretty awesome alterations and additions are implemented in VMware's latest vSphere release, vSphere 5.5. Amongst the additions are items like a newly revised and much easier to deploy version of SSO, VMware vSAN, improved vCloud services and a lot of other features. A couple stood out to me and one in particular, vSphere Flash Read Cache, was an awesome and much needed addition. I spent some time researching and got into the nitty gritty details of how it works and what someone using vFRC could expect.

vFRC is actually a hypervisor-like layer that sits on top of physical caching devices installed in ESXi hosts, devices like SSDs or PCI caching cards like FusionIO. Whether you have one caching device or mulitple, vFRC aggregates these devices and creates a cache pool that then can be allocated to Virtual Machines on the ESXi hosts (devices are not required to be placed in the cache pool). This in essence presents a situation similar to a pass-thru device and passes thru the caching capabilities to the VMs VMDKs (virtual hard disks).

The vFRC pooled resources are assigned on a per VMDK basis and each assignment is granular to the point of assigning a cache size space and cache block size per VMDK, this allows for portional assignment of cache resources per VMDK if you have limited caching hardware in your hosts. Some items to keep in mind:

• Space allocated in the cache pool for VMDKs is thick provisioned and reserved, so allocate what you know your VM/VMDK will actively use.
• Sizing your cache block size is extremely important, its obvious to say but size your cache blocks to match your workload block size. For instance if you were allocating a portion of the cache pool to a SQL Server you possibly be setting your cache block size to 64K.
• vFRC is best used and the most performance increase is gained with data that is high re-read (i.e. databases) 
• Know your workload, some performance monitoring and evaluation will go a long ways when trying to fairly and accurately size the cache amount allocated and block size.
• Your cache block size affects your system memory too, vFRC keeps cache metadata in memory for quick access. The smaller the cache block size, the more metadata is read in and kept in memory.
• By default VMs that have vFRC assigned to any of their VMDKs are set to vMotion cache with the system memory too. This will increase vMotion times but will keep the cache "warm" instead of evicting all cached data to simply rebuild once it moves. During warm ups, performance will potentially be degraded compared to a full warm cache.

Sizing your cache and cache block size is also important to acheive more hits (reads from cache), less misses (reads that had to go to disk because the data wasnt in cache) and less evictions (when data is pushed from cache to allow for other data to be read into cache). If you have your block size configured improperly, large blocks of data stored in 64K cache blocks could potentially be evicted so smaller 4 or 8K blocks blocks of data can take that space in cache. When a smaller data block fills a larger cache block the remainder of the space not used in the cache block by data is marked as invalid and cannot be used until the smaller amount of data is eventually evicted. This type fo scenario can kill performance and cause misses (bypassing cache to go to disk for data). During every of data if it is not found in cache, a miss occurs, and the data is then asyncronously read into cache then delivered to the requesting party.

Two points that are really cool: vFRC can (and most likely will) improve performance for VMs that are not actually utilizing vFRC, this occurs because vFRC is alleviating I/O requests from the storage system where other VMS share the storage, which allows them to find more "time" getting I/O requests filled. The other cool thing is the algorithm used to to evict data from cache. Some caching systems use a round robin effect and evict data on a systematic "You're next because your next in line" system, vFRC is intelligent enough (as are some other caching systems) to instead evict data based on a least used model, so data in cache that was accessed the least is evicted when space is needed.

Finishing up here, I recommend reading up on types of SSDs and Caching systems, SLC (Single Level-Cell) is faster than MLC (Multi Level-Cell), and PCIe based caching systems are normally faster than drive based SSDs. VMware recommends PCIe based caching cards with SLC flash and a deep queue depth, 256 or above. This is an awesome new feature to add to vSphere, I am all sorts of excited about it! Good caching all!

Veeam - Cannot Perform Threshold Check

I'll tell you one thing I hate and that's any kind of warning, alert, exclamation point, red triangle danger zone alert that shows up in the environments I work in. I get a little crazy and I HAVE to know what is causing it and get rid of them ASAP. Just like most admins, you want to see a clean, alert free, properly working environment so when I was helping a client relocate their Veeam backups to a new repository (off of a test location, and over to their permanent home) and start building out Veeam backups for their entire environment, I wanted it to be clean, done right and working properly. Everything seemed awesome, we had access to the backup share that became the new Veeam Backup repository and jobs were running successfully, UNTIL THE END! I instantly rent my clothes and smashed my laptop.....O.K. rewind, only one of those things happened, and it didn't result in a smashed laptop. I duct taped my clothes back together and gained some composure. 

Composed, I turned my attention to the report of the jobs. The jobs actually completed successfully, but because Veeam couldn't get a poll on the remaining space on the backup repository it was completing the jobs in a warning state, which drives me NUTS. Here is the warning I was getting: "Could not perform threshold check for backup location \\Backuplocation\Veeam due to space info retrievement fail!". 

The mice got on their wheels and I started to think about the what the possible issue could be, I instantly thought of permissions on the share, but that seemed like an odd conclusion because the share on the Backup Repository was actually wide open for admins. Turns out this is actually a known Veeam bug for CIF shares which is what I was using. Check this KB from Veeam for the details. The fix turned out to be a super simple task. Basically, this issue has been resolved in Veeam 6.5 patch 1, now this install of Veeam is only 40 days old and no patches have been applied, at the time of this writing patch 3 for Veeam 6.5 was available. Veeam states that all patches are cumulative, therefore I skipped 1 and 2 and jumped straight to patching Veeam with patch 3 (side note, patch 2 is mysteriously missing from the interwebs, conspiracy....for sure. I bet Veeam 6.5 patch 2 had an embedded version of Napster, the 1999 version. To bad I can't find patch 2 now). You can download patch 3 from here, look at the bottom of the KB for the download link, keep in mind you need a Veeam account to download it.

Installing the patch is super easy, download the patch, extract it from its archive and double click the executable. Make sure the Veeam console is closed or the install will fail. The install takes 30 seconds:

After the install is complete, when you start the Veeam console you will be prompted to update the Veeam components installed on all proxy machines.

Simply select the servers you want to push the update out to and let Veeam do its thing. You will see the progress of the install as it pushes out the updates, unpacks them, installs them and then reports back. When all is done, you should see a screen nearly identical to this, although I superimposed (or did I....) the star to indicate what a good job we all did, so, good job!:

Jumbo Frame MTU on vSphere Software iSCSI Adapters

In the book "Storage Implementation in vSphere 5.0" (SIIV5) from VMware Press, I ran across a really cool piece of information. It is regarding the MTU on port groups bound to Software iSCSI Adapters in vSphere environments. It has been a discussion I have had with a handful of people previously but in a slightly different sense. Previously when discussing this topic it was a conversation which focused on setting Jumbo frames (when using iSCSI software adapter) for a 1Gb network or leaving it at the standard 1500 MTU. I have generally heard that it does not improve network performance to use Jumbo Frames on a 1Gb connection, and if it does improve performance it is negligible and not worth it. What I found while reading SIIV5 is that myself and those whom I have had this conversation with before were possibly attributing the performance gain to the wrong portion of the environment. Where we were looking for networking performance improvements, SIIV5 actually pin points the performance gains on the ESXi host CPU level. As a review for some people and maybe new information for others here are a couple bullet points to quickly summarize the different categories of iSCSI initiators you can use with ESXi, this information is key in our discussion:

1. Independent Hardware Initiator -  This initiator functions and operates on its own outside of any need to interface with ESXi. They offload all iSCSI and network processing from the host and onto the controller. They can be managed  through their firmware and in some cases through the vSphere UI.
2. Dependent Hardware Initiator -  This initiator also can offload iSCSI and network processing from the host onto the controller, but dependent initiators depend on the ESXi host for  the network stack, configuration of the initiator and management through the commandline or the vSphere UI. Because it does has dependencies on the host, its offload capabilities are not an assumed function but is made possible through the use of TCP Offload Engine to move the processing of iSCSI and networking to the controller. Wiki Article about TOE. Requires that a vmkernel port group be created and bound to vmnic.
3. Software Initiator - ESXi provides the software initiator as a component of the vmkernel. It requires ESXi to operate and can only be configured from either the commandline or the vSphere UI. Requires that a vmkernel port group be created and bound to vmnic.

It goes without saying that because the Software iSCSI adapter has no dedicated hardware, unlike an Independent or Dependent iSCSI adapter, they lack the off loading capabilities of non-software iSCSI adapters. This can potentially put more stress on the ESXi host's CPU as it has the need to handle every datagram, fragment payloads, and recompile payloads. Because of no dedicated hardware controller backing, VMware in SIIV5 recommends to always set the MTU on Software iSCSI port groups to 9000 (Jumbo Frames), this will improve performance by minimizing the load on the ESXi hosts. This improvement manifests itself as the CPU will no longer need to process so many datagrams as it begins working with larger network payloads. As stated in SIIV5:

"To compensate for lack of offloading capabilities of the iSCSI SW initiator, enabling Jumbo Frame can significantly I/O throughput." SIIV5 pg.140, "Configuring SW Initiator with Jumbo Frames"

The question about using Jumbo Frames for 1Gb networking of not no longer exists in my mind. When using a Software iSCSI Initiator in vSphere, Jumbo Frames is always the WAY-TO-GO! Now although I do not profess to be a storage or networking Guru, I hoped to pass on this information so that others who may be wondering about these topics have a good place to start learning!

PowerCLI: ESXi Host Audit

I try to keep up with the VMware communities/message boards for PowerCLI related questions and saw one that I thought was interesting. Someone was interested in creating a script that could preform an audit of their ESXi hosts in inventory. He wanted the script to include the following:

• Product Name
• Version
• Edition
• Host Name
• Number of Cores
• Number of Physical Processors

So Mr. Buzzlightyear... if that is your real name, here it is, I know I am little late, but I think my script is a little easier to understand and use. In reality I turned the script actually into a function called: Get-VMHostAudit. Run this script, or add it to your PowershellCli profile and off you are to the races.

The script:

function Get-VMHostAudit

{

param([parameter(mandatory=$true)][string]$vCenter,

        [parameter(mandatory=$true)][string]$Username,

        [parameter(mandatory=$true)][string]$Password,

        [parameter(mandatory=$true)][ValidateSet("List","Table")]$Format)

Connect-VIServer -Server $vCenter -User $Username -Password $Password

$ServiceInstance = Get-View ServiceInstance

$LicenseManager = Get-View $ServiceInstance.Content.LicenseManager

$LicenseManagerAssign = Get-View $LicenseManager.LicenseAssignmentManager

$VMhosts=Get-VMHost

$VMhostsTotal=@()

Foreach($VMhost in $VMHosts)

    {

    $VMHostView=Get-VMHost -Name $VMHost.name | Get-View

    $VMhostID=$VMHostView.Config.Host.Value

    $VMHostLM=$LicenseManagerAssign.QueryAssignedLicenses($VMhostID) 

    $VMhostsTotal+=$VMHostView | Select Name,

        @{n='Product';e={$_.Config.Product.Name}},

        @{n='Version';e={$_.Config.Product.Version}},

        @{n='Sockets';e={$_.Hardware.CpuInfo.NumCpuPackages}},    

        @{n='CPUCores';e={$_.Hardware.CpuInfo.NumCpuCores}},

        @{n='LicenseVersion';e={$VMHostLM.AssignedLicense.Name | Select -Unique}},

        @{n='LicenseKey';e={$VMHostLM.AssignedLicense.LicenseKey | Select -Unique}}

    }

If ($Format -eq "List"){$VMhostsTotal | Fl}

If ($Format -eq "Table"){$VMhostsTotal | Ft -AutoSize}

Disconnect-VIServer -Server $vCenter -Confirm:$False -Force

} # End Function

How to run it:

Just run the script to load the function into your shell and then execute the function. Heres an example:

C:\Scripts\PowerCLI\VMHostAudit.ps1

Get-VMHostAudit -vCenter 192.168.0.1 -Username ITMonsterrrrrr -Password Cr@CkTh!s -Format Table

As I mentioned earlier, add the function to your profile also, then you will always have it when you want to run it. Here is my favorite resource for Powershell profiles. Here is an example of the output, you can choose from either listed form or table form, just hit tab after the format switch to cycle through your options.


Name           : 192.168.25.41
Product        : VMware ESXi
Version        : 5.0.0
Sockets        : 1
CPUCores       : 4
LicenseVersion : VMware vSphere 5 Enterprise Plus
LicenseKey     : 00000-00000-00000-00000-00000

Name           : 192.168.25.42
Product        : VMware ESXi
Version        : 5.0.0
Sockets        : 1
CPUCores       : 4
LicenseVersion : VMware vSphere 5 Enterprise Plus
LicenseKey     : 00000-00000-00000-00000-00000



Name          Product     Version Sockets CPUCores LicenseVersion                   LicenseKey                   
----          -------     ------- ------- -------- --------------                   ----------                   
192.168.25.41 VMware ESXi 5.0.0         1        4 VMware vSphere 5 Enterprise Plus 00000-00000-00000-00000-00000
192.168.25.42 VMware ESXi 5.0.0         1        4 VMware vSphere 5 Enterprise Plus 00000-00000-00000-00000-00000

How it works:

After reading the passed parameters of $vCenter, $Username, $Password and $Format, the script connects to vCenter. We grab a view of the vCenter instance its self, we then request a license manager view for the vCenter instance. After we have stored the license manager assignment view for the instance in the $LicenseManagerAssign variable we then grab all of the ESXi hosts for the instance in the $VMHost variable. Then we begin to cycle to through each host and do the following: We grab a view of the host, and assign its host ID to the $VMHostID variable which will be used in a second. We then query the License Manager for the assigned licenses for the ESXi host matching that $VMHostID. With that information stored in the $VMHostLM (licenses manager) variable, we then just compile all of the desired data. We pick up host name, product name, version, cpu cores, and sockets from the $VMhostView variable and add in the license and key by using calculated expressions to query the $VMHostLM variable. We actually add this data to the $VMHostsTotal array, which by doing this will allow all of the hosts to put their data into that array and presented at one time, instead of having a table for each host, we essentially create 1 table by adding each hosts data to that array. After it determines if you defined a list or table to be used, it spews the data and disconnects from vCenter! Thats all she wrote! You could easily add in functionality to export to a CSV or HTML, although with this information I dont see an HTML output being very handy, but I do see a lot of value for using CSV to archive this information. If you need help editing the code to add more functionality, just let me know!

Syntax View:

I use Powershell and Powershell ISE 3.0.

VCP5: Memory Reservations

Understanding entitlements ,and in this case memory reservations, is a key concept to pass the VCP5 exam, although this concept is easily grasped by most people, the exam presents questions regarding these subjects in ways that will have you thinking hard. Remember entitlements are not only configured in one location. Knowing these locations and the optional settings you could configure is need-to-know information. As some starting points, know what memory reservations, limits, and shares are and HOW THEY ARE DIFFERENT. Also, know where you can set memory entitlements. Below are a few screen shots to show you where you can set them:

All entitlements including disk and cpu can be set by right clicking any VM and navigating to the "Resources" tab:

Take note of the check box in the top right corner, "Reserve all guest memory (All locked)". This can be a super helpful component if you have a VM that is key to day to day operations and should always be guaranteed all of its memory, this is where you would configure that setting. Essentially, selecting this option removes the amount of RAM assigned to the VM from the pool of resources available to all other VMs, just as other reservations do for a specified amount of memory.

This next screenshot is the "Resources" page when you select to edit the settings of a vApp. If you dont know much about vApps I suggest you go here and read up on them in the vSphere Documentation Center, but basically a vApp is a container for housing VMs that are components of maybe a single application, like for instance, a web server, a database and an application server could be in a vApp for ease of management because they are all related to a certain application. Within a vApp you can power on and off the VMs in the vApp in a particular order in one click and you can control the resources of a vApp.

Here is a screen shot about memory entitlements in vSphere from the Documenation Center if you are looking for some quick reading.

SQL Saturday: #165 Lincoln

Had an awesome time at SQL Saturday #165 in Lincoln yesterday! Went down to Lincoln with a few co-workers and was able to see 3 co-workers from House of Brick Technologies present in different sessions. Great job David Pettengill, Andy Galbraith (@ DBA_ANDY) and David Klee (@kleegeek) on your presentations I enjoyed them all. This was my first SQL Saturday and it was a great experience, I love how the SQL Server community works together to help industry and technology progression.

A favorite presentation of mine yesterday was "SQL Server Internals: It's Whats for Dinner" by Christina Leo (@ChristinaLeo), it was very in depth presentation and very well pulled off. A theme of mine if you haven't read some of my other blog posts, is that I enjoy and appreciate simplicity, and her presentation was presented in a very simplistic easy to understand manner. When someone can explain a subject in such an easy to understand way, you know they know their stuff! She also used a pretty fun analogy comparing the internals of SQL Server to preparing and cooking a meal using a recipe...sounds nuts, but it was pretty awesome how she made it all relate. Great presentation Christina (who will probably never see my blog ;)).
Cant wait for SQL Saturday in Omaha April 6th!

VCP5: Resources and Refrences for Study

The VCP5 exam is a well written and structured test. It was a surprisingly difficult exam, and I was caught off guard. Based on mock exams and practice exams I was taking, as one portion of my preparation, I should have blown the test right out of the water. I feel like nothing I had done had properly helped me prepare for the exam so because of that fact I felt a little mad when I had failed the test on try 1. So in an effort to provide a better reference and study companion for people interested in taking the VCP5 exam, I plan to write a large group of blog posts containing key topics and areas of focus that will aid in helping people understand what they are studying and why they should know the information presented to pass the exam.
I took the VMware 5.0 Fast Track course in Chicago about 6 months ago (if you dont know, attending a VMware certified course is a requirement for becoming certified). I must warn everyone, taking the classes will not provide you with 90% of the knowledge you need to pass the exam as my teacher so commonly mentioned. I consider the classes to give a person about 20% of what they need to know to pass the exam. If the exam asked questions like "How do I power on a VM", then sure, the class would give you sufficient knowledge to pass. But I will tell you, you wont find little baby questions like that on the exam :)
What you WILL find in the VCP exam is that topics are covered in a way that someone might say to them selves, "Well that was a twist on what I thought the question might be". For instance, someone might study how to set the Maximum Transmission Unit to allow for Jumbo Frames. They might know from start to finish how to set the MTU, on what object you can set the it, and that 9000 is the number to configure for Jumbo Frames. But when they sit down for the test it may ask what is the default MTU, and that person may have spent so much time focused on the Jumbo Frame MTU size that they totally spaced on the default size. This to me would be a perfect example of how VMware successfully "twists" the questions of the exam to determine if the examinee has a deeper understanding of the material.
The main purpose of this post is to provide a list of references that I found helpful in my study for taking the exam. Sooooo, BOOM!, here's a bulletted list containing those references:

• My favorite reference for studying became the vSphere Documentation Center found here http://pubs.vmware.com/vsphere-50/index.jsp. A couple reasons why it was my favorite are that it is extensive and covers nearly ever topic, it is easy to understand, and most important is that it gives you steps to follow when troubleshooting common issues that a feature may have. The exam covers a lot of troubleshooting.
• The VMware Knowledge base. Found here, http://kb.vmware.com, the VMware KB is such an awesome library. My method of study was to find a topic and then physically go through the steps in a lab I have availability to at work. When I would run into an error or couldn't quite get something, the KB was invaluable.
• VMware vSphere Clustering Technical Deepdive by Duncan Epping and Frank Denneman was an awesome source for learning the internals of Clustering in vSphere.
• The VCP5 Exam Blue Print Found here: http://communities.vmware.com/servlet/JiveServlet/previewBody/16726-102-7-23055/VCP510%20Exam%20Blueprint%20Guide%201.4.pdf. I used this to know what I NEEDED to know, and then I would go about doing it in the lab and documenting what I did using screen shots.
• The Official VCP5 Certification Guide is a newly released book from VMware. If I had to summarize this book, I would say it is a more helpful version of the classes. I like how the author writes and enjoy his simplicity. The book comes with testing software which is nice, but very misleading because of how easy it is. The ebook version comes with even more testing software that is unfortunately misleadingly easy too.

I recommend having access to some degree of a vSphere lab, although I will not directly recommend using a production environment, if you only had access to that type of environment, create your self a read-only account so you can safely poke around and you wont have to be worried about accidentally hitting something you didnt want to! You do not necessarily need to be able to vMotion and enable FT, as long as you have done them before, but knowing where to find these features and know where to configure them coupled with the references above should be sufficient to be successful on the exam. Look for future posts where the title starts with VCP5, these will contain gold for studying. Good Luck!