PowerCLI: ESXi Host Audit

I try to keep up with the VMware communities/message boards for PowerCLI related questions and saw one that I thought was interesting. Someone was interested in creating a script that could preform an audit of their ESXi hosts in inventory. He wanted the script to include the following:

• Product Name
• Version
• Edition
• Host Name
• Number of Cores
• Number of Physical Processors

So Mr. Buzzlightyear... if that is your real name, here it is, I know I am little late, but I think my script is a little easier to understand and use. In reality I turned the script actually into a function called: Get-VMHostAudit. Run this script, or add it to your PowershellCli profile and off you are to the races.

The script:

function Get-VMHostAudit

{

param([parameter(mandatory=$true)][string]$vCenter,

        [parameter(mandatory=$true)][string]$Username,

        [parameter(mandatory=$true)][string]$Password,

        [parameter(mandatory=$true)][ValidateSet("List","Table")]$Format)

Connect-VIServer -Server $vCenter -User $Username -Password $Password

$ServiceInstance = Get-View ServiceInstance

$LicenseManager = Get-View $ServiceInstance.Content.LicenseManager

$LicenseManagerAssign = Get-View $LicenseManager.LicenseAssignmentManager

$VMhosts=Get-VMHost

$VMhostsTotal=@()

Foreach($VMhost in $VMHosts)

    {

    $VMHostView=Get-VMHost -Name $VMHost.name | Get-View

    $VMhostID=$VMHostView.Config.Host.Value

    $VMHostLM=$LicenseManagerAssign.QueryAssignedLicenses($VMhostID) 

    $VMhostsTotal+=$VMHostView | Select Name,

        @{n='Product';e={$_.Config.Product.Name}},

        @{n='Version';e={$_.Config.Product.Version}},

        @{n='Sockets';e={$_.Hardware.CpuInfo.NumCpuPackages}},    

        @{n='CPUCores';e={$_.Hardware.CpuInfo.NumCpuCores}},

        @{n='LicenseVersion';e={$VMHostLM.AssignedLicense.Name | Select -Unique}},

        @{n='LicenseKey';e={$VMHostLM.AssignedLicense.LicenseKey | Select -Unique}}

    }

If ($Format -eq "List"){$VMhostsTotal | Fl}

If ($Format -eq "Table"){$VMhostsTotal | Ft -AutoSize}

Disconnect-VIServer -Server $vCenter -Confirm:$False -Force

} # End Function

How to run it:

Just run the script to load the function into your shell and then execute the function. Heres an example:

C:\Scripts\PowerCLI\VMHostAudit.ps1

Get-VMHostAudit -vCenter 192.168.0.1 -Username ITMonsterrrrrr -Password Cr@CkTh!s -Format Table

As I mentioned earlier, add the function to your profile also, then you will always have it when you want to run it. Here is my favorite resource for Powershell profiles. Here is an example of the output, you can choose from either listed form or table form, just hit tab after the format switch to cycle through your options.


Name           : 192.168.25.41
Product        : VMware ESXi
Version        : 5.0.0
Sockets        : 1
CPUCores       : 4
LicenseVersion : VMware vSphere 5 Enterprise Plus
LicenseKey     : 00000-00000-00000-00000-00000

Name           : 192.168.25.42
Product        : VMware ESXi
Version        : 5.0.0
Sockets        : 1
CPUCores       : 4
LicenseVersion : VMware vSphere 5 Enterprise Plus
LicenseKey     : 00000-00000-00000-00000-00000



Name          Product     Version Sockets CPUCores LicenseVersion                   LicenseKey                   
----          -------     ------- ------- -------- --------------                   ----------                   
192.168.25.41 VMware ESXi 5.0.0         1        4 VMware vSphere 5 Enterprise Plus 00000-00000-00000-00000-00000
192.168.25.42 VMware ESXi 5.0.0         1        4 VMware vSphere 5 Enterprise Plus 00000-00000-00000-00000-00000

How it works:

After reading the passed parameters of $vCenter, $Username, $Password and $Format, the script connects to vCenter. We grab a view of the vCenter instance its self, we then request a license manager view for the vCenter instance. After we have stored the license manager assignment view for the instance in the $LicenseManagerAssign variable we then grab all of the ESXi hosts for the instance in the $VMHost variable. Then we begin to cycle to through each host and do the following: We grab a view of the host, and assign its host ID to the $VMHostID variable which will be used in a second. We then query the License Manager for the assigned licenses for the ESXi host matching that $VMHostID. With that information stored in the $VMHostLM (licenses manager) variable, we then just compile all of the desired data. We pick up host name, product name, version, cpu cores, and sockets from the $VMhostView variable and add in the license and key by using calculated expressions to query the $VMHostLM variable. We actually add this data to the $VMHostsTotal array, which by doing this will allow all of the hosts to put their data into that array and presented at one time, instead of having a table for each host, we essentially create 1 table by adding each hosts data to that array. After it determines if you defined a list or table to be used, it spews the data and disconnects from vCenter! Thats all she wrote! You could easily add in functionality to export to a CSV or HTML, although with this information I dont see an HTML output being very handy, but I do see a lot of value for using CSV to archive this information. If you need help editing the code to add more functionality, just let me know!

Syntax View:

I use Powershell and Powershell ISE 3.0.

VCP5: Memory Reservations

Understanding entitlements ,and in this case memory reservations, is a key concept to pass the VCP5 exam, although this concept is easily grasped by most people, the exam presents questions regarding these subjects in ways that will have you thinking hard. Remember entitlements are not only configured in one location. Knowing these locations and the optional settings you could configure is need-to-know information. As some starting points, know what memory reservations, limits, and shares are and HOW THEY ARE DIFFERENT. Also, know where you can set memory entitlements. Below are a few screen shots to show you where you can set them:

All entitlements including disk and cpu can be set by right clicking any VM and navigating to the "Resources" tab:

Take note of the check box in the top right corner, "Reserve all guest memory (All locked)". This can be a super helpful component if you have a VM that is key to day to day operations and should always be guaranteed all of its memory, this is where you would configure that setting. Essentially, selecting this option removes the amount of RAM assigned to the VM from the pool of resources available to all other VMs, just as other reservations do for a specified amount of memory.

This next screenshot is the "Resources" page when you select to edit the settings of a vApp. If you dont know much about vApps I suggest you go here and read up on them in the vSphere Documentation Center, but basically a vApp is a container for housing VMs that are components of maybe a single application, like for instance, a web server, a database and an application server could be in a vApp for ease of management because they are all related to a certain application. Within a vApp you can power on and off the VMs in the vApp in a particular order in one click and you can control the resources of a vApp.

Here is a screen shot about memory entitlements in vSphere from the Documenation Center if you are looking for some quick reading.

SQL Saturday: #165 Lincoln

Had an awesome time at SQL Saturday #165 in Lincoln yesterday! Went down to Lincoln with a few co-workers and was able to see 3 co-workers from House of Brick Technologies present in different sessions. Great job David Pettengill, Andy Galbraith (@ DBA_ANDY) and David Klee (@kleegeek) on your presentations I enjoyed them all. This was my first SQL Saturday and it was a great experience, I love how the SQL Server community works together to help industry and technology progression.

A favorite presentation of mine yesterday was "SQL Server Internals: It's Whats for Dinner" by Christina Leo (@ChristinaLeo), it was very in depth presentation and very well pulled off. A theme of mine if you haven't read some of my other blog posts, is that I enjoy and appreciate simplicity, and her presentation was presented in a very simplistic easy to understand manner. When someone can explain a subject in such an easy to understand way, you know they know their stuff! She also used a pretty fun analogy comparing the internals of SQL Server to preparing and cooking a meal using a recipe...sounds nuts, but it was pretty awesome how she made it all relate. Great presentation Christina (who will probably never see my blog ;)).
Cant wait for SQL Saturday in Omaha April 6th!

VCP5: Resources and Refrences for Study

The VCP5 exam is a well written and structured test. It was a surprisingly difficult exam, and I was caught off guard. Based on mock exams and practice exams I was taking, as one portion of my preparation, I should have blown the test right out of the water. I feel like nothing I had done had properly helped me prepare for the exam so because of that fact I felt a little mad when I had failed the test on try 1. So in an effort to provide a better reference and study companion for people interested in taking the VCP5 exam, I plan to write a large group of blog posts containing key topics and areas of focus that will aid in helping people understand what they are studying and why they should know the information presented to pass the exam.
I took the VMware 5.0 Fast Track course in Chicago about 6 months ago (if you dont know, attending a VMware certified course is a requirement for becoming certified). I must warn everyone, taking the classes will not provide you with 90% of the knowledge you need to pass the exam as my teacher so commonly mentioned. I consider the classes to give a person about 20% of what they need to know to pass the exam. If the exam asked questions like "How do I power on a VM", then sure, the class would give you sufficient knowledge to pass. But I will tell you, you wont find little baby questions like that on the exam :)
What you WILL find in the VCP exam is that topics are covered in a way that someone might say to them selves, "Well that was a twist on what I thought the question might be". For instance, someone might study how to set the Maximum Transmission Unit to allow for Jumbo Frames. They might know from start to finish how to set the MTU, on what object you can set the it, and that 9000 is the number to configure for Jumbo Frames. But when they sit down for the test it may ask what is the default MTU, and that person may have spent so much time focused on the Jumbo Frame MTU size that they totally spaced on the default size. This to me would be a perfect example of how VMware successfully "twists" the questions of the exam to determine if the examinee has a deeper understanding of the material.
The main purpose of this post is to provide a list of references that I found helpful in my study for taking the exam. Sooooo, BOOM!, here's a bulletted list containing those references:

• My favorite reference for studying became the vSphere Documentation Center found here http://pubs.vmware.com/vsphere-50/index.jsp. A couple reasons why it was my favorite are that it is extensive and covers nearly ever topic, it is easy to understand, and most important is that it gives you steps to follow when troubleshooting common issues that a feature may have. The exam covers a lot of troubleshooting.
• The VMware Knowledge base. Found here, http://kb.vmware.com, the VMware KB is such an awesome library. My method of study was to find a topic and then physically go through the steps in a lab I have availability to at work. When I would run into an error or couldn't quite get something, the KB was invaluable.
• VMware vSphere Clustering Technical Deepdive by Duncan Epping and Frank Denneman was an awesome source for learning the internals of Clustering in vSphere.
• The VCP5 Exam Blue Print Found here: http://communities.vmware.com/servlet/JiveServlet/previewBody/16726-102-7-23055/VCP510%20Exam%20Blueprint%20Guide%201.4.pdf. I used this to know what I NEEDED to know, and then I would go about doing it in the lab and documenting what I did using screen shots.
• The Official VCP5 Certification Guide is a newly released book from VMware. If I had to summarize this book, I would say it is a more helpful version of the classes. I like how the author writes and enjoy his simplicity. The book comes with testing software which is nice, but very misleading because of how easy it is. The ebook version comes with even more testing software that is unfortunately misleadingly easy too.

I recommend having access to some degree of a vSphere lab, although I will not directly recommend using a production environment, if you only had access to that type of environment, create your self a read-only account so you can safely poke around and you wont have to be worried about accidentally hitting something you didnt want to! You do not necessarily need to be able to vMotion and enable FT, as long as you have done them before, but knowing where to find these features and know where to configure them coupled with the references above should be sufficient to be successful on the exam. Look for future posts where the title starts with VCP5, these will contain gold for studying. Good Luck!

VMware Certified Professional 5

It has been awhile since I composed a new blog post, there was a really good reason for it...I swear! I had been studying for quite awhile to take the VCP5 exam. If you have never attempted this test, be forewarned, it is extremely hard. I had studied super hard, I had even bought the new Official VCP5 Certification Guide published by VMware and blew through the book in a week, and felt very comfortable with my knowledge of the subjects found in it. The book comes with testing software and if you buy the "Special" edition eBook it comes with more advanced testing software than just the book comes with. I was blowing those mock exams, provided in the testing software, out of the water. I had been scoring 480 out of 500, and taking only 30 minutes to do it too. I thought I was set! I signed up and took the exam and I was metaphorically pants'd. My first attempt was a 275 out of 500, needing 300 to pass. I was surprised at the difficulty of the test, they truly want you to have a deep knowledge of the material and troubleshooting methods. I felt pretty frustrated because a lit if hard work had been done, but I knew I needed to change my preparation methods. The key for me was doing more hands on versus just reading more. I signed up for the next soonest exam date I could (you have to wait 7 days), and through a lot of hard work and help from my wife and co workers I passed the exam on 20th. I plan to share my studying techniques and, recommend in detail areas of study in my next few posts while it is fresh in my mind. Thanks to everyone who helped me get to this awesome milestone in my career, thanks to all of those co-workers at House of Brick Technologies who helped me out, answered questions when I had them, and was supportive of me taking on this exam, you know who you are! More to come on prepping for the exam! WOO HOO

Deep Dive: CPU Ready Time

In my last post we tackled the topic, "VMWare ESX/ESXi CPU Scheduler",with extreme prejudice, this week will be an extension of that post. Lets talk about CPU Ready Time, how bout' it? (if you said "No", Click Here...). To put it simply: CPU Ready time is your enemy, but, to give a more formal definition of what CPU Ready Time: CPU Ready Time is when a Virtual Machine exists in a state when it is ready for processor time but cannot be scheduled on the processor. The CPU is at a point when it is "Ready" for processor time. I mentioned in my last post that CPU Ready time can be a performance killer and over allocation of vCPUs is most likely at the heart of the problem. To throw some data and thoughts at you really quick, I love me a good bulleted list and I hope you do too:

• When possible, keep like CPU allocated VMs on the same hosts or clusters. This will help reduce CPU Ready time to a safe amount. VMware suggests 5% CPU Ready time is within the safe limits for a VM to live with. We will discuss safe zones a little later in the post. Regardless of recommendations, everyone should establish their own safe and red zone numbers. Even ours don't coincide with VMware's recommendation.
• An ESXi host will try its best to keep a VM's requests for processor time, running on the same physical cores for caching purposes. If the benefits of moving the VM's processors requests to new cores is more efficient than waiting for the CPU scheduling to allow for processor time (CPU Ready Time), the ESXi host will clear the processor cache and move the existing and new processor requests to different cores, this move takes time and the effects can be felt in performance but the host has determined this to be the better choice than to let it accumulate any more CPU Ready time.
• Many times CPU over allocation occurs when P2V'ing a physical machine into your Virtual environment. Although your super serious amazing awesome database server needed 12 CPUs in the physical world, monitor overtime and determine how many of those you can knock out of the picture. Maybe it could be taken down to 8 or even 6 vCPUs. 
• In many cases, I would bet that 60-75% of workloads can fit within the 1-2 vCPU range, and I would also put money on 75-90% of workloads needing no more than 4 vCPUs. As a side note: I was reading this article on Ars Technica and found some of the hypervisor host stats for Hyper-V compared against VMware, pretty crazy and honestly, unnecessary. By the time I need 320 CPU hosts (yay Hyper-V), I would have hoped I would have been smary enough and separated that load into multiple hosts and created maybe a new cluster for high availability instead. I don't doubt there are some massive companies with an insane virtual infrastructure that could benefit from that size of hosts, but most companies and workloads don't come close to that. END OF SIDE RANT.
• I recommend keeping a close eye, however you go about doing that in your environment, on the CPU utilization and CPU Ready Time, and always look for opportunities to reduce vCPU assignment or create like allocated vCPU Virtual Machines.
• If you have hosts or clusters that have a huge amount of resources that are barely getting used by the VMs in your environment, you may be able to get away with not caring to much about mixing like vCPU assigned VMs. The effects of CPU Ready time are being masked by the fact that there are a bunch of physical resources. Wait for your work load get up to a medium amount of usage and your performance will start to ache because of that hidden CPU Ready time. 

With my awesome bullet-ed (spell checker wanted a hyphen, lame!) list aside, lets take a look at what in the world was going on in the "cliff hanger" picture I left at the bottom of my last post. Here is it once more:

So what are we looking at here? This is obviously a vSphere performace chart, it is showing the CPU Usage and CPU Ready time for a production VMware View Connection Server. To make a long story as short as I can, I was looking at the currently assigned Memory for this machine when I noticed the VM had been assigned 4 vCPUS.

I thought to my self, "Why in the world does this VM need 4 vCPUs?" Turns out after reading some additional View documentation, 4 vCPUs is the default recommended setting. So anyways, I thought just for fun I would take a look at the CPU usage to see how much this baby was actually using  Here is what I initially saw once I enabled the CPU Ready chart metric:

This VM was getting CPU Ready times on the "Real-time" chart of 2,500 to 3,700 milliseconds consistently . Its one thing to get some bad CPU Ready time in a blip or moment, but this was consistent and "reliable" CPU Ready time. This VM's performance was basically getting destroyed. To put these numbers into perspective lets take a look at how to interpret these millisecond numbers. I recommend learning and understanding what good and bad millisecond values are for CPU Ready Time, since this is how the vSphere performance charts presents this metric. Many people want % values, which are not bad, but because vSphere charting goes off of millisecond, I find it more beneficial to read the millisecond value and know what they mean.

If you know anything about how VMware saves statistical data this wont be news to you. But when you look at any performance chart BUT the "Real-time" view or the "Refreshes every 20 seconds view", the numbers you see present you with what looks like "higher" numbers. This seems this way because the stats are rolled up and its actually an average of the statistics saved.

To further describe roll-up effect, here are some basics to work off of.

• The "Real-time" performance chart is updated every 20 seconds. As a good base line, 120 - 175ms is acceptable for CPU Ready Time on the "Real-time" chart. 500 ms is our red zone. For those of you who want percentages here is the equation: 
• <currentValueINMilliseconds>/<intervalINMilliseconds>*100
• For example 170/20,000*100 which gives us a percentage of .8% CPU Ready time. 
• Doing the math, so you dont have to, our red zone is about 2.5%.
• The "Day" performance chart shows the rolled up statistics for a day in 5 minute intervals. So if our Real-time interval is 20 seconds and 500ms is the red zone, our 5 minute interval red zone would be 7500ms. 
• (5*60/20)*500=7500.  OR to simplify. From here on out our red zone is 1500ms for every minute.
• The "Week" performance chart shows the rolled up stats for a week in 30 minutes intervals. So 30*1500=45000 is the red zone for the week chart.

Although I could do the math, because I am wicked smart, I will let you all do the rest of the math for the Month and Year charts and for any custom charts you make. Make sure you check your stats intervals on your vCenter(s) as the defaults are not always used. Access the interval settings this from the "vCenters Settings" button on the home page inside of vCenter. Heres a screen shotters for assistance:



So let me finish the story from earlier about the View Connection Server I was monitoring. To sum things up really quickly, the "Real-Time" chart screen shot I shared earlier with 2,500 to 3,700ms means that for 16% of the time this VM was not able to do anything, it was just sitting there and waiting! When would numbers like that ever be acceptable?

INITIATE SCIENTIFIC STUDY BENCHMARKS

Advanced Scientific Study #1:
IT Manager: Why is this VM so dang slow?
IT Employee: Uhhhh, well it works 84% of the time...
Outcome: NOT ACCEPTABLE

Advanced Scientific Study #2:
PotentialCustomer: What kind of guarntee can we expect for up time if we decide to go with your facility?
PotentialDatacenterFacility: Uhhhh, we are up about 84% of the time around here...
Outcome: NOT ACCEPTABLE

Advanced Scientific Study #3:
Wife: How much do you love me?
Husband: Uhhhh, about 84%...
Outcome: DIVORCED

To finish up here (this post is SOOOO LONG) I asked for a downtime window from the customer, and went in and removed 2 vCPUs. The end result of removing those 2 vCPUs is what is shown in the initial screen shot highlighted by the "BOOM!". The CPU Ready time drops off and is almost non existent. One last picture, here is a screenshot showing the "Real-time" view with the CPU Ready metric active. This screenshot was taken about 30 minutes after I changed the vCPU assignment. Check out the dive CPU Ready time took, AWESOME.

I hope that this post has been a helpful tool in helping you understand CPU Ready time more fully. I hope the real world scenario described in the post helps tie it all together. Be a Monsterrrrrr at work, destroy CPU Ready time.

Deep Dive: VMWare ESX/ESXi CPU Scheduler

Reviewing the concepts covered in Larry Loucks book "Critical VMware Mistakes you Should Avoid", we will first look at the CPU Scheduler. Larry's book does a great job of giving an overview of the CPU scheduler and core concepts of how it works, I hope this post builds on top of the core concepts and I hope this post feels more like a deep dive than a general overview, soak it up you nautical beast.

The VMware CPU scheduler is no hidden secret and many people including you reading this post probably know what it is and how it works. That aside I love learning about these kind of topics, and as I have said in previous posts, these topics to me are what make IT interesting. This first post will be a relatively high level over view of how ESX (referring to both ESX and ESXi) schedules CPU, in later posts we will dive a little deeper. Lets get to it!

Both the CPU Scheduler and Hypervisor (viewing them temporarily as separate entities) play a big part in CPU scheduling. The CPU scheduling is obviously done by the hypervisor, but and in lamens terms the CPU scheduler's role is to present and assign world (think VM) executions to the physical processors keeping within the constraints of acceptable performance for the VMs while prioritizing processor requests. While the CPU scheduler determines which world (think VM) to schedule, the hypervisor holds the task of presenting the physical CPUs to each VM( here is a duh moment when thinking about hypervisors but just to say it) as if they each own the physical CPUs. Lets figure out how the CPU scheduler does its job of prioritizing processor requests.

If you are any bit familiar with vSphere you most likely understand the concepts of shares, reservations and limits in regards to physical resources, these are the user-set configuration values for each VM within a vSphere Datacenter. For reviews sake, lets take a look at them very briefly:

• Shares: Shares are simply used to prioritize resources for use by VMs when there is active resource contention. If you give a VM 1000 shares of CPU and another 500, when resource contention arises, the VM with 1000 shares will be granted access to twice as much CPU as the VM with 500 shares.
• Reservations: This is a guaranteed amount of resources RESERVED for a particular VM. When a VM is powered on the reserved amount of resources are given to that VM and are "eliminated" from the pool of open resources that other VMs can use. A VM can(should)not power on if a the host it resides on does not have enough resources to meet the reservations of that VM. (DRS rocks)
• Limits: A limit is as easy to explain as reservations, it is a limit on the amount of resources that a VM can use. I can never grasp this concept completely because personally if I wanted a VM to have a "limit" of 10 GB of RAM, I would only allocate it that much. Plus if you use limits and there are resources abundant, you could be potentially wasting available resources.

These configuration settings are known as entitlements and are what the CPU scheduler uses to determine priority when scheduling worlds (think VMs) to the physical processors. If you dont know where to edit these settings, just right click any VM and edit its settings within the vSphere client and go to the "Resources" tab. Peep the screen shot if you need a reference! BAM!

In basic terms the CPU scheduler looks at a world's entitlements compared to its used CPU resources, then the CPU scheduler affixes a numeric value to that request. The lower the number, the higher priority. If a world requests processor time and its entitlements are higher, its CPU usage is lower, and its assigned numeric value is lower than the requests for processor time from other worlds, it will be given priority. One of the core concepts is that when a world requests access to the physical CPUs, it needs access to as many cores as it has vCPUs. Although this method is becoming more "relaxed" the basic ideals apply. I highly recommend you go read my co-worker Jim Hannan's post about the CPU Co-Scheduler and its evolution from the strict Co-Scheduler in 2003 to the now [extremely] relaxed Co-Scheduler. Heres the link to his post:

http://www.houseofbrick.com/blog/vsphere-5-advantages-for-vbca-part-1.html?blogger=solutionsarchitects

To further illustrate how the CPU scheduler works and its fundamentals (although they are always being adjusted), lets take a visual looksie!:
Example1 (click for largification):

Basically what is shown here is an ESXi host with 2 CPU sockets each dual core. Currently the processor has 3 of 4 cores scheduled, which means that the only other VMs that could/will be scheduled will be either one of VM9, VM5, or VM6. Which VM will actually get the spot will be determined by the entitlements algorithm. Whichever VM has the lower numeric rating will take the last single core schedule. Both VM1 and VM3 are "disqualified" from getting  scheduled at the moment because they both have more vCPUs than cores are available. This is event is known as CPU Ready Time, which by definition means that the VM is ready and needs CPU, but there are not enough resources to accomidate the VM, so it waits....bad stuff happens if the CPU Ready time gets out of hand. My next blog post will be about CPU Ready and a real world example I just experienced at work this week. Lets take a look at another example.
Example 2 (click for largification):

In example 2 we can see that of the 4 cores of the system, only one is being accessed at the moment with 1 scheduled vCPU request. With a total of 3 cores available at the moment, VM9, VM5, VM3, and VM6 are all potential candidates for being scheduled. It will come down to entitlements once again. Lets assume that VM3 has been given more entitlements, one factor that alludes to this is that it has been assigned 2 vCPU instead of 1. So our buddy VM3 gets scheduled on 2 cores which leaves one last core for which ever of our single core VMs is entitled higher.

There are cases obviously where entitlements match and there is no real "winnner", I would refer you to the white paper mentioned at the bottom of this post for more details on that matter. SOOOO, we got all cores scheduled, good job ESX CPU Scheduler, one problem, our poor 4 vCPU, VM1, is still left out in the cold, cranking up his CPU Ready time. If CPU ready time gets to high, a VM could be potentially be waiting secondS (no I didnt fat finger the the "S", just emphasizing the point that theres multiple) for its CPU requests/needs to be met. Here is a design hint, keep VMs with like vCPU allocation on the same hosts, this will keep CPU Ready time lower. If you have a host with mostly 2 vCPU VMs, they will get scheduled quicker because 2 free cores will free up more often because other 2 vCPU VMs were just scheduled. Also, try to limit the vCPUs assigned to VMs to only the number they truely need, granted this is probably a "DUH" moment, people sometimes get a little vCPU, this is a classic case of how over allocation can hurt ya in the VMware wold. Check the VM performance in the vSphere client, my favorite at the moment is choose an advanced chart under performance and select the CPU usage by percentage and also view the CPU Ready time. CPU Ready time is safe at around 100ms to 150ms (real-time), the danger zone (welcome) for a real time graph would be 500ms. Look for my next post to get me details on this.

I am a firm believer that these concepts about the CPU Scheduler in ESX/ESXi should be known and understood to better grasp how to monitor and size your VMs, that being said, these concepts are changing and evolving with each vSphere release. Once again I will refer you to Jim Hannan's post that I linked up above, he discusses in detail the the evolution and current version of the CPU Scheduler. To make a point about the evolution of the CPU Scheduler, in both of the visual examples from up above, the relaxed Co-Scheduler will now allow some multi vCPU machines to be scheduled even if all cores are not available. Take a look at Jim's post DANGIT!

I leave you with this final screen shot, check it out, be stunned, pick your jaw up off the ground. Compare the safe and danger zones in millisecond I just shared with you for CPU Ready time, to what you see in this graph. I will talk about this is my next post. CPU Ready time for the win! (click for largification)

The link below is a white paper that I used as a companion while reading "Critical VMware Mistakes That You Should Avoid", its a REALLY good read. Many of the initial concepts found in the paper I mention in this post, but it goes way way ... way deeper than my post. You will notice its for vSphere 4.1, I feel like VMware is no longer writing these types of beneficial deep dives, so referencing the old docs is what you a lot of times have to result to.
http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf

Powershell: Get the Services Running and Their Service Accounts on a list of Remote Machines

Time for me to play Powershell Ranger and throw a quick script at you! BAM. A co-worker asked me today to whip up a quick script that could retrieve the services and their corresponding service accounts (user account who the system uses to run the service) from a list or array of remote hosts. I first thought, "Hey no Biggie....Smalls!, I will use the Get-Service cmdlet". The issue with that as seen in the screen shot below the Get-Service cmdlet, when expanding out the full results for one service, does not include a property which details the Service Account.

So instead of using the Get-Service cmdlet we will turn to the Get-WMIObject cmdlet and query the win32_service class to get the values we need. As seen in the below screenshot this method of retrieving a systems Service details contains much more data and it even inlcudes the value "StartName", which contains the Service Account name.

I recommend trying both the methods your self to see the difference:
    Get-Service -Name WinRM | Select *
    Get-WmiObject win32_service | Where {$_.name -eq "WinRM"} | Select *

Now to the Script:

• Let's review whats going on in this script. First off we create a variable " ServerArray", here is where you will add the names of the servers you wish to run this script against. Follow the instructions in the comment below to add the Server names correctly to create a variable array. If you have a list of servers in a text file, you can have Powershell grab the content of that file and create an array of server  names that way. To do that just remove everything after the "=" sign after ServerArray and enter the following code: Get-Content -Path <path to text file>, that's all! Remember for simplicitys' sake, one server name per line in the text file.
• Next we let the user running the script either define the location to save the out put file that this script will make or let the script create the default directory which is "C:\Temp\SvcsandSvcAccounts\".
• Next we have the script verify that the path exists and if it doesn't, the script will create the path to the save location using the New-Item cmdlet. Give a path to the New-Item cmdlet and it will create the file structure it needs to get to the final location.
• After we define the path to the save location and create the directory structure to it, we begin to loop through each server in $ServerArray. We tell the user with the Write-Host cmdlet which host we are currently grabbing the information for and then go through and grab 4 values for each service.
• The Get-WMIObject cmdlet uses the -ComputerName flag to connect to remote servers. In our case we pass the server names to the -ComputerName flag using the $Server variable in the foreach loop. Use the Get-WMIObject alias if you want also: gwmi.
• In this script and many others of mine, I decided to use the calculated property capabilities of Powershell only to rename the Headings on some of the returned value columns, but this is only scratching the surface of the Powershell calculated property ability. I hope to discuss calculated properties in a later blog post. a calculated property example found in the script is any line that loks like this:  @{N="Name";E={$_.Expression}}, you can define your own Names for values in the "N" field, and do calculations in the "E" field. In my example I decided to just regurgitate the $_ variable with an associated property instead of doing anything more complicated with it.
• In the final steps we do two very simple things: Sort the output using the Sort-Object cmdlet and out put the entire table to a text file called $Server-Services.txt, -or in our case, localhost-services.txt

# List multiple systems in the SystemsArray Variable: quote each system name and seperate with a comma, # this creates an array. Ex: "system1","system2","system3"


$ServerArray="localhost"
$DefineSaveLocation=""
if ($DefineSaveLocation -eq "")
    {$DefineSaveLocation="C:\Temp\SvcsandSvcAccounts\"}
$SaveLocaPath=Test-Path $DefineSaveLocation
if ($SaveLocaPath -eq $False)
    {New-Item -ItemType directory -Path $DefineSaveLocation}
cd $DefineSaveLocation
Foreach ($Server in $ ServerArray )
{
Write-Host "Retrieving Servers for $Server "    
Get-WmiObject win32_service -ComputerName $Server  | select Name,
@{N="Startup Type";E={$_.StartMode}},
@{N="Service Account";E={$_.StartName}},
@{N="System Name";E={$_.Systemname}} | Sort-Object "Name" > ".\$Server -Services.txt"
}

The generated output from the script looks like this, but you could do CSV, html, excel...whatever you like:

This was a super simple Powershell script, but with Powershell the possibilities are endless. More to come!

Shout Out: David Klee "Virtualizing Business-Critical SQL Servers – Part 2: Understanding the Physical Workload"

Another Shout Out coming full speed, course correcting to ram right into you. This time a post from David Klee, a Solutions Architect at House of Brick Technologies. In this Shout Out, bring on the bench marking, performance charting, workload tracking pie and give me the whole thing. David's posts is second part to a multipart blog post series on Virtualizing Business Critical SQL Servers. His post gives you a pretty detailed map on the areas to focus on so that you can know what you need to about your physical workload. Following his guidelines can help you effectively create baselines from scheduled periodic benchmarks and from there effectively tune your SQL Server to get the best performance. His blog post includes the following subjects:

• Benchmarks
• Disk Performance
• SQL Server and Windows Metrics
• Query Performance
• Baselines

This was a really great post and there were a lot of things I got out of it, here is a snippet of one of my favorite parts:

"Now, what good is a benchmark if you do not have a running average metric to compare it against?
A baseline is a rolling average of your repeatable benchmarks. You should routinely (and not just once a year either) benchmark your systems and compare against a rolling baseline to see how things are performing. Once completed, update your benchmarks accordingly with the updated data.
[...]
The bottom line with benchmarks is that not only do you have an objective measure of the average performance of your systems, but, in the event of a problem, you have an objective means of defending your system. It can even help point out performance problems in other systems. If a performance problem does show up in your system, you have the ability to quickly determine the area that requires more focus, and the means to prove when it is resolved."

Baselines are to often over looked because of a lack of bench marking. Too commonly as a consulting company we are asked to perform health checks on customer systems, how much more beneficial those health checks would be if we had baselines to use to contrast against the new numbers and figures....sigh....

There was one customer who had baselines back 5 years, done quarterly, in easy to read and understand documents with graphing and metadata...oh wait....no......I guess it was all a dream, right Biggie? I used to read Word Up magazine.

Heres the link:
http://www.houseofbrick.com/blog/virtualizing-business-critical-sql-servers-a-part-2-understanding-the-physical-workload.html?blogger=solutionsarchitects

Also checkout David's personal blog @:
http://www.davidklee.net/

Or follow him on Twitter:
@kleegeek

Shout Out: Jim Hannan "vSphere 5 Advantages for VBCA -- Part 4 vMotion"

Call me silly, ....nevermind....don't. But I get amazed at features within vSphere that now a days are taken for granted a lot of times. I can't seem to ever get enough of the nitty gritty details of how all the components like vMotion, HA, DRS, NUMA, etc work and orchestrate together. Good thing for me, I work with a bunch of people who share that same desire to know and understand how the vSphere components work. Check out this blog post from Jim Hannan, a co-worker of mine and a Solutions Architect at House of Brick Technologies. His post is about Virtualizing Business Critical Applications/Systems and what are the gained benefits of doing so with vSphere 5.0.
Heres a snippet from his post that I particularly enjoyed:

"vMotion, originally introduced in ESX 2.0 with vCenter 1.0, offered a feature unlike any in the industry. For many organizations, it was the quintessential reason to virtualize their workloads.Care to guess what interactive application was used by VMware to demo the first migrations?Answer: Pinball."

I wasnt in the industry at that time, but I would have loved to see that moment in person:

~~~~*Que the harp music; Movie transition to a memory in the past that didn't happen*~~~~

Me: "Did you see that?"
Guy Next to Me: "Yes, I am sitting right here.....*pause*......watching the same thing you are?"
Me: "Did you see that.....S E R I O U S L Y?"
Guy Next to Me: "Yes....."
Me: "He saved the ball right when it was about to go in!...holy crap."
Guy Next to Me: "You're missing the poin........nevermind"
Guy Next to Me: *Maybe if I dont move....or breathe...he wont know I am here, Jurassic Park Style*
Me: *Guy totally missed that awesome save, VMware rocks*

Heres the link, spend 5 minutes and read it over, its an awesome quick post:
http://www.houseofbrick.com/blog/vsphere-5-advantages-for-vbca-part-4-vmotion.html?blogger=solutionsarchitects

Book Review: "Critical VMware Mistakes You Should Avoid" by Larry Loucks

Awhile back I was doing some shopping on Amazon for a few technical books that I could read in my free time, co-workers of mine had recommended a handful of books that they thought I would like. My day job provides a book allowance for employees and I had planned to take advantage of the whole shabang at once. I had funneled down to the last bit of allotted money, and had stayed true to the list compiled by co-worker recommendations, as I poked around some VMware related books, Larry Loucks' "Critical VMware Mistakes You Should Avoid" popped up as a recommendation, "Buy this book and get this one with it for a crazy deal!". I decided to check it out and while glancing at the details page I wondered what a 115 page book could really cover in comparison to 700 page books detailing all things that had anything to do with VMware. I decided to to take the plunge and grab the book with my last remaining bit of allotment. This turned out to be an awesome idea. The easiest way for me to tell you what I think about Larry's book is to throw a bulleted list at ya, so here it is:

1. If asked to sum up what I think about the book in one sentence I would probably say something along these lines: "This book is a must read for IT professionals dealing with VMware who would like to understand some crucial deep concepts explained in a fashion that all degrees of understanding can benefit from."
2. The books starts slow, at first I found my self thinking, "OK I get what you are saying, lets move on", but after a few pages, I could not put the book down. I was traveling at the time I was reading the book, and if I wasn't working I was pretty much trying to spend my time reading this book. Basically you start a section and the section starts relatively slow by explaining the basics and in what ways people do things incorrectly, and then he goes in for the deep dive and he explains why you do it the way he suggests and ties it all together. This is the prime reason this book turned out to be so valuable to me, to see how the simple creates the foundation for the complicated, and then getting the explanation how the complicated works.
3. One of my favorite things about the book is the real world stories he shares from his consulting experiences. Larry Loucks has been in IT for over 24 years and for the past seven has specifically focused on virtualization and worked for VMware.  And as it says in the first paragraphs of the book, hind sight is 20/20. The idea of his book is to present you with that "hind sight" that he has gained to help you now.
4. This book expects that you already have a VMware foundation. While at times he does cover relatively simple basic things, you will need that foundation for when he hits you with the awesome stuff like broadcast storms and proper vLANing, transparent page sharing, CPU scheduling, details of performance metrics, etc... 
5. Lets say for instance you are an IT professional with the VCDX under the belt or a certification similar to that and above, you may not get a heck of a lot out of this book. My guess is that if you are that high in the food chain the advanced topics in the book, when they come, may not teach you anything you don't already know. Having said that, I work with some of the smartest guys in the industry for VMware performance and design. While talking with a couple of them about the content in the book, they were interested in reading it, and were excited to hear some of the principles we share with our customers which we hold as best practices reiterated in this book.
6. Pictures and Graphs....gimmie gimmie. I originally started my dive into computers and tech in the graphics and web design realm. I am right brained person whos let the left side slowly in, but I still learn best with visuals and I believe I always will. Although the graphics in this book aren't fine art or detailed beyond belief, they made a world of difference while reading through this book. Now when I think about certain deep dive topics that were discussed in this book, instantly the visuals used come to mind.
7. Don't be surprised when the authorship of the book isn't up to par with some of the more professional, or should I say, more "Official...eehhh heemmm....Boring, but more polished" authors out there. Lucky for me, Larry's writing style is how I personally enjoy reading books. His wording and structure is easy to read, you don't need to spend time wondering what he just said every 5 seconds, and I like when the author just seems relaxed and has a good time writing, which he pulls off well. To me, and maybe I'm crazy, but I think that when an author is relaxed, can make jokes in his wirting, writes in simple terms, and can still relay an excellent message, they know the stuff better than most others do.
8. Typos, their r qwiete aye feuw of tthem. This is my biggest complaint about the book. I am a reader that when I am in the zone reading and trying to absorb, when I hit a typo it totally throws me off. Maybe thats just me as a reader....but I dont think I am alone. But, when my biggest complaint about a book is typos, I think it has accomplished its mission.

In my next few blogs, I will take a few key topics from "Critical VMware Mistakes You Should Avoid", explain them in detail and describe why they are important to know and hopefully provide some visuals for those of you who are like me. Topics will include: Transparent Page Sharing (*sigh* vRAM licensing......YAY Its gone!), CPU Scheduling, and proper vLANing to avoid broadcast storms.

I give this book:

9 out of 10 monsterrrrrrs: "Highly Recommended"