Logon Performance in VDI Land

After spending hte better part of three days attempting to shave time off of login times in our VDI environment (VMware View-based), I thought I should scribe down some notes on effective troubleshooting tools and techniques. There were a lot of self-inflicted wounds this time, and I could have saved myself a lot of time if 1) I had documented the build process for my new VDI pool and 2) I had taken notes that last time I had made login optimizations.

WARNING: This post is largely unedited and probably a bit incoherent. Read at your own risk.

Group Policy:

Computer Configuration->Policies->Administrative Templates->System: Display highly detailed status messages
This setting causes the Windows login screen to provide more verbose feedback to the user about what winlogon.exe is doing at any given time. Rather than just seeing “Preparing Windows”, you will instead see things like “Processing Drive Map Preferences”. If the logon screen hangs on one of these steps for 30 seconds, you will know exactly which Group Policy setting is killing logon performance.

Event Viewer:

Windows 7 and 8 both include a Group Policy operational log under: Event Viewer->Applications and Services Logs->Microsoft->Windows->GroupPolicy->Operational.  This log contains a lot of useful information about the timing of various group policy components, and many times will contain all of the information you need to pinpoint troublesome Group Policy Settings.

If the Event Viewer does not have all of the information you need, you can enable verbose policy logging:
http://social.technet.microsoft.com/wiki/contents/articles/4506.group-policy-debug-log-settings.aspx

I typically find that this is not necessary, and that the Event Viewer has the information that I need.

Problems with User Profile loading often can be found under: Event Viewer->Applications and Services Logs->Microsoft->Windows->User Profile Service->Operational log. This log is especially useful when using roaming or mandatory profiles. Unfortunately, this entry just tracks initial profile location and loading, and does not log anything related to Active Setup.

Windows Performance Toolkit:

Part of the Assessment and Deployment Toolkit (ADK) for Windows 8.1. The Performance Toolkit includes the Windows Performance Recorder and Windows Performance Analyzer. Run the Recorder with the “Boot” performance scenario, with 1 iteration, then use the Analyzer to read the trace file that was created during reboot and logon. Make note of the relative time of each event in the boot/logon process (i.e. time of boot, time of login, time to desktop load). The Recorder only logs relative time from boot up, so you might have some trouble correlating wall-clock time with recorded event times. Try to locate processes that line up with the delays you see during login.

As an alternative, you can enable boot logging using “ProcMon”. The Performance Analyzer arguably offers better visualizations of boot issues, but ProcMon has more comprehensive process information, and may be a more familiar tool for many administrators.

Active Setup:

Active Setup is a pain. This is a pooly documented mechanism by which applications (mostly Microsoft applications) can run per-user configuration tasks (generally first-run tasks) on logon. It is synchronous, meaning each task much be completed before the next runs. Also, Active Setup runs in Winlogon.exe and blocks loading of the desktop. Because of this, Active Setup has the potential to greatly delay first time logon. As a result, it also becomes a scapegoat for logon delays, even when it is not the root cause. I have no really helpful advice for troubleshooting Active Setup other than use use the Performance Analyzer or ProcMon to locate Active Setup processes that take a long time to execute. See the following for a better explanation of the internals of Active Setup:
http://helgeklein.com/blog/2010/04/active-setup-explained/
And this for an explanation of situations in which you might want to disable Active Setup:
http://blog.ressoftware.com/index.php/2011/12/29/disable-active-setup-revealed/

SysInternals AutoRuns:

You can wade though every obscure registry key looking for processes that run at login, or you can just use AutoRuns and pull them up all in one place. Thanks to AutoRuns, I was able to locate the entry point for an irksome logon process that was running for no apparent reason. I had forgotten that under Windows Vista and later, Scheduled Tasks can use user logon events as a trigger event for starting a process. This brings us to the process that killed two days of my life…

Minor Troubles with Google Chrome:

Using The Performance Analyzer, I concluded initially that Google Chrome was adding over 30 seconds of time to logon on one of my VDI pools.  While Google is launching “GoogleUpdate.exe” at each user logon event (via a scheduled task trigger), these scheduled tasks really should not block loading of the desktop.  This task runs in other pools, without significant delay.  In this pool, the task was running for a long time (over a minute) before exiting.  The likely cause of this excessive delay is the internet-bound HTTP/HTTPS filtering that is taking place in this pool… Google cannot update itself if outbound internet access is blocked.  Still, long running or not, Chrome Update was not blocking loading of the desktop.

That being said, our users really do not need Chrome to check for updates on each and every logon, so how to we fix this?
Investigation of Active Setup showed that Active Setup for Chrome already had been completed in our Mandatory roaming profile. So why was Chrome setup running on each and every user logon? It also was configured as a Scheduled Task that runs on each user logon event. Aargh! As noted above, SysInternals AutoRuns was used to locate this entry point.

Unfortunately, Google Update is a bit on the complicated side:
http://omaha.googlecode.com/svn/wiki/GoogleUpdateOnAScheduleOverview.html

There are two separate Google Update system services, two separate Scheduled Tasks related to Google Update, and three separate task triggers, including the one that runs a logon. For now, I have just disabled the scheduled tasks in my template machine. Unfortunately, this completely disabled Google Update in the VDI pool. Also, the changes will be wiped out if we update Google manually or via SCCM in the future. Better would be a Group Policy-based solution.

Some of you may know that there actually is official registry/Group Policy support for control of Google Update. See:
https://support.google.com/installer/answer/146164
and:
http://www.chromium.org/administrators/turning-off-auto-updates
However, these setting just disable Auto Update entirely. They do not allow you to control how and when updates will apply (i.e. disable user-mode updates, but leave machine-mode updates intact.

I expect the “real” fix here would be to run a separate scheduled task script or startup script that used PowerShell to fund and remove the scheduled task triggers. That’s more time than I want to spend on this project at present.