Tag Archives: VMware View

VMware View – Provisioning/Composing hangs, Event log failures, and more!

VMware Horizon View… great product. View Composer? Thorn in my side.

Two weeks back I completed the upgrade of our View infrastructure from 5.3.2 to 6.0.1. It was a smooth upgrade, seemingly, and I was pretty pleased with how little time it took to complete the job. Victory for our team? Not so much.

Over the next week, I had dozens of complaints from IT staff that recompose operations were failing, searches for events related to these failures were returning no results (or just not completing at all), and there were multiple odd “I am getting this weird error on my desktops!” complaints.  The desktop errors all turned out to be unrelated to the upgrade (the template was out of disk space, so the user profile could not load, the View Agent installation was broken, etc. etc.), but sorting out the event log and composer problems were harder…

View 6 Event Log database bug:

Following the upgrade, I was looking into increasing the View Event Log query limit per the request of a client, who was not able to view more than the past few hours of events for his pool owing to the default event query limit of 2000 events.  I noticed that these queries, in addition to being short on useful information, also were taking several minutes to complete.  After bumping the query limit to 6000 events, we found that the queries were taking over 30 minutes to complete, and hogging up all the CPU on the Virtual Center server (where the events database is hosted)!  I verified that memory and disk were not bottlenecked on the SQL database (I could not add more CPU because I already was at the SQL Standard Edition max of four cores), and set SQL tracing to look for deadlock events.  After running into a bunch of dead ends, I finally opened a support case with VMware.

Unsurprisingly, the first response was “well, lower your query limit.”  I explained that no, I was not going to do that.  I also pointed out that selecting 6000 records from a 2.4 Gb database really should not take 30 minutes, and that engineering just needed to buckle down and fix whatever index was causing the problem.  A few days later, I was given one line of T-SQL to run against the View Events database to add a missing index.  Query got executed, index created, and voila!  Event queries started running in seconds, not hours.  Here is the T-SQL:

CREATE INDEX IX_eventid ON dbo.VE_event_data (eventid)

Your table name might be slightly different, depending on the table prefix you selected when setting up the events database.

Composer Failures:

We have seen this before… someone recomposes a pool, the job half-finishes then stops, no error.  The task cannot be canceled, the pool cannot be deleted, and all other Composer operations in the infrastructure grind to a halt.  Why?  If you call VMware support, the first thing they will tell you is “cache corruption”.  The next is “stale desktops”.  Huh?

Deleting Stale Desktops:


Clearing the Connection Server Cache:

No KB for this one that I am aware of.  Here is that they always tell me to do… ready?  You are going to like this…

  1. Shut down all of the connection servers in your farm.
  2. Turn the connection servers back on, one at a time.


The worst part is, that neither of these solutions worked.  However, what I did find was that after powering the connection servers back on, some composer operations would succeed, but it was only a matter of time before one job failed an brought operations to a halt.  Finally I noticed that when rebooting one of the connection servers (the newest one, used for testing security settings), jammed jobs would immediately resume.  After digging into the logs in C:ProgramDataVMwareVDMlogs, I found that the Connection Server was reporting literally thousands of “could not connect to vCenter server at URL…” errors per day.  Why?  Because like a noob I did not give this connection server in interface to the vCenter server.  Bad on me.  However, these critical failures do not show up in the Windows event logs, nor do they get reported up to the View Administrator console.  I had a bad connection server in my environment that was killing Composer operations, and View Administrator thinks everything is peachy.  Boo!  I have complained to VMware support, for what it is worth.  I also fixed the connection server, and things are back to “normal”, whatever that means.

I also got my manager to approve using Splunk to collect all View log files, so that I at least will have an easier time of discovering errors when they arise in the future.

Logon Performance in VDI Land

After spending hte better part of three days attempting to shave time off of login times in our VDI environment (VMware View-based), I thought I should scribe down some notes on effective troubleshooting tools and techniques. There were a lot of self-inflicted wounds this time, and I could have saved myself a lot of time if 1) I had documented the build process for my new VDI pool and 2) I had taken notes that last time I had made login optimizations.

WARNING: This post is largely unedited and probably a bit incoherent. Read at your own risk.

Group Policy:

Computer Configuration->Policies->Administrative Templates->System: Display highly detailed status messages
This setting causes the Windows login screen to provide more verbose feedback to the user about what winlogon.exe is doing at any given time. Rather than just seeing “Preparing Windows”, you will instead see things like “Processing Drive Map Preferences”. If the logon screen hangs on one of these steps for 30 seconds, you will know exactly which Group Policy setting is killing logon performance.

Event Viewer:

Windows 7 and 8 both include a Group Policy operational log under: Event Viewer->Applications and Services Logs->Microsoft->Windows->GroupPolicy->Operational.  This log contains a lot of useful information about the timing of various group policy components, and many times will contain all of the information you need to pinpoint troublesome Group Policy Settings.

If the Event Viewer does not have all of the information you need, you can enable verbose policy logging:

I typically find that this is not necessary, and that the Event Viewer has the information that I need.

Problems with User Profile loading often can be found under: Event Viewer->Applications and Services Logs->Microsoft->Windows->User Profile Service->Operational log. This log is especially useful when using roaming or mandatory profiles. Unfortunately, this entry just tracks initial profile location and loading, and does not log anything related to Active Setup.

Windows Performance Toolkit:

Part of the Assessment and Deployment Toolkit (ADK) for Windows 8.1. The Performance Toolkit includes the Windows Performance Recorder and Windows Performance Analyzer. Run the Recorder with the “Boot” performance scenario, with 1 iteration, then use the Analyzer to read the trace file that was created during reboot and logon. Make note of the relative time of each event in the boot/logon process (i.e. time of boot, time of login, time to desktop load). The Recorder only logs relative time from boot up, so you might have some trouble correlating wall-clock time with recorded event times. Try to locate processes that line up with the delays you see during login.

As an alternative, you can enable boot logging using “ProcMon”. The Performance Analyzer arguably offers better visualizations of boot issues, but ProcMon has more comprehensive process information, and may be a more familiar tool for many administrators.

Active Setup:

Active Setup is a pain. This is a pooly documented mechanism by which applications (mostly Microsoft applications) can run per-user configuration tasks (generally first-run tasks) on logon. It is synchronous, meaning each task much be completed before the next runs. Also, Active Setup runs in Winlogon.exe and blocks loading of the desktop. Because of this, Active Setup has the potential to greatly delay first time logon. As a result, it also becomes a scapegoat for logon delays, even when it is not the root cause. I have no really helpful advice for troubleshooting Active Setup other than use use the Performance Analyzer or ProcMon to locate Active Setup processes that take a long time to execute. See the following for a better explanation of the internals of Active Setup:
And this for an explanation of situations in which you might want to disable Active Setup:

SysInternals AutoRuns:

You can wade though every obscure registry key looking for processes that run at login, or you can just use AutoRuns and pull them up all in one place. Thanks to AutoRuns, I was able to locate the entry point for an irksome logon process that was running for no apparent reason. I had forgotten that under Windows Vista and later, Scheduled Tasks can use user logon events as a trigger event for starting a process. This brings us to the process that killed two days of my life…

Minor Troubles with Google Chrome:

Using The Performance Analyzer, I concluded initially that Google Chrome was adding over 30 seconds of time to logon on one of my VDI pools.  While Google is launching “GoogleUpdate.exe” at each user logon event (via a scheduled task trigger), these scheduled tasks really should not block loading of the desktop.  This task runs in other pools, without significant delay.  In this pool, the task was running for a long time (over a minute) before exiting.  The likely cause of this excessive delay is the internet-bound HTTP/HTTPS filtering that is taking place in this pool… Google cannot update itself if outbound internet access is blocked.  Still, long running or not, Chrome Update was not blocking loading of the desktop.

That being said, our users really do not need Chrome to check for updates on each and every logon, so how to we fix this?
Investigation of Active Setup showed that Active Setup for Chrome already had been completed in our Mandatory roaming profile. So why was Chrome setup running on each and every user logon? It also was configured as a Scheduled Task that runs on each user logon event. Aargh! As noted above, SysInternals AutoRuns was used to locate this entry point.

Unfortunately, Google Update is a bit on the complicated side:

There are two separate Google Update system services, two separate Scheduled Tasks related to Google Update, and three separate task triggers, including the one that runs a logon. For now, I have just disabled the scheduled tasks in my template machine. Unfortunately, this completely disabled Google Update in the VDI pool. Also, the changes will be wiped out if we update Google manually or via SCCM in the future. Better would be a Group Policy-based solution.

Some of you may know that there actually is official registry/Group Policy support for control of Google Update. See:
However, these setting just disable Auto Update entirely. They do not allow you to control how and when updates will apply (i.e. disable user-mode updates, but leave machine-mode updates intact.

I expect the “real” fix here would be to run a separate scheduled task script or startup script that used PowerShell to fund and remove the scheduled task triggers. That’s more time than I want to spend on this project at present.

Parting Scripts – Add a new network printer and set it as default

Some time back, I discovered that a Group Policy Preference that we had applied to a VMware View VDI pool was adding an additional 30 seconds of time staring at the blue spinning donut at each VDI desktop logon.  The policy in question was a printer policy.  Colleagues at other Higher Ed institutions confirmed that they had the same problem with GPP printer preferences.  It has been reported that using the “Mandatory Printers” policy is faster, but this policy does not allow you to assign a default printer.

Enter good old VBScript…

The following script will install a defined network printer and set it as default. If the print share does not exist, an error will be returned. 95% of the code in this script was lifted from my own “KillAndExec.vbs” script from last year. There really is only two lines of new code in here. It is good having a code library to draw on, because it would have taken be days to generate this stuff from scratch. VBScript is so obtuse… so why do I keep using it? Hmmmm….

VDI Profile Loading Delays

We are noticing that it takes rather a long time for users to log in to our VDI environment (~2 minutes, in some circumstances).  I did some analysis of login times using Sysinternals Procmon.  (Enable boot logging, use the “view process tree” feature to look at process times at logon.  See http://blogs.technet.com/b/markrussinovich/archive/2012/07/02/3506849.aspx for details).  What I found was that a child process of explorer.exe called “ie4uinit.exe” was running for most of this time.  This process appears to be part of Microsoft “Active Setup” (discussed in some detail here: http://blog.ressoftware.com/index.php/2011/12/29/disable-active-setup-revealed/).

So what if we disable Active Setup?  Noise on the Internet suggests that this is possible , simply be deleting the key:
HKEY_LOCAL_MACHINESoftwareMicrosoftActive Setup
as suggested here:

However, there is some indication that this could have unintended consequences.  In my case, it immediately caused a logon script to fail to run.  Bummer!

What other solutions are possible?  Members of the Windows in Higher Education mailing list recently recommended using mandatory profiles.  There is a reasonably good rundown of the mandatory profile creation process here:
Missing details are:

  1. It is possible for the mandatory roaming profile to be stored locally (i.e. “C:UsersVDI_Mandatory.V2” to avoid over-the-network profile copy delays.  However, in our View environment, using a network location appears to be faster!
  2. The mandatory roaming profile can be specified using the Group Policy settings in Computer -> Policies -> Administrative Templates -> System -> User Profiles.  (See “Set roaming profile path for all users logging onto this computer” and “Delete cached copies of roaming profile”.)

In testing, I found initial logon times were reduced from two minutes to approximately 20 seconds.  Good!  (But still not great.)  Additional benefits are that it is no longer necessary to run the logon script that I developed to customize the Start Menu and Task Bar.  I also can remove the Group Policy preferences that clean up local profiles on the computer.

Prepare SCCM Clients for Cloning

Annoying task… configuring VMware View desktops for use in an environment that utilizes VMware View.  Some say, don’t put the management agent on the View desktop, just rebuild your desktops every time there is a security patch.  I say, even if recomposing your pools is fast and easy, I still do not want to do it with every patch release.

Best practice for preparing a reference computer for deployment in an SCCM environment is to not include the SCCM client.  However, SCCM client installation is SLOW, so I would like to save time and CPU load by including the software.  Documentation how to do this is sketchy.  Also, I really want a script fired off by the QuickPrep process to do the prep work, so that someone does not forget that it needs to be done.

Here is my first pass at the script… again, not too pretty, but functional:

'  NAME:    sccmClientPrep.vbs
'  AUTHOR:  J. Greg Mackinnon
'  DATE:    2013-02-01
'  COMMENT: prepares SCCM client for cloning
'           Requires:
'             - Certutil.exe in %systemroot%system32 (included with Win7)
'           Returns:
'           1 - CCMEXEC service stop failure
'           2 - Machine Certificate Store deletion failure
'           4 - SMS Certificate Store deletion failure
'           8 - SMSCFG.INI deltion failure
option explicit

'        CONSTANTS
const MACH_STORE = "My"
const SMS_STORE = "SMS"
const SVCNAME = "ccmexec"
const TIMEOUT = "120"

'        OBJECTS
dim oShell
set oShell = CreateObject("WScript.Shell")

'        VARIABLES
dim sSysRoot,sCUPath,sINIPath
dim iRet, iExit

sSysRoot = oShell.ExpandEnvironmentStrings("%SystemRoot%")
sCUPath = sSysRoot & "system32certutil.exe"
sINIPath = sSysRoot & "SMSCFG.INI"
iExit = 0

function stopSvc(sSvcName,iTimeout)
' Stops the Windows service with name matching input string "sSvcName".
' Times out in iTimeout seconds.
' Needs routine to verify that sSvcName is a valid service name.
	dim bDone 
	dim iSecs 
	bDone = False
	iSecs = 0

	'Objects and Collections:
	dim cSvcs
	dim oWMI, oSvc
	Set oWMI = GetObject("winmgmts:{impersonationLevel=impersonate}!\.rootcimv2")
	Set cSvcs = oWMI.ExecQuery("Select * from Win32_Service Where Name = '" & sSvcName & "'")

	'Stop the service if it is running, exit if it is not running
	For Each oSvc In cSvcs
		if oSvc.State = "Running" then
			stopSvc = 0
			exit function
		end if
		exit for 'Only on service in collection

	'Check on the service until stopped.  Timeout in iTimeout seconds.
	Do while bDone = False
		Set cSvcs = oWMI.ExecQuery("Select * from Win32_Service Where Name = '" & sSvcName & "'")
		bDone = True
		For Each oSvc In cSvcs
			If oSvc.State  "Stopped" Then
				bDone = False
				WScript.Sleep 1000
				Exit For
			End If
		iSecs = iSecs + 1
		If iSecs >= iTimeout Then
			stopSvc = 1
			exit function
		End If
	stopSvc = 0
end function

function delCert(sStore,sSerial)
'Deletes certificate in certificate store "sStore" with serial number "sSerial"
'Returns: The ExitCode from certutil.exe
'   - WScript.Shell object named "oShell"
'   - Defined path to "certutil.exe" named "sCUPath"
'   - Presence of certutil.exe on the system
	dim oExec, oStdOut
	dim sLine
	'wscript.echo "About to execute: " & sCUPath & " -delstore " & sStore & " " & sSerial
	set oExec = oShell.Exec(sCUPath & " -delstore " & sStore & " " & sSerial)
	Set oStdOut = oExec.StdOut
	Do While oExec.Status = 0
		WScript.Sleep 100
	'Uncomment the next four lines to debug certutil:
	'Do Until oStdOut.AtEndOfStream
	'	sLine = oStdOut.ReadLine
	'	wscript.echo sLine
	delCert = oExec.ExitCode
end function

function getCert(sStore)
'Gets the serial numbers of certificates in the machine store specified by "sStore"
'Sends the captured serial numbers to the "delCert" function for deletion.
'Returns: 1 - If cert deletion files, 0 - If no errors are detected.
'   - WScript.Shell object named "oShell"
'   - Defined path to "certutil.exe" named "sCUPath"
'   - Presence of certutil.exe on the system
	dim oExec, oStdOut
	dim bFail
	dim i, iRet
	dim sLine, sSerial
	Set oExec = oShell.Exec(sCUPath & " -store " & sStore)
	Set oStdOut = oExec.StdOut
	Do Until oStdOut.AtEndOfStream
		sLine = oStdOut.ReadLine
		if InStr(1,sLine,"Serial Number",1) then
			i = CInt(InStr(1,sLine,":",1) + 2)
			sSerial = Mid(sLine,i)
			iRet = delCert(sStore,sSerial)
			if (iRet  0) then
				wscript.echo "Certificate deletion failed"
				bFail = True
			end if
		end if
	if bFail = True then
		getCert = 1
		getCert = 0
	end if
end function

function delFile(sFile)
'Deletes the file specified by "sFile"
'Requires existing Wscript.Shell object named "oShell"
	dim oFSO, oFile
	set oFSO = CreateObject("Scripting.FileSystemObject") 

	'wscript.echo "About to delete file: " & sFile
	if oFSO.FileExists(sFile) then
		'Delete method will force a WSH quit if it fails, so we need to disable exit-on-error:
		On Error Resume Next
		set oFile = oFSO.GetFile(sFile)
		delFile = oFile.Delete(True)
		if Err.Number = 0 then
			delFile = 0
			delFile = Err.Number
		end if
		'Exit code for the function could be changed here if you are concerned about
		' the requested file to delete not being present on the system.
		'wscript.echo "File " & sFile & " does not exist."
		delFile = 0
	end if
end function

'          MAIN
iRet = stopSvc(SVCNAME, TIMEOUT)
'wscript.echo "Return from stopSvc: " & iRet
if iRet  0 then
	iExit = iExit + 1
end if

iRet = getCert(MACH_STORE)
'wscript.echo "Return from cert deletion for store " & MACH_STORE & ": " & iRet
if iRet  0 then
	iExit = iExit + 2
end if

iRet = getCert(SMS_STORE)
'wscript.echo "Return from cert deletion for store " & SMS_STORE & ": " & iRet
if iRet  0 then
	iExit = iExit + 4
end if

iRet = delFile(sINIPath)
'wscript.echo "Return from file deletion: " & iRet
if iRet  0 then
	iExit = iExit + 8
end if

WScript.Quit iExit

View Desktop Template Building Notes – Nothing is Easy

This week I have been working on updating our VMware View template for our public terminals (those are kiosk and lab systems).  As always, simple things have become difficult, and time has disappeared like cookies at a preschool party. Here are some resources that were useful to me in cleaning up the reference system:

Default User Profile Settings:

Why must this be so hard?  Under XP, you used to be able to tweak the Default User profile to within an inch of its life, then copy it.  Easy!  But MS maintains that this “caused problems” of some vague nature, and so the process is now not possible.  Instead, we are supposed to use the “CopyProfile” action in the unattend.xml file used by sysprep to trigger copying of the “Administrator” profile to “Default”.  However, this operation does not copy all settings… aargh!  And the settings that are excluded are not documented… double aargh!  And if you have any profile traces left over on your system for any user other than “Administrator”, sysprep will fail miserably… triple aargh!

Fortunately, there is help available:

I found especially helpful the following post on managing Windows 7 Taskbar links:

Ultimately, I used a variation on this script to pin items to the Win 7 taskbar:
(I’ll post the final script I developed separately.  I could not find a fully functional script on the internet.  I had to do some significant mods to the scripts in the above post to get something that does everything that I want.)

Hey, would it not be nice if there were an Group Policy Preference for “shortcuts” to have “taskbar” as a target location?  Yes… yes it would.

Configuration Manager Client Preparation:

I understand you are not supposed to clone a system that has the SCCM client installed on it.  MS documents steps that should be taken to remove the fingerprints of an existing SCCM client prior to cloning… what, so script?

I did some digging and found that the OS Deployment Task Sequences include a step that is supposed to do just this.  After more digging, I determined that the tool for this step is embedded in the SCCM OSD Capture Media.  I generated an OSD Capture Media set, extracted the contents, and found a small executable named “OsdPrepareSmsClient.exe”.  Perhaps this is what I was looking for?  I will test it and find out.

User Profile Cleanup:

Some of our lab admins used to use the profile cleaner utility from MS to erase user profiles from Win2000/XP on logout.  That utility is gone, but you can now use Group Policy to force deletion of all non-Administrator profiles on a schedule.  Unfortunately, this only happens on system restart.

I have settled on “DelProf2.exe” as a replacement:
This excellent freeware utility by Helge Klein will delete all profiles that are not in use, and will clean up the ProfileList registry entries as well.  You can exclude specific named profiles as well.

Java Update… make it stop! :


Unofficial Mozilla Builds for Windows:

(Includes “BlueGriffon” installers that do not embed the obnoxious “iminent toolbar”.  Kudos to Glazman for this excellent basic HTML editor.  Boo to the invasive toolbar installer.)

VMware View – Implementing Idle User Auto-Logout

We are going live with out first public VMware View terminals this week (Wyse P25 “zero-clients”… nice units).  I had what I expected would be a easy list of “little jobs” to be completed before going live. Famous last words…

One item on the list was implementing an “idle user logout” process.  This process would detect when a View session had gone idle, and would disconnect the session automatically (preferably after prompting the user).  This disconnected session then would be logged out by View Manager after a fixed amount of time.

This proved rather more difficult than I had predicted.  I tried several solutions before arriving at one that worked.  Among the failed solutions:

  • Using Group Policy to configure Remote Desktop Session Manager idle session limits.  The View configuration documents imply that this should work, but it does not.  I expect that the policies would be effective if you were connecting to your View desktops using RDP, but PCoIP sessions just will not disconnect automatically (at least, they would not for me).
  • Using the Windows Task Scheduler to configure a disconnect script that will trigger on idle.  This did not work for two reasons.  First, the Task Scheduler only evaluates for idle conditions every 15 minutes.  Second, for the Task Scheduler, “idle” means not only that the user is not directing mouse and keyboard to the computer, but that the CPU also is not doing anything.  As a result, we could not get consistent auto-logout times.

The solution that we settled on involved the use of a custom screensaver developed by the “Grim Admin”.  “ScreenSaver Operations”:

This is a great little utility that accomplishes what the “WinExit” screensaver used to for Win XP.  (WinExit cannot easily be used on Win7, and is a bit hostile to 64-bit Windows).  Screensaver Operations has a well-written README describing the use of registry entries to control the screensaver globally (i.e. for all users on the computer).  I set these registry operations as Group Policy Preferences, and we are in business.

Two slight complications… since the screensaver is 32-bit, you need to use the “sysnative” filesystem redirector if your want the screensaver to trigger 64-bit executables.  In our case, I wanted the screensaver to launch “tsdiscon.exe” (to disconnect the View session), so I had to use the path:
Additionally, you will need to specify the full path to the screensaver in the Group Policy dialogs (i.e. %SystemRoot%SysWOW64Screensaver Operations.scr).  If you fail to do so, the screensaver will appear to be configured in the Control Panel, and you will be able to preview it by clicking the “preview” button, but the screensaver WILL NEVER START.

Ashamedly I will admit that this little challenge too much longer to accomplish than it should have.  No wonder lab managers burn out so easily.