Windows Backup Performance Testing with PowerShell

While developing our new Windows file services infrastructure, we wanted to test our pre-production platform to see if there are any file server-side bottlenecks that will cause unacceptable delays in backup processing. Here are UVM we still are using EMC Networker for Enterprise backup (no comments on our satisfaction with EMC will be provided at this time). EMC provides a tool “uasm.exe” that is used at the core of the “save.exe” and “recover.exe” commands on the backup client. If we use “uasm.exe” to backup all of the file server data to “null”, it is possible that we will be able to detect disk, HBA, and other local I/O bottlenecks before they bite us in production.

Since Networker will break up our file server into multiple “save sets”, and run a user-definable number of save set backup processes in parallel, it also is important for us to determine the required number of parallel backup processes required to complete backup in a timely fashion. Thus, we want to run several parallel “uasm.exe” processes in our tests.

PowerShell, with the assistance of “cmd.exe”, and some finesses, can get this job done. Hurdles I ran into while scripting this test follow:

  1. During development, PowerShell consumed huge amounts of CPU while redirecting uasm.exe output to the PowerShell $null object. Interestingly, previous tests using uasm.exe with cmd.exe did not show this problem. To fix this, each uasm job is spawned from a one-line cmd.exe “bat” script, which is included below.
  2. Remember that PowerShell uses the null object “$null”, but that cmd.exe uses the handle “nul” (with one “L”). If you redirect to “null”, you will soon fill up your disk with a file named “null”.
  3. When wanted to examine running jobs, it was difficult to determine which directory a jobs was working on. This was because I initially created a scriptblock object and passed parameters to it when starting a job. For example:
    [scriptblock] $sb = {
    $uasmBlock = {
    	param ([string]$sPath)
    	[string[]] $argList = '/c','c:localscriptsuasm_cmd.bat',$sPath
    	& cmd.exe $argList
    }
    $jobs += start-job -Name $myJob -ScriptBlock $sb -ArgumentList $dir1
    

    However, when inspecting the job object’s “command” property, we see “$sPath” in the output. We want the variable expanded. How to do this? Create the scriptblock object in-line when starting the job:

    [string] $cmd = '& cmd.exe "/c","c:localscriptsuasm_cmd.bat",' + $dir
    $jobs += Start-Job -Name $jobName `
    	-ScriptBlock ([scriptblock]::create($cmd))
    

    This makes for more compact code, too.

  4. To check on jobs that have completed, I create an array named “$djs” (Done Jobs), populated by piping the $jobs array and filtering for “completed” jobs. I inspect $djs to see if jobs are present. In my first pass, I used the check:
    if ($djs.count -gt 0)

    Meaning, continue if there is anything in the array $djs. However, this check did not work well because output from the $jobs object would put a null item in $djs on creation, meaning that if there were no running jobs, $djs would still have a count of one! I fixed this by changing the test:

    if ($djs[0] -ne $null)

    Meaning, if the first entry in $djs is not a null object, then proceed.

The full script follows:

#uasm_jobQueue.ps1, 2011-09-30, author: J. Greg Mackinnon
#Tests performance of disk when accessed by Networker backup commands.
#   Creates a queue of directories to test ($q), then uses external command 
#   "uasm.exe" to backup these directories to null.
#Change the "$wp" variable to set the number of uasm 'worker processes' to be 
#   used during the test.
#Note: PowerShell $null object causes very high CPU utilization when used for
#   this purpose.  Instead, we call "uasm_cmd.bat" which uses the CMD.exe 'nul'
#   re-director.  'nul' does not have the same problems as $null.

set-psdebug -strict

[int] $wp = 4

# Initialize the log file:
[string] $logfile = "s:uasm_test.log"
remove-item $logfile -Force
[datetime] $startTime = Get-Date
[string] "Start Time: " + $startTime | Out-File $logfile -Append

##Create work queue array:
# Add shared directories:
[String[]] $q = gci S:shared | ? {$_.Attributes.tostring() -match "Directory"}`
	| sort-object -Property Name | % {$_.FullName}
# Add remaining targets to queue:
$q += 'H:','I:','J:','K:','L:','M:','S:sis','S:software','s:r25'
	
[int] $dc = 0			#Count of completed (done) jobs.
[int] $qc = $q.Count	#Initial count of jobs in the queue
[int] $qi = 0			#Queue Index - current location in queue
[int] $jc = 0			#Job count - number of running jobs
$jobs = @()				#Jobs array - intended to contain running PS jobs.
	
while ($dc -lt $qc) { # Completed jobs is less than total jobs in queue
	# Keep running jobs until completed jobs is less than total jobs in queue, 
	#  and our queue count is less than the current queue index.
	while (($jobs.count -lt $wp) -and ($qc -gt $qi)) { 
		[string] $jobName = 'qJob_' + $qi + '_';
		[string] $dir = '"' + $q[$qi] + '"'
		[string] $cmd = '& cmd.exe "/c","c:localscriptsuasm_cmd.bat",' + $dir
		#Start the job defined in $cmd string.  Use this rather than a pre-
		#  defined scriptblock object because this allows us to see the expanded
		#  job command string when debugging. (i.e. $jobs[0].command)
		$jobs += Start-Job -Name $jobName `
			-ScriptBlock ([scriptblock]::create($cmd))
		$qi++ #Increment the queue index.
	}
	$djs = @(); #Completed jobs array
	$djs += $jobs | ? {$_.State -eq "Completed"} ;
	# $djs array will always have a count of at least 1.  However, if the 
	#    first entry is not empty (null), then there must be completed jobs to
	#    be retrieved.
	if ($djs[0] -ne $null) { 
		$dc += $djs.count;
		$djs | Receive-Job | Out-File $logfile -Append; #Log completed jobs
		$djs | Remove-Job -Force;
		Remove-Variable djs;
		$jobs = @($jobs | ? {$_.State -eq "Running"}); #rebuild jobs array.
	}
	Start-Sleep -Seconds 3
}


# Complete logging:
[datetime] $endTime = Get-Date
[string] "End Time: " + $endTime | Out-File $logfile -Append 
$elapsedTime = $endTime - $startTime
[string] $outstr =  "Elapsed Time: " + [math]::floor($elapsedTime.TotalHours)`
	+ " hours, " + $elapsedTime.minutes + " minutes, " + $elapsedTime.seconds`
	+ " seconds."
$outstr | out-file -Append $logfile

The “uasm_cmd.bat” file called in the above code block contains the following one line:

"c:program fileslegatonsrbinuasm.exe" -s %1 > nul

Advertisements

Migrating from NetApp to Windows File Servers with PowerShell – part 4

In our final entry in the Windows File Server management with PowerShell series, we have a quick look at managing shares using powershell.

The one feature of the NetApp that I will miss most was the automatic home share. Similar to Samba, you can tell the filer a set of directories under which user home directories are located. The filer will create these shares on-demand for a connecting user. Even under Server 2008 R2, Windows has no such capability. So what is the solution? Well, we have to create a share for every user. Augh! Fortunaltely, a beefy box can handle all these shares fairly gracefully, and with PowerShell, creating the shares is fast.

Some logic we added to this script includes probing for existing shares using “net share” before we attempt to create the share. If output of “net share” shows that the physical directory path is not correct, we delete the existing share and re-create it. If the share exists, and has the correct path, we don’t touch it. If the share does not exist, we create it.

I learned a fair bit about invoking external commands with PowerShell while writing this script. To summarize, PowerShell is not much like cmd.exe. When invoking a command with multiple parameters, you very often need to separate the parameters into array components, otherwise PowerShell does not know where to delimit the parameters. For example, let’s say we want to run the command:

net share jsmith=h:homes1jjsmith /GRANT:DOMAINjsmith,FULL

The username and path are to be populated using variables in the script, so we try:

& net share ($uname)=($path) /GRANT:DOMAIN($uname),FULL

Looks good, right? Unfortunately, not. The PS invocation command (“&”) parses the command parameters incorrectly, and thus passes a bunch of trash to the “net.exe” external command, and thus puts nasty red errors all over your console.

What we need to do is put the command parameters into an array, and feed that to the PS invocation command:

[string] $cmd = 'net.exe'
[string[]] $params = 'share', ($uname + '=' + $path),`
     ('/GRANT:DOMAIN' + $uname + ',FULL')
& $cmd $params

Or perhaps more readably:

[string] $cmd = 'net.exe'
[array] $params = @()
[string] $params += 'share'
[string] $params += $uname + '=' + $path
[string] $params += '/GRANT:DOMAIN' + $uname + ',FULL'
& $cmd $params

I do something similar below, showing once again that there is more than one way to code a cat (and that some cats are better looking than others):

# Shares all directories at the third directory level of the 
#	volumes specified in $homeVols.
# Assumes that all subdirectories will have a name matching an 
#	existing CAMPUS user ID.

Set-PSDebug -Strict

[string[]] $homeVols = @()
$homeVols = "h:","i:","j:","k:","l:","m:","n:"

foreach ($vol in $homeVols) {
    $subDirs1 = gci $vol | ? {$_.Attributes.tostring() -match "Directory"}
    foreach ($dir1 in $subDirs1) {
        $subDirs2 = gci $dir1.FullName | ? {$_.Attributes.tostring() `
			-match "Directory"}
        foreach ($dir2 in $subDirs2) {
            $uhomes = gci $dir2.FullName | ? {$_.Attributes.tostring() `
				-match "Directory"}
            if ($uhomes.count -ge 1) { foreach ($uhome in $uhomes) {
				[bool]$doShare = $false
                [string] $cmd = "net.exe"
                [string] $arg1 = "share"
				[string] $arg2 = ($uhome.name)
				[string] $out = & net $arg1, $arg2 2>&1
				[string] $homePath = $vol + $dir1.name + "" + $dir2.name + "" `
					+ $uhome.name				
				[string] $homePathRX = $vol + "" + $dir1.name + "\" `
					+ $dir2.name + "\" + $uhome.name
				if ($out -match "does not exist") {
					write-output "Share does not exist for $homePath"
					[bool] $doShare = $true
				} elseif ($out -notmatch $homePathRX) {
					write-output "Share not defined correctly for path $homePath"
					[bool] $doShare = $true
					#Delete existing share
					[string] $arg3 = "/DELETE"
					[string] $out = & net $arg1, $arg2, $arg3 2>&1
				}
				
				if ($doShare) {
	                [string] $arg2 = ($uhome.name + "=" + $homePath)
	                [string] $arg3 = "/Grant:CAMPUS" + $uhome.name + ",FULL"
	                #write-host "about to share: " $uhome.name
	                # The next line will call the "net" command to share the 
					#	current home directory.
	                # If no mapping can be made between the directory name and 
					#	an AD user, "No Mapping" will be in the error output.
					
	                [string] $out = & net $arg1, $arg2, $arg3 2>&1   
	                if ($out -match "No Mapping") {
						Write-Output `
							"User was not defined for share on path $homePath"
	                   $uhome.name | out-file -FilePath h:noMapUsers.txt -append 
	                } else {
						Write-Output "Successfully shared $homePath"
					}
				}
            }} #End if ($uhomes.count -ge 1)
        } #End $dir2 in $subDirs2
    } #End $dir1 in $subDirs1
} #End $vol in $homeVols

Migrating from NetApp to Windows File Servers with PowerShell – part 3

In our old NetApp environment we utilized quotas extensively. We want to map our NetApp quotas to Windows File System Resource Manager quotas in a failrly seamless fashion. Fortunately, with PowerShell, this is almost easy. Microsoft gives us the FSRM.FsrmQuotaManager and FSRM.FsrmQuotaTemplateManager “COM” object classes, documented here:
http://msdn.microsoft.com/en-us/library/bb613257(VS.85).aspx
http://msdn.microsoft.com/en-us/library/bb613280(v=VS.85).aspx
PowerShell works with these with ease.

I got a jump start on these scripts from my brethren on the Internet. Perhaps the most helpful post was Ben Lye’s entry on “Simple-talk”:
http://www.simple-talk.com/sysadmin/exchange/implementing-windows-server-2008-file-system-quotas/

First task… create FSRM quota templates that are similar to our NetApp qtree quotas. We don’t need to use “templates”, but it makes management a bit easier, so we will:

#Creates simple hard-limit directory quotas of the GB values specified in $tholds.

Set-PSDebug -Strict
$qtm = New-Object -com Fsrm.FsrmQuotaTemplateManager

[int[]] $tholds = 25,50,100,150,200,300,400,600,800,1200

foreach ($thold in $tholds) {
	$templ = $qtm.CreateTemplate()
	[string] $tName = [string]$($thold) + " Gb Shared Directory"
	$templ.Name = $tName
	[long] $tLimit = $thold * 1073741824
	$templ.QuotaLimit = $tLimit
	$templ.Commit()
	Clear-Variable templ
	Remove-Variable templ
}

Now we want to map our NetApp quotas to the new FSRM quotas. Challenges here are:

  1. extracting structured data from the NetApp “quotas” file.
  2. mapping NetApp quotas to the nearest-larger FSRM quota template.

The first challenge is handled using the .NET RegEx and String classes. It was a hack job, but it worked.

The second challege was handled using the powershell “switch” command. The one roadblock I ran into here was caused by a failure to cast a variable as the appropriate data type. For those who did not already know, you never, uh, I mean always should cast your valiables. Why? Look at the switch in the code below… what do you think happens when you compare the un-cast pipeline variable ($_) to “100”? Did you answer “geez dude, I don’t know, that’s a gibberish.”? Join the club. Did you answer “PowerShell does a string comparison, dude! $outQuota was an array of strings! What the heck were you thinking? Do you want 1000 to be considered a larger value than 25?”? If so, then you are correct. If you want to do mathematical comparison, you must cast your valiables an numbers. And by the way, do not cast large numbers as “[int]”. Why? Because [int] will not exceed 32-bits in length. Use [long] or [double] instead.

# Applies quota templates to s:shared subdirectories, mapped from input file
# specified in $inQuota (the NetApp quotas file)
# NetApp quota limits are matched to the closest larger Windows quota template.

Set-PSDebug -Strict

#parse NetApp quota file:
$inQuota = Get-Content '\filesetc$quotas'
$outQuota = @()
#following loop strips all but the shared directory name and hard limit from input:
foreach ($line in $inQuota) {
	$arrLine = @()
	if ($line -match "/vol/vol2/") {
	[string] $ind0 = [regex]::split($line," tree ") | Select-Object -Index 0 |`
		% {$_.trim()} | % {$_.replace("/vol/vol2/", "")} | % {$_.replace('"', '')} 
	[string] $ind1 = [regex]::split($line," tree ") | Select-Object -Index 1 |`
		% {$_.split("G")} | select-object -index 0 | % {$_.trim()}
	$outQuota += ,@($ind0,$ind1)
	}
}
Remove-Variable inQuota

#instantiate quota manager:
$qm = New-Object -com Fsrm.FsrmQuotaManager

foreach ($quota in $outQuota) {
	#determine Windows Quota Template that is closest to NetApp quota:
	[long] $qVal = $quota[1]
	switch ($qVal) {
		{$_ -le 1200} {[string]$qtl = "1200"}
		{$_ -le 800} {[string]$qtl = "800"}
		{$_ -le 600} {[string]$qtl = "600"}
		{$_ -le 400} {[string]$qtl = "400"}
		{$_ -le 300} {[string]$qtl = "300"}
		{$_ -le 200} {[string]$qtl = "200"}
		{$_ -le 150} {[string]$qtl = "150"}
		{$_ -le 100} {[string]$qtl = "100"}
		{$_ -le 50} {[string]$qtl = "50"}
		{$_ -le 25} {[string]$qtl = "auto"}
	}
	# values 25 or smaller use an autoapply template, so skip these...
	if ($qtl -notmatch "auto") { 
		# create a quota object by retrieving an existing auto-apply quota, then update:
		[string] $deptDir = 's:shared' + $quota[0]
		$qObj = $qm.GetQuota($deptDir)
		[string] $templ = $qtl + " Gb Shared Directory"
		$qObj.ApplyTemplate($templ)
		$qObj.Commit()
		Clear-Variable qObj
		Remove-Variable qObj
	}
}

Now moving on to user home directory quotas… we used “AutoApplyTemplates” for this situation. AutoApply templates force afixed quota onto all new subdirectories of a parent. In this way, we avoid having to set a template manually for every new user home directory. Our file server contains six volumes (h:-m:), each containing a “homes1” subdirectory, each of which in turn contins subdirectories 0-9, a-z. Individual home directories are nested within these. We needed to apply AutoApplyTemplates to these 0-9,a-z directories. Easy…

# Applies an Auto-Apply quota to all directories at the second directory level 
# of the volumes specified in $homeVols.

Set-PSDebug -Strict

[string[]] $homeVols = @()
$homeVols = "h:","i:","j:","k:","l:","m:"

$qm = New-Object -com Fsrm.FsrmQuotaManager

foreach ($vol in $homeVols) {
    $subDirs1 = gci $vol | ? {$_.Attributes.tostring() -match "Directory"}
    foreach ($dir1 in $subDirs1) {
        [array] $subDirs2 = gci $dir1.FullName | ? {$_.Attributes.tostring() -match "Directory"}
        if ($subDirs2.count -ge 1) { foreach ($dir2 in $subDirs2) {
            [string] $uPath = ($vol + $dir1.name + '' + $dir2.name)
			[string] $template = "10 Gb Home Directory"
            $quota = $qm.CreateAutoApplyQuota($template,$uPath)
            $quota.Commit()
            clear-variable quota
            remove-variable quota
        }} #End if ($subDirs2.count -ge 1)
    } #End $dir1 in $subDirs1
} #End $vol in $homeVols

On the NetApp, we had a single volume that hosted users with a larger home directory quota. Under Windows, we can simply set a different directory quota for these individuals which overrides the AutoApplyTemplate. I have done this for the existing 25Gb quota users already, using PowerShell:

# Applies a new quota to the third-level directory of the volumes specified in $homeVols, 
#  under the path $homeVol:homes2[a-z][samAccountName]

Set-PSDebug -Strict

[string[]] $homeVols = @()
$homeVols = "n:","m:"

$qm = New-Object -com Fsrm.FsrmQuotaManager
$qtm = New-Object -com Fsrm.FsrmQuotaTemplateManager

foreach ($vol in $homeVols) {
    [string] $dir1 = $vol + "homes1"
    $subDirs2 = gci $dir1 | ? {$_.Attributes.tostring() -match "Directory"}
    foreach ($dir2 in $subDirs2) {
        [array] $uhomes = gci $dir2.FullName | ? {$_.Attributes.tostring() -match "Directory"}
        if ($uhomes.count -ge 1) { foreach ($uhome in $uhomes) {
            [string] $uHomePath = ($dir1 + '' + $dir2.name + '' + $uhome.name)
            $quota = $qm.GetQuota($uHomePath)
            $quota.ApplyTemplate("25 Gb Home Directory")
            $quota.Commit()
            clear-variable quota
            remove-variable quota
        }} #End if ($uhomes.count -ge 1)
    } #End $dir2 in $subDirs2

} #End $vol in $homeVols

And that is all I have for quotas for now. We will need to enhance our templates with threshold alerts and notifications in the future. This could prove interesting as the IFsrmObject.IFsrmQuotaBase class looks a bit challenging to deal with:
http://msdn.microsoft.com/en-us/library/bb613248(v=VS.85).aspx

Migrating from NetApp to Windows File Servers with PowerShell – part 2

Previously we saw how PowerShell and RoboCopy can be used to sync multi-terabyte file shares from NetApp to Windows. What I did not tell you was that this script choked and died horribly on a single share in our infrastructure. You may have seen it commented out in the previous script? “#,’R25′”?

CollegeNet Resource25… my old enemy. These clowns worked around a bug in their product (an inability to read an open text column in an Oracle DB table) by copying every text row in the database to its own file on a file server, and to make matters worse they copy all of the files to the same directory. Why is this bad? Ever try to get a directory listing on a directory with 480,000 1k files? It’s bad news. Worse, it kills robocopy. Fortunately, we have a workaround.

The archive utility “7-zip” is able to wrap up the nasty directory into a single small file, which we then can unpack on the new file server. Not familiar with 7-Zip? For shame! Get it now, it’s free:
http://www.7-zip.org/

7-zip ignores most file attributes, which seems to speed up the copy process a bit. Using robocopy, ouy sync operation would either run for hours on this single directory, or just hang up forever. With 7-zip, we get the job done in 30 minutes. Still slow, but better than never.

Troublesome files are found in the R25 “text_comments” directory, a subdirectory of “text”. We have prod, pre-prod, and test environments, and so need to do a few separate 7-zip archives. Note that a little compresson goes a long way here. When using “tar” archives, my archive was several gb in size. With the lowest level of compression, we squeeze down to only about 14 Mb. How is this possible? Well, a lot of our text comment files were empty, but uncompressed they still take up one block of storage. Over 480,000 blocks, this really adds up.

Code snippet follows.

#Sync R25 problem dirs

Set-PSDebug -Strict

# Initialize the log file:
[string] $logfile = "s:r25Sync.log"
remove-item $logfile -Force
[datetime] $startTime = Get-Date
[string] "Start Time: " + $startTime | Out-File $logfile -Append

function zipit {
	param ([string]$source)
	[string] $cmd = "c:localbin7za.exe"
	[string] $arg1 = "a" #add (to archive) mode
	[string] $arg2 = join-path -Path $Env:TEMP -ChildPath $($($source | `
		Split-Path -Leaf) + ".7z") # filespec for archive
	[string] $arg3 = $source #spec for source directory
	[string] $arg4 = "-mx=1" #compression level... minimal for performance
	#[string] $arg4 = "-mtm=on" #timestamp preservation - commented out for perf.
	#[string] $arg5 = "-mtc=on"
	#[string] $arg6 = "-mta=on"
	#invoke command, route output to null for performance.
	& $cmd $arg1,$arg2,$arg3,$arg4 > $null 
}

function unzipit {
	param ([string]$dest)
	[string] $cmd = "c:localbin7za.exe"
	[string] $arg1 = "x" #extract archive mode
	[string] $arg2 = join-path -Path $Env:TEMP -ChildPath $($($dest | `
		Split-Path -Leaf) + ".7z")
	[string] $arg3 = "-aoa" #overwrite existing files
	#destination directory specification:
	[string] $arg4 = '-o"' + $(split-path -Parent $dest) + '"' 
	#invoke command, route to null for performance:
	& $cmd $arg1,$arg2,$arg3,$arg4 > $null 
	Remove-Item $arg2 -Force # delete archive
}

[String[]] $zips = "V3.3","V3.3.1","PRODWinXpText"
[string] $sourceD = "\filesr25"
[string] $destD = "s:r25"

foreach ($zip in $zips) {
	Get-Date | Out-File $logfile -Append 
	[string] "Compressing directory: " + $zip | Out-File $logfile -Append 
	zipIt $(join-path -Path $sourceD -ChildPath $zip)
	Get-Date | Out-File $logfile -Append 
	[string] "Uncompressing to:" + $destD | Out-File $logfile -Append
	unzipit $(Join-Path -Path $destD -ChildPath $zip)
}

Get-Date | Out-File $logfile -Append 
[string] "Syncing remaining files using Robocopy..." | Out-File $logfile -Append
$xd1 = "\filesr25V3.3" 
$xd2 = "\filesr25V3.3.1" 
$xd3 = "\filesr25PRODWinXPtext"
$xd4 = "\filesr25~snapshot"
$roboArgs = @("/e","/copy:datso","/purge","/nfl","/ndl","/np","/r:0","/mt:4",`
	"/b",$sourceD,$destD,"/xd",$xd1,$xd2,$xd3,$xd4)

& robocopy.exe $roboArgs

Get-Date | Out-File $logfile -Append 
[string] "Done with Robocopy..." | Out-File $logfile -Append

# Complete logging:
[datetime] $endTime = Get-Date
[string] "End Time: " + $endTime | Out-File $logfile -Append 
$elapsedTime = $endTime - $startTime
[string] $outstr =  "Elapsed Time: " + [math]::floor($elapsedTime.TotalHours)`
	+ " hours, " + $elapsedTime.minutes + " minutes, " + $elapsedTime.seconds`
	+ " seconds."
$outstr | out-file -Append $logfile

Migrating from NetApp to Windows File Servers with PowerShell – part 1

We are retiring our NetApp filer this year. It was nice knowing you, NetApp. Thank you for the no-hassle performance, agile volume management, and excellent customer support. We will not miss your insane pricing, and subtle incompatibilities with modern Windows clients.

In this multi-part series, I will be sharing PowerShell code developed to assist with our migration. In part one, we will look at bulk copy operations with RoboCopy. In part 2, we will look at a situation where RoboCopy fails to get the job done. In future parts, we will look at automated share and quota management and migration.

Migrating large amounts of data off a NetApp is not particularly straightforward. The only real option we have is to copy data off of the filer CIFS shares to their Windows counterparts. Fortunately, with the multi-threading power utility “robocopy” we can move data between shares pretty quickly. Unfortunately, robocopy only multi-threads file copy operations, not directory search operations. So, while initial data transfers with robocopy take place really quickly, subsequent sync operations are slower than expected. MS also released a utility called “RichCopy” whish supports multi-thread directory searching, but this utility is not supported by MS, and has some significant bugs (i.e. it crashes all the time). What to do?

PowerShell to the rescue! Using PowerShell jobs, we can spawn off a separate robocopy job for each subdirectory of a source share, and run an arbitrary number of parallel directory copies. With some experimentation, I determined that I could run ten simultaneous robocopy operations without overwhelming CPU or disk channels on the filer. Under this arrangement, or file sync Window has been reduced from almost 48 hours to a mere 2.5 hours.

Some tricky bits in the development of this script where:

  • PowerShell jobs and job queuing are critical to completing this script in a timely fashion. Syntax for “start-job” is tricky. See my post on backup performance testing for more comments on working with jobs.
  • Robocopy fails top copy a number of source files. This is mitigated though the use of the “/b” switch (backup mode).
  • The PowerShell cmdlet “receive-jobs” fails to capture output from a variety of job commands unless you assign the job to an object. To reliably capture the output of commands within our jobs, I needed to assign the jobs to our $jobs array.
  • I needed to do some post processing on the log file. In doing so, I needed to find UNC paths for our source filer “\files”. It is important to remember that, when using regular expressions, “” is the escape character. So, to match for “”, we need to enter “\”. To match for “\” we need to enter “\\”, as in:
     get-content $logfile | select-string -Pattern "\\files" | ...
  • Initially I allowed the script to process only one top level directory at a time (i.e. Start with \filessoftware, and only proceed to \filesshared when “software” completes). The problem with this was, I was preventing the script from running an optimal job count. Furthermore, a single hung job could bring the whole script to a halt. To combat this, I start the script by building a master queue array “$q”, which holds all of the directories for which I am going to start a job. The result of using a master queue is a considerable improvement in sustained throughput.
  • When building an array with a loop (i.e. while…) you may have trouble with the first item added to the array if you do not initialize the array before starting to loop. In my case, I needed to initialize “[array]$jobs = @()” before using the array to hold job objects in the “while” loop. Failing to do so caused “$jobs” to become a single job object when the number of jobs was equal to one. Bad news, if you are expecting to use array properties such as $jobs.count, or to call in index of the object (i.e. $jobs[0]).
  • ISE programs like the native PowerShell ISE, or Quest PowerGUI make script development much easier. However, production environments are not the same as the debug environment, so keep these tips in mind:
    1. Log your script actions! Use lots of out-file calls. If you are feeling slick, you can enclose these in “if ($debug)” clauses, and set the $debug variable as a script parameter (which I did no do here).
    2. When running in production, watch the log file in real-time using “get-content -wait”. I know it is not a cool as the Gnu command “tail”, but it is close.
  • Scoping… careful of the “global” scope. Initially I modified the $jobs and $dc variables in the global scope from within the “collectJobs” function. This worked fine in my ISE and at the PowerShell prompt. However, when running as a scheduled task, these calls failed miserably. I changed the calls to use the “script” scope, and the script now runs as a scheduled task successfully.

Below is the script I developed for this job… it contains paths specific to our infrastructure, but easily could be modified. Change the “while ($jobcount -lt 10)” loop to set the number of simultaneous robocopy processes to be used by the script…

# FilerSync_jobQueue.ps1
# JGM, 2011-09-29
# Copies all content of the paths specified in the $srcShares array to 
# corresponding paths on the local server.
# Keeps data on all copy jobs in an array "$q".
# We will use up to 10 simultaneous robocopy operations.

set-psdebug -strict

# Initialize the log file:
[string] $logfile = "s:files_to_local.log"
remove-item $logfile -Force
[datetime] $startTime = Get-Date
[string] "Start Time: " + $startTime | Out-File $logfile -Append

# Initialize the Source file server root directories:
[String[]] $srcShares1 = "adfs$","JMP$","tsFlexConfig","software","mca","sis","shared"`
	#,"R25"
	#R25 removed from this sync process as the "text_comments" directory kills
	#robocopy.  We will sync this structure separately.
[String[]] $srcShares2 = "uvol_t1_1$q-home","uvol_t1_2$q-home","uvol_t1_3$q-home",`
	"uvol_t1_4$q-home","uvol_t1_5$q-home","uvol_t2_1$q-home",`
	"vol1$qtree-home"
	
[String[]] $q = @() #queue array

function collectJobs { 
#Detects jobs with status of Completed or Stopped.
#Collects jobs output to log file, increments the "done jobs" count, 
#Then rebuilds the $jobs array to contain only running jobs.
#Modifies variables in the script scope.
	$djs = @(); #Completed jobs array
	$djs += $script:jobs | ? {$_.State -match "Completed|Stopped"} ;
	[string]$('$djs.count = ' + $djs.count + ' ; POssible number of jobs completed in this colletion cycle.') | Out-File $logfile -Append;
	if ($djs[0] -ne $null) { #First item in done jobs array should not be null.
		$script:dc += $djs.count; #increment job count
		[string]$('$script:dc = ' + $script:dc + ' ; Total number of completed jobs.') | Out-File $logfile -Append;
		$djs | Receive-Job | Out-File $logfile -Append; #log job output to file
		$djs | Remove-Job -Force;
		Remove-Variable djs;
		$script:jobs = @($script:jobs | ? {$_.State -eq "Running"}) ; #rebuild jobs arr
		[string]$('$script:jobs.count = ' + $script:jobs.Count + ' ; Exiting function...') | Out-File $logfile -Append
	} else {
		[string]$('$djs[0] is null.  No jobs completed in this cycle.') | Out-File $logfile -Append
	}
}
	
# Loop though the source directories:
foreach ($rootPath in $srcShares1) {
    [string] $srcPath = "\files" + $rootPath # Full Source Directory path.  
	#Switch maps the source directory to a destination volume stored in $target 
    switch ($rootPath) {
        shared {[string] $target = "S:shared"}
        software {[string] $target = "S:software"}
        mca {[string] $target = "S:mca"}
        sis {[string] $target = "S:sis"}
        adfs$ {[string] $target = "S:adfs"}
        tsFlexConfig {[string] $target = "s:tsFlexConfig"}
        JMP$ {[string] $target = "s:JMP"}
        R25 {[string] $target = "S:R25"}
    }
    #Enumerate directories to copy:
	$dirs1 = @()
	$dirs1 += gci $srcPath | sort-object -Property Name `
		| ? {$_.Attributes.tostring() -match "Directory"} `
		| ? {$_.Name -notmatch "~snapshot"}
	#Copy files in the root directory:
	[string] $sd = '"' + $srcPath + '"';
	[string] $dd = '"' + $target + '"';
	[Array[]] $q += ,@($sd,$dd,'"/COPY:DATSO"','"/LEV:1"' )
	# Add to queue:
	if ($dirs1[0] -ne $null) {
		foreach ($d in $dirs1) {
			[string] $sd = '"' + $d.FullName + '"';
	    	[string] $dd = '"' + $target + "" + $d.Name + '"';
			$q += ,@($sd,$dd,'"/COPY:DATSO"','"/e"')
		}
	}
}
foreach ($rootPath in $srcShares2) {   
    [string] $srcPath = "\files" + $rootPath # Full Source Directory path.
	#Switch maps the source directory to a destination volume stored in $target 
    switch ($rootPath) {
        uvol_t1_1$q-home {[string] $target = "H:homes1"}
        uvol_t1_2$q-home {[string] $target = "I:homes1"}
        uvol_t1_3$q-home {[string] $target = "J:homes1"}
        uvol_t1_4$q-home {[string] $target = "K:homes1"}
        uvol_t1_5$q-home {[string] $target = "L:homes1"}
        uvol_t2_1$q-home {[string] $target = "M:homes1"}
        vol1$qtree-home {[string] $target = "J:homes2"}
    }
    #Enumerate directories to copy:
	[array]$dirs1 = gci -Force $srcPath | sort-object -Property Name `
		| ? {$_.Attributes.tostring() -match "Directory"}
	if ($dirs1[0] -ne $null) {
		foreach ($d in $dirs1) {
			[string] $sd = '"' + $d.FullName + '"'
			[string] $dd = '"' + $target + "" + $d.Name + '"'
			$q += ,@($sd,$dd,'"/COPY:DAT"','"/e"')
		}
	}
}

[string] $queueFile = "s:files_to_local_queue.csv"
Remove-Item -Force $queueFile
foreach ($i in $q) {[string]$($i[0]+", "+$i[1]+", "+$i[2]+", "+$i[3]) >> $queueFile }

New-Variable -Name dc -Option AllScope -Value 0
[int] $dc = 0			#Count of completed (done) jobs.
[int] $qc = $q.Count	#Initial count of jobs in the queue
[int] $qi = 0			#Queue Index - current location in queue
[int] $jc = 0			#Job count - number of running jobs
$jobs = @()

while ($qc -gt $qi) { # Problem here as some "done jobs" are not getting captured.
	while ($jobs.count -lt 10) {
		[string] $('In ($jobs.count -lt 10) loop...') | out-file -Append $logFile
		[string] $('$jobs.count is now: ' + $jobs.count) | out-file -Append $logFile
		[string] $jobName = 'qJob_' + $qi + '_';
		[string] $sd = $q[$qi][0]; [string]$dd = $q[$qi][1];
		[string] $cpo = $q[$qi][2]; [string] $lev = $q[$qi][3]; 
		[string]$cmd = "& robocopy.exe $lev,$cpo,`"/dcopy:t`",`"/purge`",`"/nfl`",`"/ndl`",`"/np`",`"/r:0`",`"/mt:4`",`"/b`",$sd,$dd";
		[string] $('Starting job with source: ' + $sd +' and destination: ' + $dd) | out-file -Append $logFile
		$jobs += Start-Job -Name $jobName -ScriptBlock ([scriptblock]::create($cmd))
		[string] $('Job started.  Incrementing $qi to: ' + [string]$($qi + 1)) | out-file -Append $logFile
		$qi++
	}
	[string] $("About to run collectJobs function...") | out-file -Append $logFile
	collectJobs
	[string] $('Function done.  $jobs.count is now: ' + $jobs.count)| out-file -Append $logFile
	[string] $('$jobs.count = '+$jobs.Count+' ; Sleeping for three seconds...') | out-file -Append $logFile
	Start-Sleep -Seconds 3
}
#Wait up to two hours for remaining jobs to complete:
[string] $('Started last job in queue. Waiting up to three hours for completion...') | out-file -Append $logFile
$jobs | Wait-Job -Timeout 7200 | Stop-Job
collectJobs

# Complete logging:
[datetime] $endTime = Get-Date
[string] "End Time: " + $endTime | Out-File $logfile -Append 
$elapsedTime = $endTime - $startTime
[string] $out =  "Elapsed Time: " + [math]::floor($elapsedTime.TotalHours)`
	+ " hours, " + $elapsedTime.minutes + " minutes, " + $elapsedTime.seconds`
	+ " seconds."
$out | out-file -Append $logfile

#Create an error log from the session log.  Convert error codes to descriptions:
[string] $errFile = 's:files_to_local.err'
remove-item $errFile -force
[string] $out = "Failed jobs:"; $out | out-file -Append $logfile
$jobs | out-file -Append $errFile
$jobs | % {$jobs.command} | out-file -Append $errFile
[string] $out = "Failed files/directories:"; $out | out-file -Append $errFile
Get-Content $logfile | Select-String -Pattern "\\files"`
	| select-string -NotMatch -pattern "^   Source" `
	| % {
		$a = $_.toString(); 
		if ($a -match "ERROR 32 ")  {[string]$e = 'fileInUse:        '};
		if ($a -match "ERROR 267 ") {[string]$e = 'directoryInvalid: '};
		if ($a -match "ERROR 112 ") {[string]$e = 'notEnoughSpace:   '};
		if ($a -match "ERROR 5 ")   {[string]$e = 'accessDenied:     '};
		if ($a -match "ERROR 3 ")   {[string]$e = 'cannotFindPath:   '};
		$i = $a.IndexOf("\f");
		$f = $a.substring($i);
		Write-Output "$e$f" | Out-File $errFile -Force -Append
	}

Mozilla Thunderbird – Implementing the ISP Hook in 5.0 and later

Out buddies at MozillaMessaging are at it again… new with Thunderbird 5.0, all of the “jar” files previously present in the Thunderbird installation directory have been collapsed into a single “omni.jar” file, apparently for program load optimization.

This all would be fine with me if the omni.jar were a “normal” zip file, and the previous jars were. Instead, this is an “optimized” jar, that is not recognized by 7-zip as a valid zip archive. It is not clear to me how the jar is “optimized”, or how to re-apply optimizations when modifying the original, nor do I particularly care as load optimization is of little concern to us… we are not operating with 10 year old equipment, for the most part, so who cares?

I have had to work around the problem by using “jar.exe” from the Java JDK. This program extracts and re-creates omni.jar in a way that Thunderbird does not mind. The resultant file size is about the same, too.

Another quirk of the new version is that the default “prefs.js” file, while present in omni.jar, is not copied into the resultant Program Files directory at install time. On a clean install, there is no default prefs.js! I had to populate a new prefs.js into the “core” installer directory, outside of the new “omni.jar”.

Finally, there are no more “localized/nonlocalized” directories in the installer, just that which is within “omni”, and that which is not. So, I put our prefs.js in .coredefaultsprofile (a new directory in the installer zip). Previously it was in .nonlocalizeddefaultsprofile. Likewise, our mailnews.js and ISP Hook “.rdf” files also go in “core” instead of “localized”.

Other than that, the RDF ISP Hook file that we are using is unchanged from the one documented here:
https://sharepoint.uvm.edu/sites/ad/distribution/appconfig/Thunderbird.aspx
Prefs.js is the same as in the SharePoint site, with the modifications noted here:
http://blog.uvm.edu/jgm/2009/12/15/thunderbird-3-re-implementing-the-isp-hook-for-customized-deployments/

What a bunch of gibberish talk… sorry for the dry post. I am guessing if you are a Thunderbird customizer, you will know what I am talking about. If not, you won’t be reading anyway.

On to the build script, in Windows BAT/CMD language:

REM Thunderbird customized build script for UVM.
REM Updated September 2011 for Thunderbird 5.0 support.
REM REQUIRES:
REM – 7z.exe, 7zr.exe and sed.exe in parallel "..bin" directory
REM – Unmodified Thunderbird installer in .source directory
REM – all required config files in .config directory
REM (including 7z control file, ISP Hook RDF file, and modified prefs.js)
REM – local JDK install with "jar.exe". Path to jar.exe will need to be updated in the jdk environment variable
REM OUTPUT: Fully modified Thunderbird installer in .Installer directory.
REM @echo on

set jdk="c:Program Files (x86)Javajdk1.6.0_27bin"

Echo Cleaning up old builds…
del .Installer*.exe
rmdir /s /q .build
set /P tbver=Enter Thunderbird version number to build (i.e. "6.0.2"):

Echo Extracting setup files from OEM Installer…
mkdir .buildtemp
..bin7zr x .source*.exe -o.build

Echo Extracting omni.jar contents…
mkdir .buildtemp
cd .buildtemp
%jdk%jar.exe xf ..coreomni.jar

Echo modifying messenger functions…
……binsed.exe –binary "s/NewMailAccount(msgWindow, okCallback);/MsgAccountWizard(okCallback);/" .chromemessengercontentmessengermsgMail3PaneWindow_new.js
MOVE /Y .chromemessengercontentmessengermsgMail3PaneWindow_new.js .chromemessengercontentmessengermsgMail3PaneWindow.js

Echo modifying default mailnews preferences…
……binsed.exe –binary "s/try_ssl", 0)/try_ssl", 2)/" .defaultsprefmailnews_new.js
MOVE /Y .defaultsprefmailnews_new.js .defaultsprefmailnews.js

Echo moving UVM modified prefs.js into place (note that this file is not actually used by Thunderbird!)
copy /Y ….configprefs.js .defaultsprofileprefs.js

Echo Repacking omni.jar…
del /f /q ..coreomni.jar
%jdk%jar.exe cf ..coreomni.jar *

Echo Copying UVM Custom ISP file to source…
cd ….
mkdir .buildcoreispen-US
copy /Y .configUVMMail.rdf .buildcoreispen-USUVMMail.rdf
Echo Copying UVM default prefs.js to core directory (tbird no longer has a prefs.js by default, but it will be used if present)…
mkdir .buildcoredefaultsprofile
copy /Y .configprefs.js .buildcoredefaultsprofileprefs.js

Echo Deleting temporary files that should not be present in the installer…
rmdir /s /q .buildtemp

Echo Repackaging Thunderbird installer…
..bin7zr a .InstallerUVM_Thunderbird_setup_%tbver%.7z .build*
copy /b ..bin7zS.sfx + .configconfig.txt + .InstallerUVM_Thunderbird_setup_%tbver%.7z .InstallerUVM_Thunderbird_setup_%tbver%.exe

Echo Cleaning up installation source…
del /s /f /q .build*.*
rmdir /s /q .buildcore
rmdir /s /q .build
del /f /q .InstallerUVM_Thunderbird_setup_%tbver%.7z

SharePoint site use confirmation URL

We have been using the SharePoint automatic site usage confirmation and deletion of Windows SharePoint Services 3.0. An annoying aspect of this feature (and there are many) is that if a site that was use in use gets deleted, and we subsequently restore the site, the site still is not confirmed as in use and will get deleted on the next site deletion timer job.

To combat this, we need to acknowledge the site as in use manually. Unfortunately, MS does not document the site usage confirmation URL anywhere. Fortunately, I was able to dig it out eventually:

Just use the following URL as a template, and substitute the name of the site to confirm in place of [sitename]:
https://sharepoint.uvm.edu/sites/%5Bsitename%5D/_layouts/useconfirmation.aspx