VMware Performance Charts broken again – fix your connection string.

Following upgrade of our Virtual Center server to the vSphere 5 version, we have been struggling with crashing services and memory exhaustion.  Well, the server got an upgrade from 8Gb to 16Gb of RAM this am, so it is now swimming in excess memory.  Despite this, the Perf Charts in vCenter have gone dead again.

We were seeing errors similar to those described here:
We got the “”perf charts service experienced an internal error” when looking at the Performance tab for any object in the vSphere Client.  A look at the latest “vctomcat-stderr[…].log” file in “C:Program FilesVMwareInfrastructuretomcatlogs” reveals JDBC connection errors.  The damn service is trying to connect to the standby partner of our SQL database mirrored pair!  So where are the connection strings stored for this service.  Well, no thanks to VMware documentation, I discovered the connection string stored here:
C:ProgramDataVMwareVMware VirtualCentervcdb.properties

All I had to do was append “;failoverPartner=[hostName]” to the line starting with “url=”, then restart the tomcat service (the “VMware VirtualCenter Management WebServices” service).  Viola… performance reports are back.

Now back to fixing everything else that is broken… also known as “everything”.


Windows file server aliases, NetBIOS over TCP, and Incomplete Documentation

It was inevitable… some people have started to notice that their bloody Multi-Function Devices (the overpriced beasts formerly known as “printer/scanner/copiers”) no longer can scan to our file server.  Why?  Because we got all smarty pants and tried to disabled “NetBT” (NetBIOS over TCP) for access to our new file server.  We figured “Hey, CIFS over TCP has been available since, what, 1999?  We won’t have any problems with back level clients!”.  Rubes.  Turns out a lot of MFDs still require NetBT.  Rather than be a pain, I have decided to enable NetBT on the new server.  After all, it is mainly the laggards that suffer the consequences of back-level clients (and besides, if I don’t do this I will either have to go update their firmware myself, or locate some mythological pile of gold to pay for new MFDs).

The tricky bit for us was that our clients are using a name other than the server’s primary name to access the server.  Modern clients don’t care.  SMB2-capable systems only require that the server name being used has a valid matching DNS entry, and if you want Kerberos to work, you need to add a matching Service Principal Name (SPN).  We tested that our a long time back, and had no SMB2 client issues at go live.  SMB1 is a bit more fussy… you need to tell the server not to get bent out of shape when the requested server name is not the primary server name by adding the “DisableStrictNameChecking” DWORD in the HLKMSystemCurrentControlSetServicesLanManServerParameters registry key.  We picked up on that one on migration night.

But now, after activating NetBIOS over TCP, client systems still are giving us grief.  Upon connection, we can see the NetBIOS Session Service reporting “Called Name Not Present” in a wire trace.  What gives?  DNS, SPN, and WINS entries are not enough for good-old LanMan?  Sounds like the server needs some additional convincing.

It turns out the key is in “optional names”.  When I added a Reg_multi_sz entry named “OptionalNames” to the HLKMSystemCurrentControlSetServicesLanManServerParameters registry key, populated it with the alias name of our file server, activated NetBIOS over TCP, then rebooted, I found that the server started granting connection requests on TCP Port 139 (the legacy NetBT CIFS port).  Horray!  As an added bonus, our windows server is now registering both its primary name and its supported alias in WINS and DNS… no need for static entries.  Now to see if the MFDs are satisfied…

It really is a pity that there is no one article on this subject in TechNet.  It appears that the folks at Microsoft have little belief that anyone would be so asinine as to want to have a file server with an alias, and support for CIFS over NetBT, and support for CIFS over TCP, and support for modern SMB2 clients.  I’ll bet a consultants would say “Gee, I have never seen this before”.  We get that a lot.

Articles that helped during this troubleshooting process: