Discovering Ports in Use on your network:

As part of our firewall reimplementation, I have been struggling with understanding the exact port needs of the services on our hosts. Previously, I discussed a procedure for discovering all network shares being served by the servers you manage:

http://www.uvm.edu/~jgm/wordpress/?p=96

Today, I used this procedure as a jumping off point for discovering all servers using high-order RPC ports, and the RPC end-point mapper. I followed the procedure above to discover all available hosts in a subnet. Next we use the excellent SysInternals tool “PSExec” to gather “netstat” information on this list of hosts. Here is the command:

for /f %c in (availablehosts.txt) do echo %c >> epmsys2.txt && psexec.exe \%c -e netstat -ano | find “135” >> epmsys2.txt

Taking it apart…

• Start a “for” loop for each server listed in the “available hosts” file.
• Start each pass though the loop by “echoing” the host name, and appending the output to a capture file.
• Next, use PSEXec to execute “netstat” on each host. Use “-e” to reduce resources used on the target host (does not load user environment remotely).
• Pipe netstat output though “find”. Filter for port 135 (the DCE RPC Endpoint Mapper).
• Direct output from psexec/netstat to the same file.

I ran this command two times… once for TCP Port 135, and then again for TCP port 6150 (which is the first “high order” RPC port available on our servers). The result is a file content like this:

WINUPDATE.campus.ad.uvm.edu
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 692
printers.campus.ad.uvm.edu
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 692
TCP 132.198.102.14:135 132.198.92.189:2917 TIME_WAIT 0
TCP 132.198.102.14:135 132.198.92.189:2918 TIME_WAIT 0
TCP 132.198.102.14:135 132.198.92.189:2919 TIME_WAIT 0
TCP 132.198.102.14:135 132.198.92.189:2920 TIME_WAIT 0

 

Every server listens with the EPM, but only a few have active connections. These are the ones that are actually using the service. As we expected, the Domain Controllers have many active EPM connections. What I did not expect is that the print server is also very busy… I wonder why?

Microsoft Network Load Balancing on VMWare ESX Server

I have been working on implementing the new “Terminal Services Gateway” service that will be released with Server 2008.  In order to cluster TS Gateways, we need to have a network load balancing solution in place.  Hardware solutions are supported, but getting access to those would be a pain.  Thus, I am back to looking at Microsoft Network Load Balancing services.

My last attempt as using NLB ran into some troubles… NLB-fronted Server 2003 terminal servers were pokey at best.  This time, I thought I should look at the implications of running Microsoft NLB on ESX server, which is the platform I am using for the TS Gateways.

Unsurprisingly, I learned some new things…

• Microsoft NLB is not supported by VMWare:
  http://pubs.vmware.com/vi301/mscs/wwhelp/wwhimpl/common/html/wwhelp.htm?context=mscs&file=mscs_intro.3.4.html
  (Configuration assistance is available only for “Microsoft Cluster Services” (MSCS)).
• When using NLB on ESX, it is recommended that you use “multicast” mode.  This is mentioned several places:

• http://pubs.vmware.com/vi301/server_config/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_adv_netwk.6.5.html
  Here, it is noted that if you are using NLB in Unicast mode, you will need to disable the NIC failover “notify switches” flag on your ESX VSwitch.  This is uindesirable because it will increase the switch convergence time in the event of a failover (since the switches will have to rebuild their lookup tables following connection failures).
• http://pubs.vmware.com/esx254/admin/wwhelp/wwhimpl/common/html/wwhelp.htm?context=admin&file=esx25admin_cluster.11.13.html
  This document outlines the procedure for setting up NLB on Windows 2000/ESX Server 2.5.  Here you are specifically instructed to use Multicast mode.  No reason is given.
• http://pubs.vmware.com/vi301/server_config/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_security_cfgnet.15.13.html
  In this document, procedures for securing ESX server switch ports are outlined.  It is recommended that you prevent the ability of an OS to change the MAC address of its adapter (specifically, to prevent network adapter impersonation that could be used in an attack).  NLB multicast mode does not need to alter the NICs MAC address, and this is compatible with this hardening procedure

It is suggested that IGMP multicast is not necessary in a virtual network environment:
http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf
Thus, we will forgo the use of “IGMP Multicast mode” NLB config… Which is just as well as IGMP appears to break NLB on our ESX cluster!

Another advantage of multicast is that it does limit inter-node communication, so we do not need to add a second virtual network adapter to each ESX guest just to allow the NLB manager to be run on a NLB node!  This is not specifically referenced in any Server 2008 documentation that I can find… it is just something that I noticed… When I had the cluster configured in Unicast mode, the NLB manager complained mightily about its inability to talk to its sibling node.  In Multicast mode, I get the same warning, but parameter reconfiguration succeeds.

Connections to the cluster still are a bit slow on initial lookup… why why why?

Networking Wierdness? Disable IPv6 on Server 2008

If your network is like ours (and I hope that it is not), your DNS server is happily accepting IPv6 address registrations, but your routers can’t makes heads or tails of all those packets with those long addresses on them.  Is this a problem?  You betcha’.

If a client (Windows Server 2008, Vista, or really just about any recent-vintage Linux distro) is configured to use IPv6 by default, and it is fortunate enough to find an IPv6 address registered in DNS for a server to which it is attempting to connect, guess which protocol it will use to connect to the server?  But what if your router can’t pass IPv6 traffic?  If you happed to be using a Vista client, it appears that after a few seconds, we fall back on IPv4.  However, my Server 2008 RC0 clients are timing out and failing to connect to each other.  Bummer!

Rather than going though the impossible process of fixing IPv6 on campus, I have decided to kill IPv6 on my Server 2008 systems.  Here is where I got my information:

http://www.microsoft.com/technet/community/columns/cableguy/cg0506.mspx

The instructions are for Vista, but they work on Server 2008, too.  I just set this registry DWORD:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicestcpip6ParametersDisabledComponents

to Hexidecimal 0xFF, thus disabling all IPv6 on the server.  A quick reboot forced the server to re-register all DNS entries, thus making IPv6 go away.  Voila, we are back in business (and RDP connections to my new servers from Vista clients are working much more smoothly).

Generating SSL certificates for multi-named systems

Recently we changed the name of an application server (groan).  This application has a web front end that requires SSL.  See the problem yet?  New name=certificate mismatch.  To limit this issue, I plan to maintain both the new and old names in DNS, but how to prevent SSL from “breaking”?  One solution would be to multi-home the system, create a separate IIS site for the new IP, and assign this a second SSL certificate.  Unfortunately, our application is hostile to this approach… without making our implementation overly complex, we can have only one IIS site serving the application.

I believe the solution is to add a “Subject Alternative Name” (SAN) to the SSL cert.  Unfortunately, IIS does not make this easy.  The certificate request wizard does not allow for the specification of a SAN.  Once again, it is the command line to the rescue…

The following KB details use of the certreq.exe command line tool to generate a certificate signing request with SAN, suitable for submission to a third-party CA:

http://support.microsoft.com/kb/931351/en-us

The instructions worked fairly well for me, except that I needed to change the “RequestType” to “PKCS10” from “CMC”, as shown here:

http://www.microsoft.com/technet/prodtechnol/office/livecomm/library/confcerts/lcscon_9.mspx

So, here is a representation of my certreq.exe .inf file:

[Version]

Signature=”$Windows NT$

[NewRequest]
Subject = “CN=[FQDN of Server],OU=Enterprise Technology Services,O=University of Vermont,L=Burlington,ST=Vermont,C=US”
Exportable = FALSE
KeyLength = 1024   
KeySpec = 1             ; Key Exchange
KeyUsage = 0xA0     ; Digital Signature, Key Encipherment
MachineKeySet = True
ProviderName = “Microsoft RSA SChannel Cryptographic Provider”
ProviderType = 12
RequestType = PKCS10

[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1 ; Server Authentication
[RequestAttributes]
CertificateTemplate = WebServer
SAN=”dns=[FQDN from CN]&dns=[Original FQDN of Server]”

Note the syntax of the “SAN” line.  Be sure to use “dns=” twice… once for each FQDN.

New generate the certificate request file:

certreq.exe -new cerreq.inf certreq.req

Cut and paste the contents of your .req file into the application for your third-party CA cert.

Today I used:

http://certs.ipsca.com/srvc/Buy.asp

Look!  Free certificates for “.edu” (higher education, and the like) customers!  Certificate generation is a bit slow, but then what do you expect for free?  I wonder what will happen when their root certificate expires in 2009?

Discovering shares on the network

Here was a fun little exercise… in attempting to impose more restrictive firewall rules on our central Windows server subnets, I wanted to enumerate all of the servers which require “public” CIFS protocol access.  Naturally we want to take a programmatic approach to save time.

Here is what I came up with… it is not the work of a networking genius, but it worked:

Tools required:

• GnuWin32 “cut.exe” defined in your %PATH%, GnuWin32 XARGS (optional)
• Remote registry rights on all systems to be queried
• A windows CMD console, ’cause I am lame and can’t take the time to learn PowerShell.

The process:

First, we want to discover all of the servers in a network range that have valid registered DNS names:

FOR /L %d in ([starting octet],1,254) DO nslookup [network].%d | find “Name:” | cut.exe -c 10-100 >> networkhosts.txt

This command will use the FOR command to run “nslookup” on the IP address “[network].%D” several times, starting with [starting octet], incrementing by one, then terminating at “254”.  The output of the lookup is sent to the windows (not GnuWin32) “find” command, which will locate the output line containing “Name:” (this will be the actual DNS name of the system).  The output line is trimmed of all information except the DNS name using “cut” (DNS names start at the 10th character of the NSLookup output). Output is sent to a file for later use. 

Next we test to see if the discovered host names are available.  By doing so, we prevent wasting time on operations against servers that are not available:

FOR /F %d in (networkhosts.txt) do ping -n 1 %d && echo %d >> availablehosts.txt

We use FOR again to ping all of the host in the file that we create above.  If the ping succeeds (meaning the host is there),  repeat the host name with an “echo”, and send that output to a file. 

Now we actually need to see if the hosts have any publicly available network shares… One approach would be use the the old-school “net view” command to display visible shares on the remote hosts:

FOR /F %S in (avaiablehosts.txt) DO net view \%S >> VizShares.txt

This is useful, although it will not disclose “hidden shares”.  To get around the “hidden” shares problem, we can perform a remote registry query to see all shares (other than the defaults) which are made available at the startup of the “server” service.  These shares are published in:
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServiceslanmanserverShares
We will use “reg.exe” and another FOR loop to see what hidden shares exist:

FOR /F %H in (availablehosts.txt) DO echo %H >> hiddenhosts.txt & reg query \%HHKEY_LOCAL_MACHINESYSTEMCurrentControlSetServiceslanmanserverShares /v /f *$ | find “$” | cut.exe -d$ -f1 >> hiddenhosts.txt

We start this loop with an ECHO command so that we can insert the name of the host being investigated into our output file.  The /v flag in REG.exe returns only values under the selected key, and /f *$ filters the output to lines containing only share names ending in “$” (in other words, hidden shares).  Output is passed though “find” to filter out all but the lines that have the actual share name in them.  Output is then sent through “cut” to trim everything but the actual share name from output, and then send the trimmed output to our final file for human analysis.

An industrious admin would chain these commands in a single task.  However, I wanted to check each output file for validity before proceeding… we have some old DNS entries that were making trouble with the remote registry commands.  If I had not removed them, the script could have taken many hours to complete.  If you want to do this, the GnuWin32 tool “xargs” will be invaluable, as it will allow you to pass standard output into commands that do not support standard input, such as “net view”, and “reg.exe”.  For example, we could nest the second “FOR” loop above into the first, and then pipe the output to XARGS NET VIEW”

| Xargs.exe net view

Tossing Out Unresponsive Guests

Ever have a guest you wish you could throw out of your house?  The unresponsive lump of flesh that sits in your favorite chair, eating your food, using up your soap, smelling up the facilities, while contributing nothing to the household economy?  I know I have, but unfortunately social decorum prevents me from acting on my base desire to toss the sponger out into the rain.

Fortunately, we are not bound my the same laws of etiquette in the IT world.  When your VM Guest cops and attitude with you, you can kill it.  Unfortunately, VMWare does not document all of the ways in which you can put an end to pestilential guests.  Fortunately, we have a Platinum support contract, and I was thus able to learn how to send a proper kill signal to a guest.  Here is the drill:

• Verify that ESX “thinks” your guest is still running:
  • vmware-cmd "<pathToVmxFile>/<vmxfile>.vmx" getstate
  • Should return:
    getstate() = on
• Scan the running VM Guest process names to find the VM “World ID” of the hung guest:
  • cat /proc/vmware/vm/*/names
  • Output returned should be similar to this:
    vmid=1105 pid=-1 cfgFile="/vmfs/volumes/46f13af6-27b2dd91-6873-00145e6d4c2c/hyperion11/hyperion11.vmx" uuid="50 26 41 af 30 dd cb 48-2d 9a ed 48 6e 6d db 6c" displayName="hyperion11"
  • Take note of the “vmid” above (in this case, “1105”.  This is the vmid of the “world”, or “cartel” of you guest OS.
• Send a “kill signal 9” to the discovered world ID:
  • /usr/lib/vmware/bin/vmkload_app -k 9 <vmid>
  • Output should be similar to this:
    08 09:40:43.189: Sending signal '9' to world 1104.
• Verify that the guest is not running:
  • vmware-cmd "<pathToVmxFile>/<vmxfile>.vmx" getstate
  • Returns:
    getstate() = off
• You should then be able to start the guest normally.