Yes, its official, the problem with Machine SID duplication in the Windows world is a myth (except in some specific circumstances).

Read all about it here:

http://blogs.technet.com/markrussinovich/archive/2009/11/03/3291024.aspx

On November 3 2009, Sysinternals retired NewSID, a utility that changes a computers machine Security Identifier (machine SID). I wrote NewSID in 1997 (its original name was NTSID) because the only tool available at the time for changing machine SIDs was the Microsoft Sysprep tool, and Sysprep doesn’t support changing the SIDs of computers that have applications installed. A machine SID is a unique identifier generated by Windows Setup that Windows uses as the basis for the SIDs for administrator-defined local accounts and groups. After a user logs on to a system, they are represented by their account and group SIDs with respect to object authorization (permissions checks). If two machines have the same machine SID, then accounts or groups on those systems might have the same SID. It’s therefore obvious that having multiple computers with the same machine SID on a network poses a security risk, right? At least that’s been the conventional wisdom.

The reason that I began considering NewSID for retirement is that, although people generally reported success with it on Windows Vista, I hadn’t fully tested it myself and I got occasional reports that some Windows component would fail after NewSID was used. When I set out to look into the reports I took a step back to understand how duplicate SIDs could cause problems, a belief that I had taken on faith like everyone else. The more I thought about it, the more I became convinced that machine SID duplication – having multiple computers with the same machine SID – doesn’t pose any problem, security or otherwise. I took my conclusion to the Windows security and deployment teams and no one could come up with a scenario where two systems with the same machine SID, whether in a Workgroup or a Domain, would cause an issue. At that point the decision to retire NewSID became obvious.

I realize that the news that it’s okay to have duplicate machine SIDs comes as a surprise to many, especially since changing SIDs on imaged systems has been a fundamental principal of image deployment since Windows NT’s inception. This blog post debunks the myth with facts by first describing the machine SID, explaining how Windows uses SIDs, and then showing that – with one exception – Windows never exposes a machine SID outside its computer, proving that it’s okay to have systems with the same machine SID.

Now, one of the biggest problem for us when deployment new servers is that we want to get everything as automated as possible. One of the most complex to automate is networking configuration. That is, you need to figure out which nic to team and which to set as heartbeat or backup nic, etc. In this, I use nic and network adapter interchangeably.

The most persistent problem is getting a ordered list of network adapters from Windows. Windows itself does not present nics in any order. So your slot 0 (embedded) port 1 could be named as “local area network connection 3″ or called “Broadcom 1GB PCI Ethernet Adapter 4″ and the slot 10 port 2 nic could be named as the first nic.

We have a standard for deploying servers. The first nic (usually the embedded port 1) is teamed with the first nic card’s port 1 (e.g. slot 2 port 1). Port 2 Embedded is used for backup and the 4th port (in this case, Slot 2 port 2) is used of heartbeat if its a cluster. Now why do we want to do that? The basic idea is that this reduce logistic work for the cabling people and for us also. We all know which port is used for what and don’t have to keep referring to deployment documents when configuring nics (except to get the ip addresses).

If you look into the register, there is a value in HKLM\CurrentControlSet\System\Enum\PCI keys (or the nic’s enum keys) which presents the network adapter with “LocationInformation” with “PCI bus X, device Y, function Z”.

• The smaller the PCI number. e.g PCI bus 1, the higher the order of the nic, ie. first nic.
• The smaller the function number within the same PCI bus, the higher the order of the nic, i.e. function 0 could be port 1 and function 1 port 2

Note: I am only talking about network adapters here.

With this information its very easy then to grab this registry value and transform it into some ranking system for the nics. The formula is very easy, multiple PCI number by 10 and add the function number to it (i.e ranking = (10 x pci number) + function number). The smaller the number the higher up in the order.

So all we need to do now is to:

• use WMI win32_networkadapter to loop through all nics,
• grab their DeviceID and PNPDevicID (which is the registery key under ENUM, e.g. PCI\VEN_10DE&DEV_07DC&SUBSYS_2A65103C&REV_A2\3&2411E6FE&1&78.
• Use the PNPDeviceID to look for the LocationInformation value,
• Create the ranking and sort them and pump them into a dictionary object with the ranking number (I transform those numbers to absolute ranking,e.g. 1, 2,3,etc) and the deviceID.

The final result in the dictionary object should look something like this (where key is the ranking and item is DeviceID of the network adapter):

key     item
==      ===
1        11
2        7
3        12
4        9
5        8

So now all I need is to grab the deviceIDs of order pair nic with ranking 1 and 3 and team them, get the rank 2 and setup the backup IP addresss, etc

Now I created the scripts and got it working on Windows 2003. Then I tried it on Windows 2008 and it failed!

When I looked at the registry value, the LocationInformation is now something like this:

@system32\drivers\pci.sys,#65536;PCI bus %1, device %2, function %3;(3,0,0)

My jaw dropped to think that I finally cracked it and Windows 2008 changed the format. I couldn’t believe my eyes to think that now I cannot easily get PCI and function numbers from registry. I searched the Internet for some answers and all pointed to using some device programming function to retrieve these values.

However, I started to looked a few more of these values, I found something was different in each value. I looked closer and voila! The answer is in the brackets the end of the value! Windows 2008 have moved the PCI, device and function values in a bracket

@system32\drivers\pci.sys,#65536;PCI bus %1, device %2, function %3;(3,0,0)

So in this case, the values are PCI bus 3, device 0, function 0.

Now all I have to do is to change my script logic to grab the values between the brackets instead for Windows 2008 servers.

One of the things we needed to do when deploying a server is that we need to setup the partitions correctly before we deploy it. Of course, we could always load up the smart start utility and perform the RAID configuration. But when you are doing remote deployment, that is, someone in London racked and powered up the server and you deploy it, it not always fun to load the ISO CD image, mostly because of the synchronization of the stupid mouse cursor over the WAN in some locations. And even in locations where the WAN is good, getting the cursor to go where you want to with the smart start it no fun. The other point is also it takes time for Smart Start to run.

So what we did was to include the command line equivalent of the array configuration called hpacucli.exe (you can download this from HP) into our deployment image. We can then launch this from the command prompt after its booted up in WinPE.

In our configuration our standard is always RAID 1 with 2 x HDD, 50% C:, the rest D: and we only need to configure one partition (the first partition).

Type the command “hpacucli ctrl all show config”, and take note of the size, disk description and slot number. You should see a result like this.

Smart Array P400 in Slot 1 (sn: xxxxxxxx)

array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (36.0 GB, RAID 1+0, OK)
logicaldrive 2 (32.3 GB, RAID 1+0, OK)

physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)

The above results actually shows that I have already configured RAID partitions (okay this is reuse kit), so I want to repartition it.

So I issue the following commands to destroy any existing partitions and redo with only 1 partition. Note the size=36000 will not give me exact 36 GB, since 36 GB is not really 36000 kbytes. If you want to find out the exact number, I suggest you do a display on an existing server of the same configuration to find the size to put.

hpacucli controller slot=1 logicaldrive all delete forced
hpacucli ctrl slot=1 create type=LD drives=2I:1:1,2I:1:2 raid=1 size=36000
hpacucli ctrl slot=1 create type=LD drives=2I:1:1,2I:1:2 raid=1 size=Max
hpacucli rescan
hpacucli ctrl all show config

Hope this helps some of you.

One of the think I hate about firefox is that when you internal certificates like those issued by HP Integrated Lights-Out’s (iLO) web page, you have to confirm to download the cert and add it to your cert store. There is not one-click solution to this as yet and neither can I disable this in FireFox even if I only intend to use it in an intranet environment.

More troublesome is when I update the firmware in my iLO and hence a new cerificate is generated and when you try to launch the web page for iLO access again you will encounter the following error.

Error code: sec_error_reused_issuer_and_serial

It took me a while to figure it out from the googling, but the solution is simply to delete the cert8.db from your FireFox profile, usually found in  C:\Documents and Settings\<yourname>\Application Data\Mozilla\Firefox\Profiles\<yourprofile>. Easier still just search for cert8.db in your computer and delete that file. 

Oh….. you need close FireFox first

Recently, we found that some servers’ and workstations’ host record (A RR) has been dropped off our DNS servers. All the machines were running a DHCP clients and we are using MS product for both DHCP and DNS.

The DHCP servers are configured to update the host record on behalf of the client. When we checked the scopes, we found that they have a 30 days lease duration. Very quickly it was established that its possible that the host records could have gone stale in the DNS due to the lease duration and got scavenged.

A quick chat with our DNS team and we found our that the DNS servers have a setting of about 14 days before a record is scavenged in a daily routine. Namingly the values are 7 days for non-refresh interval and 7 days for refresh interval.  (see DNS Aging/Scavenging Simplied)

Now with a  30 days lease, the client will renew itself at 50% lease which is 15 days. When the client first gets its lease from the DHCP server, the server will register the host record via DDNS. It will do so again in 15 days when the client renews its lease. It the cleint is unsuccessful, it wil try again at 87.5% lease with a rebinding broadcast.

As we can see from here, there is a gap of 1 day in which the host record could be considered stale and scavenged.

Reducing the lease duration will resolve the issue, but by how much?

Some would choose a lease duration where 50% lease is 14 days of less, which gives you about 28 days lease duration. I don’t like that calculation. What if for some reasons the renew did not happen on the 14th day? If it is not successful on the 14th day, then it wil try to renew itself on around 24 days (87.5% lease). This mean potentially, I will have 10 days where the host record could be gone.

Being an production guy, I like to be more cautious and prefer to have the lease duration in which 87.5% lease of 14 days, giving 16 days (16 days is on the edge, so I would choose 15 days lease), to balance between network noise and technical effectiveness. This will allow the host record to not go stale until after the 2nd dhcp refresh interval (87.5% of lease window). If by 87.5%  lease and renewal still don’t happen, sometime must be wrong with the client anyway and it needs to be fixed.

So if you have an environment in which you integrate DHCP and DNS, you will need to consider the DNS non-refresh and refresh intervals when considering your dhcp lease duration, to avoid host records disappearing when you need them.

We had a bunch of HP Proliant DL580 G5 running with 4 x quad core CPUs. With all the cores turned on, this would give us 16 CPUs. When trying to build the servers into Window 2003, we found that Windows startup page (the one with the moving bar at the bottom) would hang and not continue.

After some trial and error, we found that if we disabled “”one-half of cores per Physical Processor” from the BIOS (effective with 8 CPUs), the server would then boot up correct. So we continued with the Windows build and completed building the servers into 32-bit Windows 2003 Enterprise Edition.

We then went to turn on the above settings again and indeed it was stuck at the Windows startup page again. Turned it off and it work. It was a strange symptom because I don’t believe that Window 2002 Ent Ed cannot support 16 CPUs but then we do have other Windows 2003 Ent Ed running with 16 CPUs without any issues.

At first we thought it has to do with the 8 way SMP support found in http://technet.microsoft.com/en-us/library/cc758523.aspx, but this has more to do with socket support then cores support

Anyway, to cut the story short, one of my colleague actually found the problem. Apparently, this has to do with the NIC’s firmware version of external PCI network adapter and in DL580 G5 it was a HP NC360T PCIe DP Gigabit Server Adapter.

After upgrade the firmware to the latest, the issue never happened again.

In my work, I have seen network cards and driver causing various issues to Windows start up issue and this is another one in the basket of knowledge for me.

We were doing a round of upgrades in our Windows estate and we found that one of the Windows 2000 server had a broken terminal service.

All the following checked out, but the server was just not listening on port 3389:

• terminal services is running (reboot server like a gooogle times) and not error in eventlogs
• no other applications is locking 3389 preventing it from being used
• no ipsec policies was set
• no ipfilter was configured.
• when I run tsadmin on the server (obvious remotely won’t work),  I could see my only session

In other words, the server is behaving just like all other normal Windows 2000 servers with a working terminal service.

Then while reading one of the resources off google search, I decided to have a look at the network adapter settings in the RDP-TCP properties in tsconfig.

Strange enough, it listed 2 network cards, but one of them was a NC7770 card, while we had only NC7778 cards on this server. So, I double  checked device manager (with show hidden device option) for the card, but it does not exists. But when I did a search of the adapter name in the registry, I could find that the original network registry values are still in the register. The card is identified to be sitting on the PCI bus 3 and bus 3 shows its empty. So ideed its a phantom card.

The default network card settings on RDP-TCP to use “All adapters configured for this protocol” and my guess is that the server must have picked the NC7770 to bind the protocol to it and hence it did not work

So I chose the correct network adapter applied the settings, rebooted the server and terminal services is working correctly again.

P.S. I am definitely sheduling this server to be rebuilt to Windows 2003..

We had a strange server that would BSOD regularly with stop codes 7F or 9C, in the later days it was BSODing every 10 mins or so.

So we had the motherboard change and it stop BSODing for about 1 hour and started again!

Diagnostics was ran but all were successful (how typical!) and HP was really reluctant to bring down other parts as the diagnostics showed okay. In fact, this is not the first time I have seen HP servers with good diagnostics but after changing on of the parts, everything is resolved.

So we insisted of getting some other parts down to change. I was not really involved in this but on that day, I was one shift to be the hardware engineer escort to our datacenter. So I did a quick check on the BSOD errors and found that most of the discussion was around memory or CPU issues.

So when I met the HP engineer in the datacenter, I eagerly asked if he had brought spare memory and CPU to test. Instead, they brought down an array controller to swap out! What rubbish and waste of time, I thought to myself, as the BSOD error does not show any of the symptoms pointing to a faulty array controller.

Anyway, I did not want to get the controller changed and insisted that we did some memory and CPU swap around to test.

So the engineerings took out 2 banks (out of 4) of memory, booted up the server, 10 mins later BSOD occurred.Then he swapped the other 2 banks and booted up and still the same problem. At this point, it could not be the memory, unless we are very unlucky, its unlikely that we had 2 faulty memory modules exhibiting the same symptoms.

Next was the CPU test, he took out one of the CPU and boot up… 10 mins later, BSOD occured. Then a swap the CPU, boot up…10 mins later no BSOD, so we wait for another 15-20 mins and the server is still running. So it was indeed a CPU causing this issue after all.

Anyway, it was later at night and I asked the engineer to come in the next day with a new CPU. I left the server running overnight with one CPU and it was still running the next day I went back to the office.

Problem solved.

Recently we were going through a upgrade exercise for our entire Windows estate to patch them to out latest security and company specific components. We encountered, so far, two servers that when rebooted, Windows 2000 will not start up and they showed this error

Windows 2000 could not start because of an error in the software. Please report this problem as Loader Error 3.

Starting up in recovery console or safe mode is no hope as we encountered the same error. So this points to a fundamental issue with the Operating System.

Searches on the internet seems to indicate boot partitioning issue, with running fixboot as a solution. But for our case, Windows did try to start, you can see the Windows startup (black) screen and the cursor progress bar moving for a short while before it stops

Then we found an MSKB article that seems to be related to our problem: System may not start when creating a large number of logical units and volumes

Both broken systems were SQL server boxes and we know that they do have a large number of SAN volumes and we are constant allocating and deallocating drives due to new disk space requirements.

So I fired up a copy of our WinPE boot disk (or if you have your Windows 2000 CD), go into command prompt and checked the size of the c:\winnt\system32\config\system file. Is about 9 MB, which is pretty near the limit described in the KB.

So I copied out the system file into my work server, used regedit to delete both SCSI and STORAGE keys in controlset001 (I ignored the steps to remove controlset002 as its not necessary for this issue), copied back to the broken machine and reboot and viola! the server is back up alive.

Recently I needed a function to calculate the nth day of the month, that is, first sunday, 2nd thursday, last friday, etc. I thought such functions would be easy to find in the internet, however, I couldn’t find a complete function and so decided to write one:

Apologise for the formatting, wordpress don’t like programming codes in their blogs

Option Explicit

Wscript.echo nthDayOfMonth(now(), 4, "wed") 'get first month of this month
Wscript.echo nthDayOfMonth("22/05/2007", 3, "wed") 'get 3rd wed of ref month/year

'======================================================
Function nthDayOfMonth(strRefDate, IntOrder, strDayName)
'======================================================

'calculates the first, 2nd...last day (sun ... sat) based on the ref date
'Returns the date in dd/mm/yyyy format

'strRefDate = A reference date for the calculation: format dd/mm/yyyy
'IntOrder = 1 - 5 (1=first, 2=2nd, >=5: last)
'strDayName = sun, mon, tue, wed, thu, fri, sat

Dim dicDayNum
Dim intRefMonth
Dim intRefYear
Dim dtFirstDay, dtLastDay
Dim intFirstDayNum, intLastDayNum
Dim strDayNameL
Dim intBackStep, intFwdStep

nthDayOfMonth = ""

Set dicDayNum = CreateObject("Scripting.Dictionary")
dicDayNum.Add "sun", 1
dicDayNum.Add "mon", 2
dicDayNum.Add "tue", 3
dicDayNum.Add "wed", 4
dicDayNum.Add "thu", 5
dicDayNum.Add "fri", 6
dicDayNum.Add "sat", 7

strDayNameL = lcase(strDayName)

if intOrder <= 0 then exit function
if dicDayNum.item(strDayNameL) = "" then exit function

intRefMonth = month(strRefDate)
intRefYear = year(strRefDate)
dtFirstDay = dateserial(intRefYear, intRefMonth, 1)

if IntOrder >= 5 then 'calculate the last xx day of month

dtLastDay = (DateAdd("m", 1, dtFirstDay)) - 1
intLastDayNum = weekday(dtLastDay, vbSunday)

if dicDayNum.item(strDayNameL) = intLastDayNum then
nthDayOfMonth = dtLastDay

elseif intLastDayNum < dicDayNum.item(strDayNameL) then
intBackStep = -((7 + intLastDayNum) - dicDayNum.item(strDayNameL))
nthDayOfMonth = DateAdd("d", intBackStep, dtLastDay)

else
intBackStep = dicDayNum.item(strDayNameL) - intLastDayNum
nthDayOfMonth = dateadd("d", intBackStep, dtLastDay)

end if

else 'calculates the other nth order days

intFirstdayNum = weekday(dtFirstDay, vbSunday)

if dicDayNum.item(strDayNameL) = intFirstdayNum then
intFwdStep = (intOrder -1) * 7
nthDayOfMonth = dateadd("d", intFwdStep, dtfirstday)

elseif intFirstdayNum > dicDayNum.item(strDayNameL) then
intFwdStep = (7 - intFirstdayNum) + dicDayNum.item(strDayNameL) + ((intOrder -1) * 7)
nthDayOfMonth = dateadd("d", intFwdStep, dtFirstDay)

else
intFwdStep = dicDayNum.item(strDayNameL) - intFirstdayNum + ((intOrder -1) * 7)
nthDayOfMonth = dateadd("d", intFwdStep, dtFirstDay)

end if

end if

End Function