In the studio today with AshenCult, a new Philadelphia black metal band. To say that I’m excited about this is kind of an understatement; I can count the number of black metal bands in the city on one hand so for a new one to appear out of nowhere is just awesome. For me to record their first release, even better. A lot of the area bands are very punky and AshenCult is no exception — they love putting punk beats over traditional melodic black metal riffs, like if Immortal had gone to Drexel. Still, even with the heavy punk influence, it’s 100% black metal, a good hybrid of Dissection-ish melody and early American ugliness.

Today marks my first session using Reaper instead of Cubase. I had been using Cubase 5 but it was just not playing well some of my plugins. Reaper, on the other hand, does everything perfectly, costs a praction of the price, is community-focused, and has so many features that Cubase didn’t… After years using Cubase, it’s going to take some time to get used to. I had to install Reaper 4 Alpha and then do a lot of hotkey changes just to get some really basic editing functionality going but now that that’s done, I’m feeling good. I’m sure I’ll have more to say as I get deeper into it.

I never thought I’d be one of those guys who gets excited about RAM upgrades or service packs but, well, there I was, counting down the days to July 16 when Microsoft put SP1 on TechNet. It goes live to the world next week. SP1 doesn’t offer much in the way of new features for most people but one in particular was of great interest to me: Dynamic RAM for Hyper-V. I have a large number of Hyper-V machines, a few of which are clustered, and a couple are highly-utilized. Since processor and HD utilization will adjust based on need, RAM becomes the issue because as soon as you allocate those 2 or 4 or 8 or more GB to your virtual machine and start it up, that RAM is gone, regardless of how much it’s actually using. No more. Awesome.

Normally, I’d wait a little bit to install the update on production servers but one of my networks, a client with a cluster of virtual terminal servers, is using almost all 24GB of their RAM. This update is a big help and will save them quite a bit of money over adding more RAM. I first installed SP1 on my office desktop and it went in without a hitch. One guy in the office tried installing it on his desktop and it kept bombing out. Still, the need was there, so I moved all resources over to one node of the cluster, installed the SP, rebooted, and all was well. It was when I tried the second node that the problem started.

The SP seemed to install correctly. After reboot, it said the update failed and it rolled back, rebooted, and presented me with the most generic of errors. It said the service pack failed, “Element not found.” Might have been ELEMENT_NOT_FOUND. A message saying “Error not found” and a link to a MS KB at the bottom gave a boring, nonspecific troubleshooting guide: SFC /SCANNOW, running the Windows System Update Readiness Tool, and doing an in-place upgrade (AKA repair install). First two didn’t help, third was not a viable option.

I started reading. Windows Vista introduced something called CBS, Component Based Servicing, which you can read about here. While certain notifications about Windows Updates are still held in windowsupdate.log, the details about what is happening is held in %SYSTEMROOT%\logs\CBS\cbs.log. This log is backed up and recreated every time you install hotfix for any other update.

After hours (and hours and hours) of troubleshooting, trying to figure out what was different between my nodes, I posted to Microsoft’s official forum, uploaded my CBS folder, and they gave me my answer. Turns out, a specific hotfix was missing files: this one. The Microsoft rep who gave me the answer quoted my log and gave the answer.

2011-02-17 12:41:18, Info                  CBS    Doqe:   q-uninstall: Inf: wvid.inf , Ranking: 2, Device-Install: 0, Key: 113, Identity: wvid.inf, Culture=neutral, Type=driverUpdate , Version=6.1.7600.16475 , PublicKeyToken=31bf3856ad364e35, ProcessorArchitecture=amd64, versionScope=NonSxS

2011-02-17 12:41:18, Info                  CBS    Perf: Doqe: Uninstall started.

2011-02-17 12:41:18, Info                  CBS    Doqe: [Forward] Uninstalling driver updates, Count 54

2011-02-17 12:41:19, Info                  CBS    DriverUpdateUninstallUpdates failed [HRESULT = 0x80070490 – ERROR_NOT_FOUND]

Files from the security update

MS10-010: Vulnerability in Windows Server 2008 Hyper-V could allow denial of service

are missing

That’s great and all but how did they come to that conclusion?

Identity: wvid.inf , Culture=neutral, Type=driverUpdate , Version=6.1.7600.16475

this tells me that the update causes the issue.

Yeah, but what from that says that the specific hotfix was the problem? How did they find that line in my 45MB+ log file?

google? look at the inf file version on the KB page:

http://support.microsoft.com/kb/977894

“ERROR_NOT_FOUND” must appear 100 times at least, so do references to wvid.inf. I asked for them to walk me through it, step by step.

I opened the Log file, and searched for Error so I found this line. I looked what the installer was doing and saw that it tries to remove the inf. I googled what the inf is and found this Hyper-V update.

Still doesn’t explain how they got to that specific error. I did look at it a bit and found that while there are hundreds of errors that look like this:

Failed to get session package state for package: Package_6_for_KB2482017~31bf3856ad364e35~amd64~~6.1.1.1 [HRESULT = 0x80070490 – ERROR_NOT_FOUND]

The one that he pointed out was unique and had no similar lines anywhere. It does make sense when you look at it: it tries to uninstall a hotfix, can’t find a file, and fails. But there are A LOT of errors in the file and you’d think that somewhere it would identify a critical failure and point to this as the cause for the entire service pack bombing out. I asked again whether they had a way of filtering out the ERROR_NOT_FOUND messages that didn’t matter VS those that did and didn’t get a response yet. I get this feeling that they just know what to look for and used CTRL + F until they found an error that didn’t look like the others.

With the KB identified, I had to find a way to uninstall it. Going through Add/Remove programs didn’t work, it tried to uninstall, rebooted and failed. Using WUSA /UNINSTALL /KB: seemed to work but, as I later learned, is not supported or, well, working. The end result was to search the registry for the KB, pull out every key, reinstall the hotfix, reboot, install the SP, and reboot. Success. But frustrating. Our company’s virtual host also failed its SP1 install though some of our VMs did not. Two other guys in the office installed it on their Windows 7 machines without difficulty. When I have some time, I’ll check out our VH’s log and see if I can apply this experience. Either way, I find it extremely frustrating that I had to do this much work to fix what should be a very simple process. If the Service Pack install made use of the Windows Event log, filtering errors would have been easy. If the error message given after the failure was more clear, I could have troubleshot effectively. If SFC /SCANNOW found missing files — since there WERE missing files — that would have helped. If the Windows System Update Readiness tool searched more thoroughly, that would have helped too. While the Microsoft rep was very knowledgeable and rather prompt, I shouldn’t have to resort to that in the first place. This Service Pack was made available to TechNet not three days ago and it failed our first 3 of 6 tests; another poster on the MS forum said that one of their cluster nodes also wouldn’t install it so he just wiped and reinstalled. This is unacceptable and does not bode well for Microsoft’s PR in the coming weeks.

Abserso is in the studio to record a new 7″. I was really impressed with these guys when Unrest played with them last year so I made a point to bug their bassist Dylan about recording down here. Really techy stuff, thrashy death metal. Awesome.

I’m very pleased with this recording so far. We’re on what should be the last take of guitars right now, bass and vocals tomorrow. This is the first time tracking with the Big Ben driving my converter and I’m shocked at the difference. Everything just sounds clearer and more real, all those things people talk about when they upgrade their conversion. I also got a new comp yesterday, a sampled SSL 4000 (I assume?) buss compressor for Nebula made by Analog in the Box, and the VTMM2 seems to be working since I reset my BIOS to fail-safe defaults. Google that stuff if you don’t know what it is and care. It’s all crucial for what I do.

Kill!

I have a real love-hate relationship with Symantec. I love some of their corporate products. Backup Exec has so many features, the price is very competitive, I like how they license their Hyper-V Agent (though SBS Edition should include one…) and when it works, it works really well… But it doesn’t work really well, usually. It’s prone to weird errors and things will just stop working until you restart services or reboot servers or pause and unpause your server or put a device back online. Backup Exec System Recovery 2010 AKA Norton Ghost with a Higher Price Tag is a fantastic program. It’s my go-to P2V solution, the backup and recovery works well and is easy to manage.

Except for the whole there’s-this-known-issue-where-your-server-will-BSOD-when-you-first-use-it thing. Yeah, that thing. Because I always have to look through Google for this link, I’m posting it here. Maybe someone else will find it and it will help them. This link has the basics though all it really does is link to the MS KB with the hotfix, which is http://support.microsoft.com/kb/980856. This page gives more details and pertains to BESR and BE2010 installed on the same machine. Symantec isn’t too clear about why this happens other than to say that the problem is the Symantec VSS Writer.

I had a lot of fun fixing this last night when the password to extract the MS hotfix didn’t work. Wonderful! I remembered I left the MSU file on another client’s desktop, the only nice thing about needing to install this all the time, so I popped in, emailed it to myself, downloaded, installed, rebooted, and all was good! But this shit is still shit and I’m tired of shit.

Yes it is.

I bought an Apogee Big Ben last night. It wasn’t exactly high on my to-do list but the price was absurd so I just couldn’t pass it up. It’s a good investment, one of those things that immediately makes every single microphone I use sound better just by turning the damn thing on and it will be good to have when I upgrade my AD/DA conversion down the line. Read about it here, if you want.