Too Much At One Time

I’ve had quite a bit of off-hours work, but not a lot of off-hours time in which to do it (mostly having to do with house issues, apartment dwellers should forgive me for being jealous of them a fair portion of the time).  The end result of this is cramming several days worth of work into a window of a few hours.

Last night I had to patch several servers via Windows Update, upgrade the memory in one server (which required re-cabling due to a half installed cable arm), updated the MS DPM agent on two other servers, updated the firmware on our Barracuda spam firewall (which was the second Barracuda update in a row that created more problems than it solved), and replaced the batteries in our battery backup unit (which itself required carefully shutting down several different servers and processes).

After everything came back up I caught a non-production virtual machine that wasn’t starting (which will be a story for a different post), the Citrix servers were running slow, and I was having issues getting a database process to start correctly.  After wrestling with the host of issues for an hour and resolving them for the most part I took off while I was ahead, or so I thought.

In my rush to wrap up I forgot my cardinal rule when touching anything to do with e-mail: test with an outbound and reply it back.  That night my diagnosis of Exchange consisted of making sure Outlook wasn’t popping an error up in the tray before I got bogged down on the other issues.  To make matters worse a user e-mailed me to let me know it wasn’t working, but unfortunately he e-mailed me at the time when the server was down due to the battery replacement, so I thought nothing more of it and told him that it should be working (while I only testing the OWA splash page).  I admit that I also improperly relied on my Windows Mobile phone for testing, an unreliable device even when everything is working properly.

In the end the error was caused by the Exchange 2003 server booting up before (probably by seconds) any of the domain controllers and as a result most of the Exchange services did not start.  It’s worth noting that my near-production Exchange 2007 server did not experience this fault.  Long term I should have a more reliable test mechanism (this happened before after an extended power outage), but most of all I just need to remember to perform my diagnostic procedures before attempting to fix the first issue that grabs my time.

Cold Call Blues

To every salesman who cold calls and cold e-mails me I have a message: I know your situation, but there’s nothing I can do for you.  No, I can’t answer your call to talk for a minute, because it easily turns into five after I sit through your pitch.  No, I do not owe you a ‘courtesy callback’ just because you were one of the many who have left me a voicemail.  As well, I have no interest in any of the following: resold telco services, rebuilt toner cartridges, or ‘scrap’ dealers looking for six month old computer gear.  My place of work has needed contracting services in the past which were typically attached to some piece of software, I myself haven’t needed the services of a generic technical consultant for nearly a decade, so those too are a waste of time.   On that note, I particularly don’t care for cold calls from consulting firms that aren’t even in my state.

I have a soft spot for many of these companies as I used to work for consulting companies.  Even now my job depends on the ability of our sales staff to cajole customers into either buying our products or buying more of them.  One vendor in particular, from whom I have bought no products from ever, has been cold calling me once every week or two for ten years.  About five years ago I tried to tell them to give it up but they’ve persisted.  Needless to say I’ve been through generations of sales people; every six months it’s a different voice that’s desperate to get a sale, any sale, from my disembodied voicemail call tag.

Unfortunately you cold-callers, since we migrated to a new voicemail system many years ago I now use it to expedite your voicemails right into the digital void, never to be heard, ever.  I wish I had it in me to manage a half dozen vendor relationships, but we do not have that much business and what we do have I need to use as a carrot for our occasional goofy request from our vendor with whom we have a long term trusted relationship.

Diagnosing Antiquated Citrix

For quite a while I had an issue on one of our Citrix servers (let’s call it termserv1) where I was receiving “This patch package could not be opened”/”Package corrupt” errors when trying to install patches for Presentation Server 4.0.  Since the server doesn’t hang directly off the Internet I had put off fixing it for several years in the not unrealistic hope that I might one day be able to ‘upgrade’ out of the issue.

Our other Citrix servers are quite a bit more important and they had to be patched.  After every patch I waited for something bad to happen where the newly patched boxes would have some communication issue with the unpatched server.  I had gotten lucky for a long time, but this past week my luck finally ran out when I installed the version 6 hotfix rollup (what will probably be the last hotfix rollup for Presentation Server 4.0).  The ‘good’ servers reported that they were unable to contact the licensing server.  I figured, somewhat incorrectly as it turns out, that the issue was related to the fact that the termserv1, which was the license server, had not been patched.

The reason I had put off fixing the issue is that the fix was rather non-existent.  The first step was to delete the Presentation Server install entries by using the Windows Installer Cleanup Utility.  The second step was to re-install Presentation Server (which for me also required renaming the files ‘ctxdwavo.exe’ and  ‘mfreg.exe’ AND rebooting).  The third step was to hope for the best, and barring that, to fix whatever didn’t work, which in this case meant pretty much everything.

The core issue was that termserv1 was the main server that I relied on in our transition from MetaFrame 1.8 to Presentation Server 4.0 and it had taken quite a bit of work and a long phone call to Citrix to get all of the licensing pieces to work.  From that point on I relied on termserv1 to be the master browser/data collector and main license server.  As it turned out, the licensing issue was related to the fact that the licensing service from Citrix is not patched with hotfix updates and is in fact a separate download.  However, after reinstalling on termserv1 the ICA browser was not working and the server would refuse direct connections with either ‘Protocol Driver Error’ or  just ‘cannot connect to the Citrix XenApp server’  both of which can be an indication of almost any issue.

Emboldened, I decided to uninstall Presentation Server (now that the installer was working) and reinstall.  The uninstall went fine, however when I went to reinstall it refused to join a farm without a SQL database link (we use the included Access database).  I wound up having to selectively restore the unpatched Citrix back onto the box.  After the restore the browser worked again, but ICA connections to the server did not.  Figuring that I would just have to fight my way through it I reapplied the patch and set out to find out what was wrong.

Since I was getting the unhelpful errors from the XenApp client I decided to see if I could get the web interface working to the point that I could get a helpful error or two.  This turned out to be a day long time sink; getting the antiquated version of the Citrix web functions working on a newer server (Win2k3 r2) whose own version of Presentation server wasn’t working anyway, was asking too much given the immediacy of the need to get the server back into full operation again (although it hosts production applications, they’re used sparingly).  I got to the point of getting the “exception has been thrown by the target of an invocation” when trying to create a new web server before giving up.

Frustrated, I turned back to the regular clients.  My first plan of attack was the fact that pointing the XenApp client at one of the two ‘good’ server worked, and termserv1 didn’t, which told me that there was a browser issue of some sort.  Somewhere along the line I came across a Citrix forum post that suggested deleting and recreating the ICA connection.  After doing that the published items still didn’t work, but direct connections did!  After running ‘query farm /load’ I discovered that termserv1 and the other two servers didn’t really see themselves in the same farm (‘query server’ would return all the servers, but ‘query farm /load’  would return incomplete lists).  I performed some actions with ‘dscheck’ and ‘dsmaint failover’, but the real fix (I think) was going into the farm properties in the MetaFrame management console and toggling the affinity between the servers.  Before doing that the servers had two different affinity sets.  Setting it one place made it consistent and everything in Program Neighborhood began working properly.

*UPDATE: If any of the above is helpful it’s a coincidence.  After rebooting termserv1 the IMA service failed to come back up and the management console was inaccessible on any of the servers.  I hacked around for the better part of a (Sun)day, but as it was closing in on midnight I threw in the towel and rebuilt the farm.  I was at least able to rely on a screen shot I had taken as a business continuity hedge so that I’d know which apps I had to rebuild.

ROM Updates

For the past ten years I’ve been responsible for the Nortel switches at my current employer but with this last upgrade our vendor talked me into getting HP Procurves instead.  I don’t regret the decision as they were quite affordable, but I’m underwhelmed by the feature set on the routers compared to our older Nortel 460s.  The VLANs don’t seem to set up as smoothly and the initial ROM version that they shipped with did not include an interface on the web management console for updating the firmware.  “That’s okay”, I figured “I’ll update via TFTP like I had for years with my Cisco and Nortel stuff”. As it turns out that’s a no-go as well, the initial ROM version only ships with the ability to update over the serial port.  I’m puzzled as to how HP can ship a product in 2009 that doesn’t include features that have been included for the last ten years in even the cheapest networking gear.

It wasn’t long after my adventure with the switches that I had to update the ILO card on our AiO StorageWorks equipment (1200r if I recall correctly).  I need to point out that first of all the ILO card that comes with the AiO is a vastly inferior to the ILO2 card that comes on the later model Proliants (in fact, it’s labeled as a ‘LO100i’ card and isn’t an ILO at all).  It goes without saying that the equipment didn’t ship with the latest firmware on the ILO card, but the update had to be performed using basically the same mechanism that I used to update Compaq equipment in the mid-nineties: reboot onto an image.  It was USB at least and not floppy, but still, what’s the point of a remote management adapter if you have to be physically present to update it?

Last on my list is my Barracuda spam firewall.  I made the mistake of updating the firmware last night at around 10:30 at night.  The web based update screen provides a VERY poor indication of how the update is advancing.  The unhelpful gauge on the ‘update’ web page eventually locked up completely making it epically unhelpful.  Is it still updating?  Who knows?  Is it okay to reboot?  I hope so because that’s exactly what I did at 11:30 last night.  Barracuda’s web site was as useless as always and reminded me of when our device wasn’t properly authenticating e-mail addresses against our Active Directory LDAP setup.  The fix in that case was to point it at a Global Catalog AD server instead of just any AD server, but just like with my ROM issue Barracuda’s support and web site provided no clues, just the standard bunk that comes packed in with the device itself.  I guess since their tech support never solves any problems, they wouldn’t have any service call details to post to their site.

Custom Virus Removal

Over the past several years I’ve had to clean malware from many PCs for friends and acquaintances.  This used to be rather trivial at first, but has grown to be a rather huge, complex undertaking as time has progressed with some cleanings taking several days.  The reason for this is two-fold.  First, malware writers have gotten much more clever and now put together software that is difficult to impossible to remove.  Secondly, attack vectors are much more ubiquitous.  At one point in time only ‘naughty’ computer users, those who trafficked in illegal software or pornography, got horrible malware infections but now the attacks are much more subtle.  Even though I consider myself somewhat wise in the area of computer security, I’ve seen a few phishing type emails that I’ve almost fell for, and that’s to say nothing a of the virus writers who infect normally safe web sites with their malicious code.

Typically when I get an infected machine it has either has no anti-virus package, or it has the original demo of an anti-virus package that came with computer and is now three years out of date.  Needless to say, the best thing that you can do to protect your computer is to have an anti-virus package installed and up to date.  I know they are overpriced packages of dreadful software that slows down your computer, but the cure is still better than the disease.  It’s worth pointing out as well, that many ISPs now offer a free anti-virus package for their customers to use.

Anyway, since the infected PCs I get have no anti-virus installed, and typically you cannot install much of anything on an actively infected system, the system needs to be manually cleaned to the point of making it operational.  What usually makes this easier is having an operational computer available to do web searches and other work on.  First, I reboot in Safe mode (press F8 just before XP boots). If possible, I’ll just use a ‘system restore point’ (you’re prompted on doing so when rebooting in Safe mode), but this is only possible if you know the approximate time range of the infection (as well, any apps installed after the restore point will be removed). If that’s not possible then, this is where it gets more painful, I first will manually clean out the ‘Run’ key of the Windows registry while making a note of what files are listed there.  More often than not I find myself doing web searches on some files to see if they are legit or not.  If they are not legit, I’ll make a note of those malware packages and what other files they rely on so that I can hunt them down in the system.  What I have found to be most effective in dealing with malware files is replacing them with an empty read-only file of the same name.  This way if the virus tries to recreate itself it will have trouble doing so.

So for instance, if a piece of malware has two files named ’PS1.exe’ and ‘winsysrun.exe’ I delete those two files and copy a blank text document in as those same names. This will confuse/corrupt most malware, particularly if the new files cannot be written to. In order to better ensure this I will (on systems not running XP pro) use a painful program called ‘cacls’ (or the related, very slightly less painful ‘fileacl’) to remove all access rights to the file.  The main problem with this whole procedure is that there are typically MANY file names so I’ll wind up creating a custom batch file to do most of the heavy lifting. I’ll do this one of two ways (or both).  On my extra, operational computer, I will download a list of files associated with the malware from an anti-virus site and plug those into a batch file and/or I will get a directory listing by date of the \windows\system32 directory where many virus files can be found.  To aid in that search I’ll  add the ‘company’ item to the columns in the detail view of the directory so that I can determine which date groupings to look for.  Any files that do not have a company listed should be viewed with suspicion and ‘googled’.

Once I have my lists of files I’ll then either manually copy in my ‘del’ and ‘copy’ commands (and ‘cacls’ and or ‘attrib’), use search and replace methods (adding in a comma where I want my commands to go and then search/replace the easier to type comma with the commands that I want), or as I did in one extreme case, compiling the list into Excel and adding the commands in columns and exporting it as a CSV and just ‘search/replace’-ing the commas with spaces.

At this point the viruses on the PC will typically be disabled enough that the computer can be booted up and an anti-virus package installed to clean off the system.  Typically you will want to do this with the computer NOT hooked to the Internet so that usually means porting the virus signature updates from the clean computer over to the ‘dirty’ computer via, ideally, CD media (as non-read only USB sticks can become infection vectors themselves).   I’ll often put the computer through two full scans of at least one package and a third scan through a different package just to verify it’s integrity (such as free web based scanners from Panda or even Adaware).

On a final note, if a computer has a ‘root kit’ malware installed, this means that portions of the OS itself have been replaced with malicious files.  Detection of this software is difficult to say the least and cleaning them off is virtually impossible.  The only real resolution for a root kit infection is to format the drive.  This makes it all the more important to get a system restore disk of some sort from the manufacturer.  Unfortunately a fair portion of the malware I have seen recently has been of this sort and the computers I have worked on that were in this state were ‘uncleanable’.