Category Archives: Server Support

Too Much At One Time

Server Support,

I’ve had quite a bit of off-hours work, but not a lot of off-hours time in which to do it (mostly having to do with house issues, apartment dwellers should forgive me for being jealous of them a fair portion of the time).  The end result of this is cramming several days worth of work into a window of a few hours.

Last night I had to patch several servers via Windows Update, upgrade the memory in one server (which required re-cabling due to a half installed cable arm), updated the MS DPM agent on two other servers, updated the firmware on our Barracuda spam firewall (which was the second Barracuda update in a row that created more problems than it solved), and replaced the batteries in our battery backup unit (which itself required carefully shutting down several different servers and processes).

After everything came back up I caught a non-production virtual machine that wasn’t starting (which will be a story for a different post), the Citrix servers were running slow, and I was having issues getting a database process to start correctly.  After wrestling with the host of issues for an hour and resolving them for the most part I took off while I was ahead, or so I thought.

In my rush to wrap up I forgot my cardinal rule when touching anything to do with e-mail: test with an outbound and reply it back.  That night my diagnosis of Exchange consisted of making sure Outlook wasn’t popping an error up in the tray before I got bogged down on the other issues.  To make matters worse a user e-mailed me to let me know it wasn’t working, but unfortunately he e-mailed me at the time when the server was down due to the battery replacement, so I thought nothing more of it and told him that it should be working (while I only testing the OWA splash page).  I admit that I also improperly relied on my Windows Mobile phone for testing, an unreliable device even when everything is working properly.

In the end the error was caused by the Exchange 2003 server booting up before (probably by seconds) any of the domain controllers and as a result most of the Exchange services did not start.  It’s worth noting that my near-production Exchange 2007 server did not experience this fault.  Long term I should have a more reliable test mechanism (this happened before after an extended power outage), but most of all I just need to remember to perform my diagnostic procedures before attempting to fix the first issue that grabs my time.

ROM Updates

Network Communications, Server Support

For the past ten years I’ve been responsible for the Nortel switches at my current employer but with this last upgrade our vendor talked me into getting HP Procurves instead.  I don’t regret the decision as they were quite affordable, but I’m underwhelmed by the feature set on the routers compared to our older Nortel 460s.  The VLANs don’t seem to set up as smoothly and the initial ROM version that they shipped with did not include an interface on the web management console for updating the firmware.  “That’s okay”, I figured “I’ll update via TFTP like I had for years with my Cisco and Nortel stuff”. As it turns out that’s a no-go as well, the initial ROM version only ships with the ability to update over the serial port.  I’m puzzled as to how HP can ship a product in 2009 that doesn’t include features that have been included for the last ten years in even the cheapest networking gear.

It wasn’t long after my adventure with the switches that I had to update the ILO card on our AiO StorageWorks equipment (1200r if I recall correctly).  I need to point out that first of all the ILO card that comes with the AiO is a vastly inferior to the ILO2 card that comes on the later model Proliants (in fact, it’s labeled as a ‘LO100i’ card and isn’t an ILO at all).  It goes without saying that the equipment didn’t ship with the latest firmware on the ILO card, but the update had to be performed using basically the same mechanism that I used to update Compaq equipment in the mid-nineties: reboot onto an image.  It was USB at least and not floppy, but still, what’s the point of a remote management adapter if you have to be physically present to update it?

Last on my list is my Barracuda spam firewall.  I made the mistake of updating the firmware last night at around 10:30 at night.  The web based update screen provides a VERY poor indication of how the update is advancing.  The unhelpful gauge on the ‘update’ web page eventually locked up completely making it epically unhelpful.  Is it still updating?  Who knows?  Is it okay to reboot?  I hope so because that’s exactly what I did at 11:30 last night.  Barracuda’s web site was as useless as always and reminded me of when our device wasn’t properly authenticating e-mail addresses against our Active Directory LDAP setup.  The fix in that case was to point it at a Global Catalog AD server instead of just any AD server, but just like with my ROM issue Barracuda’s support and web site provided no clues, just the standard bunk that comes packed in with the device itself.  I guess since their tech support never solves any problems, they wouldn’t have any service call details to post to their site.