Cold Call Blues

To every salesman who cold calls and cold e-mails me I have a message: I know your situation, but there’s nothing I can do for you.  No, I can’t answer your call to talk for a minute, because it easily turns into five after I sit through your pitch.  No, I do not owe you a ‘courtesy callback’ just because you were one of the many who have left me a voicemail.  As well, I have no interest in any of the following: resold telco services, rebuilt toner cartridges, or ‘scrap’ dealers looking for six month old computer gear.  My place of work has needed contracting services in the past which were typically attached to some piece of software, I myself haven’t needed the services of a generic technical consultant for nearly a decade, so those too are a waste of time.   On that note, I particularly don’t care for cold calls from consulting firms that aren’t even in my state.

I have a soft spot for many of these companies as I used to work for consulting companies.  Even now my job depends on the ability of our sales staff to cajole customers into either buying our products or buying more of them.  One vendor in particular, from whom I have bought no products from ever, has been cold calling me once every week or two for ten years.  About five years ago I tried to tell them to give it up but they’ve persisted.  Needless to say I’ve been through generations of sales people; every six months it’s a different voice that’s desperate to get a sale, any sale, from my disembodied voicemail call tag.

Unfortunately you cold-callers, since we migrated to a new voicemail system many years ago I now use it to expedite your voicemails right into the digital void, never to be heard, ever.  I wish I had it in me to manage a half dozen vendor relationships, but we do not have that much business and what we do have I need to use as a carrot for our occasional goofy request from our vendor with whom we have a long term trusted relationship.

Diagnosing Antiquated Citrix

For quite a while I had an issue on one of our Citrix servers (let’s call it termserv1) where I was receiving “This patch package could not be opened”/”Package corrupt” errors when trying to install patches for Presentation Server 4.0.  Since the server doesn’t hang directly off the Internet I had put off fixing it for several years in the not unrealistic hope that I might one day be able to ‘upgrade’ out of the issue.

Our other Citrix servers are quite a bit more important and they had to be patched.  After every patch I waited for something bad to happen where the newly patched boxes would have some communication issue with the unpatched server.  I had gotten lucky for a long time, but this past week my luck finally ran out when I installed the version 6 hotfix rollup (what will probably be the last hotfix rollup for Presentation Server 4.0).  The ‘good’ servers reported that they were unable to contact the licensing server.  I figured, somewhat incorrectly as it turns out, that the issue was related to the fact that the termserv1, which was the license server, had not been patched.

The reason I had put off fixing the issue is that the fix was rather non-existent.  The first step was to delete the Presentation Server install entries by using the Windows Installer Cleanup Utility.  The second step was to re-install Presentation Server (which for me also required renaming the files ‘ctxdwavo.exe’ and  ‘mfreg.exe’ AND rebooting).  The third step was to hope for the best, and barring that, to fix whatever didn’t work, which in this case meant pretty much everything.

The core issue was that termserv1 was the main server that I relied on in our transition from MetaFrame 1.8 to Presentation Server 4.0 and it had taken quite a bit of work and a long phone call to Citrix to get all of the licensing pieces to work.  From that point on I relied on termserv1 to be the master browser/data collector and main license server.  As it turned out, the licensing issue was related to the fact that the licensing service from Citrix is not patched with hotfix updates and is in fact a separate download.  However, after reinstalling on termserv1 the ICA browser was not working and the server would refuse direct connections with either ‘Protocol Driver Error’ or  just ‘cannot connect to the Citrix XenApp server’  both of which can be an indication of almost any issue.

Emboldened, I decided to uninstall Presentation Server (now that the installer was working) and reinstall.  The uninstall went fine, however when I went to reinstall it refused to join a farm without a SQL database link (we use the included Access database).  I wound up having to selectively restore the unpatched Citrix back onto the box.  After the restore the browser worked again, but ICA connections to the server did not.  Figuring that I would just have to fight my way through it I reapplied the patch and set out to find out what was wrong.

Since I was getting the unhelpful errors from the XenApp client I decided to see if I could get the web interface working to the point that I could get a helpful error or two.  This turned out to be a day long time sink; getting the antiquated version of the Citrix web functions working on a newer server (Win2k3 r2) whose own version of Presentation server wasn’t working anyway, was asking too much given the immediacy of the need to get the server back into full operation again (although it hosts production applications, they’re used sparingly).  I got to the point of getting the “exception has been thrown by the target of an invocation” when trying to create a new web server before giving up.

Frustrated, I turned back to the regular clients.  My first plan of attack was the fact that pointing the XenApp client at one of the two ‘good’ server worked, and termserv1 didn’t, which told me that there was a browser issue of some sort.  Somewhere along the line I came across a Citrix forum post that suggested deleting and recreating the ICA connection.  After doing that the published items still didn’t work, but direct connections did!  After running ‘query farm /load’ I discovered that termserv1 and the other two servers didn’t really see themselves in the same farm (‘query server’ would return all the servers, but ‘query farm /load’  would return incomplete lists).  I performed some actions with ‘dscheck’ and ‘dsmaint failover’, but the real fix (I think) was going into the farm properties in the MetaFrame management console and toggling the affinity between the servers.  Before doing that the servers had two different affinity sets.  Setting it one place made it consistent and everything in Program Neighborhood began working properly.

*UPDATE: If any of the above is helpful it’s a coincidence.  After rebooting termserv1 the IMA service failed to come back up and the management console was inaccessible on any of the servers.  I hacked around for the better part of a (Sun)day, but as it was closing in on midnight I threw in the towel and rebuilt the farm.  I was at least able to rely on a screen shot I had taken as a business continuity hedge so that I’d know which apps I had to rebuild.

ROM Updates

For the past ten years I’ve been responsible for the Nortel switches at my current employer but with this last upgrade our vendor talked me into getting HP Procurves instead.  I don’t regret the decision as they were quite affordable, but I’m underwhelmed by the feature set on the routers compared to our older Nortel 460s.  The VLANs don’t seem to set up as smoothly and the initial ROM version that they shipped with did not include an interface on the web management console for updating the firmware.  “That’s okay”, I figured “I’ll update via TFTP like I had for years with my Cisco and Nortel stuff”. As it turns out that’s a no-go as well, the initial ROM version only ships with the ability to update over the serial port.  I’m puzzled as to how HP can ship a product in 2009 that doesn’t include features that have been included for the last ten years in even the cheapest networking gear.

It wasn’t long after my adventure with the switches that I had to update the ILO card on our AiO StorageWorks equipment (1200r if I recall correctly).  I need to point out that first of all the ILO card that comes with the AiO is a vastly inferior to the ILO2 card that comes on the later model Proliants (in fact, it’s labeled as a ‘LO100i’ card and isn’t an ILO at all).  It goes without saying that the equipment didn’t ship with the latest firmware on the ILO card, but the update had to be performed using basically the same mechanism that I used to update Compaq equipment in the mid-nineties: reboot onto an image.  It was USB at least and not floppy, but still, what’s the point of a remote management adapter if you have to be physically present to update it?

Last on my list is my Barracuda spam firewall.  I made the mistake of updating the firmware last night at around 10:30 at night.  The web based update screen provides a VERY poor indication of how the update is advancing.  The unhelpful gauge on the ‘update’ web page eventually locked up completely making it epically unhelpful.  Is it still updating?  Who knows?  Is it okay to reboot?  I hope so because that’s exactly what I did at 11:30 last night.  Barracuda’s web site was as useless as always and reminded me of when our device wasn’t properly authenticating e-mail addresses against our Active Directory LDAP setup.  The fix in that case was to point it at a Global Catalog AD server instead of just any AD server, but just like with my ROM issue Barracuda’s support and web site provided no clues, just the standard bunk that comes packed in with the device itself.  I guess since their tech support never solves any problems, they wouldn’t have any service call details to post to their site.

Custom Virus Removal

Over the past several years I’ve had to clean malware from many PCs for friends and acquaintances.  This used to be rather trivial at first, but has grown to be a rather huge, complex undertaking as time has progressed with some cleanings taking several days.  The reason for this is two-fold.  First, malware writers have gotten much more clever and now put together software that is difficult to impossible to remove.  Secondly, attack vectors are much more ubiquitous.  At one point in time only ‘naughty’ computer users, those who trafficked in illegal software or pornography, got horrible malware infections but now the attacks are much more subtle.  Even though I consider myself somewhat wise in the area of computer security, I’ve seen a few phishing type emails that I’ve almost fell for, and that’s to say nothing a of the virus writers who infect normally safe web sites with their malicious code.

Typically when I get an infected machine it has either has no anti-virus package, or it has the original demo of an anti-virus package that came with computer and is now three years out of date.  Needless to say, the best thing that you can do to protect your computer is to have an anti-virus package installed and up to date.  I know they are overpriced packages of dreadful software that slows down your computer, but the cure is still better than the disease.  It’s worth pointing out as well, that many ISPs now offer a free anti-virus package for their customers to use.

Anyway, since the infected PCs I get have no anti-virus installed, and typically you cannot install much of anything on an actively infected system, the system needs to be manually cleaned to the point of making it operational.  What usually makes this easier is having an operational computer available to do web searches and other work on.  First, I reboot in Safe mode (press F8 just before XP boots). If possible, I’ll just use a ‘system restore point’ (you’re prompted on doing so when rebooting in Safe mode), but this is only possible if you know the approximate time range of the infection (as well, any apps installed after the restore point will be removed). If that’s not possible then, this is where it gets more painful, I first will manually clean out the ‘Run’ key of the Windows registry while making a note of what files are listed there.  More often than not I find myself doing web searches on some files to see if they are legit or not.  If they are not legit, I’ll make a note of those malware packages and what other files they rely on so that I can hunt them down in the system.  What I have found to be most effective in dealing with malware files is replacing them with an empty read-only file of the same name.  This way if the virus tries to recreate itself it will have trouble doing so.

So for instance, if a piece of malware has two files named ’PS1.exe’ and ‘winsysrun.exe’ I delete those two files and copy a blank text document in as those same names. This will confuse/corrupt most malware, particularly if the new files cannot be written to. In order to better ensure this I will (on systems not running XP pro) use a painful program called ‘cacls’ (or the related, very slightly less painful ‘fileacl’) to remove all access rights to the file.  The main problem with this whole procedure is that there are typically MANY file names so I’ll wind up creating a custom batch file to do most of the heavy lifting. I’ll do this one of two ways (or both).  On my extra, operational computer, I will download a list of files associated with the malware from an anti-virus site and plug those into a batch file and/or I will get a directory listing by date of the \windows\system32 directory where many virus files can be found.  To aid in that search I’ll  add the ‘company’ item to the columns in the detail view of the directory so that I can determine which date groupings to look for.  Any files that do not have a company listed should be viewed with suspicion and ‘googled’.

Once I have my lists of files I’ll then either manually copy in my ‘del’ and ‘copy’ commands (and ‘cacls’ and or ‘attrib’), use search and replace methods (adding in a comma where I want my commands to go and then search/replace the easier to type comma with the commands that I want), or as I did in one extreme case, compiling the list into Excel and adding the commands in columns and exporting it as a CSV and just ‘search/replace’-ing the commas with spaces.

At this point the viruses on the PC will typically be disabled enough that the computer can be booted up and an anti-virus package installed to clean off the system.  Typically you will want to do this with the computer NOT hooked to the Internet so that usually means porting the virus signature updates from the clean computer over to the ‘dirty’ computer via, ideally, CD media (as non-read only USB sticks can become infection vectors themselves).   I’ll often put the computer through two full scans of at least one package and a third scan through a different package just to verify it’s integrity (such as free web based scanners from Panda or even Adaware).

On a final note, if a computer has a ‘root kit’ malware installed, this means that portions of the OS itself have been replaced with malicious files.  Detection of this software is difficult to say the least and cleaning them off is virtually impossible.  The only real resolution for a root kit infection is to format the drive.  This makes it all the more important to get a system restore disk of some sort from the manufacturer.  Unfortunately a fair portion of the malware I have seen recently has been of this sort and the computers I have worked on that were in this state were ‘uncleanable’.

Visual Basic Project – Positive Pay translation program

Last year the company I work for decided to begin using the Positive Pay process offered by Key Total Treasury*.  The issue we had when we were looking to implement was that we had three separate bank accounts and no easy way to program an export within each system to match the format needed for the Positive Pay file used by Key Bank.  Fortunately though, we could get CSV (comma delimited) check reports rather easily out of each system.  Although I knew next to nothing about programming in any of the three systems that we use, I knew that I could cobble something together in Visual Basic that could translate, or convert, the CSV file formats to the semi-proprietary fixed width format required by Key Bank.  After a period of use, I requested permission to make a more generic version of our translator programs that others might make use of.  In my spare time I acquired enough knowledge about XML programming to put together the program below that uses an XML initialization file:

http://cid-f206759e5151f410.skydrive.live.com/self.aspx/PositivePayFiles/positive%20pay%20translator.zip (requires a Live ID I think)

The 2.0 version of the .Net runtime is required, and also although it’s not required I would recommend compiling your own version from the source due to the sensitive nature of the information that the program handles (otherwise the executable is under the ‘bin’ folder).  If you do not have Visual Studio, you can download the free Express version.  If you do not use Windows, Mono’s migration tool states that the program should run under the Mono framework, though I have no experience in that area.

When exporting a CSV from your system the fields will need to be in the following order (unless you change the ‘Read_Exported_File’ method in the program):

  • The first field is the check number
  • The second field is the check date
  • The third field is the check amount
  • The fourth field is the void code
  • The fifth field is the vendor name

With the program is an example of the XML configuration file (ppaysettings.xml) which has the following options:

  • AcctName – generally the bank’s name.  It is used for operator verification
  • AcctNum – the bank account number that will appear in the positive pay file
  • VoidCode – string code in the CSV that designates that a check has been voided (can be blank)
  • NonVoidCode – string code in the CSV that designates that a check has NOT been voided (can be blank)
  • OmitPayee – do not put the customer name in the Positive Pay file.  Required for checks printed with dot matrix printers (True or False only, not blank)

In the not too distant future I should have a more fully vetted version along with some exception conditions that would be related to how wire transfers are handled.

As a note, this version drops checks that are canceled in the same run.  This is typically due to a printer malfunction and since the check is never cut, it is never verified against the positive pay data anyway.

Notes 11/28/2008:

This is an early build and may not be bug free as I haven’t tested it fully.  The information in the Paysettings.xml file is not required, but the file itself is at this time.

*Key Bank/Key Total Treasury does not endorse anything to do with this program