"Too Important to Patch" - Wait? What?
I recently had a routine "can you help our business partner" type call from a client. Their business partner could receive email from them, but could not send email to them.
After a bit of digging in the SMTP header of a failed note, it turned out that the business partner was running a very old version of QMAIL, which has a problem with ESMTP and DNS responses larger than 512 bytes. My client (the destination for the email) had recently gone to an email scanning service, so the total return on an MX record request was well over 1.5kb.
So far, not so exciting, you say - patch the server and be done with it! So why am I writing this up on isc.edu?
This is where it gets interesting. I called the business partner, and their verbatim response was "Gee, I don't know. Applying that patch will involve taking the mail server down, our CEO won't approve that. Is there some other way to do this?"
Wait, what? Did I hear that right? Let me check my watch - what century is this again? This is a patch from 2007 for goodness sake! I can see needing to follow a change control procedure, schedule an outage, maybe for after-hours, but they are an application development shop, not the Department of Defense! If they're running a mail system that hasn't been patched in 4 years, chances are that someone else already owns them, and they've got bigger problems than just this.
Anyway, after a frank and honest (and tactful, though that part was a bit more difficult) discussion, they did apply the needed patch, along with a truckload of other system updates that had been delayed since forever.
I've encountered a few situations where it makes some snse for system admins to defer patching for extended periods of time:
Servers that support military personnel in active operations are often mandated by policy as "frozen". In our current global environment, these freeze periods can extend into months and years.
Servers that support long-range space exploration missions will often end up running operating systems that are no longer supported, on hardware that has been end-of-lifed years ago, or on hardware or OS's that were one-shot custom efforts. In cases like this, the hardware is generally air-gapped or otherwise isolated from sources of attack.
Some servers in support-challenged situations might also be "frozen" for specified periods of time - if I remember correctly, the servers in some of the Antarctic missions (really, no pun!) are in this category. (If I'm mistaken on this example, I know that sysadmin for those systems is a reader, please correct me!)
So the question I have for our readers is: What situations or applications have you seen that might defer patches and updates for an extended periods of time? Did you consider those reasons or policies to be legitimate? Did you come up with a compromise or workaround to get patches applied, or did you have to follow policy and not apply updates? Did this end up with a system compromise, and if so, did the policy protect the system administrator, or did they end up taking the blame anyway?
I'm really looking forward to feedback from our readers on this, please use the contact form to let us know what you've seen!
===============
Rob VandenBrink Metafore
Comments
- Ignorance and/or arrogance (It's never happened to us before, so it won't happen now).
> Did you consider those reasons or policies to be legitimate?
No.
> Did you come up with a compromise or workaround to get patches applied, or did you have to follow policy and not apply updates?
- Followed their decision/policy. It's not my network - they are the owners.
> Did this end up with a system compromise, and if so, did the policy protect the system administrator, or did they end up taking the blame anyway?
- No - they lucked out. But you know who the fall guy is...
.
PC.Tech
Jul 6th 2011
1 decade ago
James
Jul 6th 2011
1 decade ago
However these consoles are not permitted to see the world and no one is allowed access to the boxes besides me. So while I worry that someone will come along and plug an infected USB stick into the box I have to hope that the rules and locks are enough...
ChrisP
Jul 6th 2011
1 decade ago
My little company has a backup email server running HMail at another location to avoid loosing incoming email... Would this option not suffice for these guys for an overnight cutover?? Yikes!
ChrisP
Jul 6th 2011
1 decade ago
Leitchy
Jul 6th 2011
1 decade ago
Martin Scharm
Jul 6th 2011
1 decade ago
Nick
Jul 6th 2011
1 decade ago
There is a reason for an email trail of you explaining the importance of the updates and what could happen if they do not do it. You keep all correspondence with your explanation to why the patches are important and their "no" responses. This is CYA so WHEN they do get hacked they cannot say it is your fault and any attempt to do so could make them libel.
Lee
Jul 6th 2011
1 decade ago
Either due to FDA controls on the system/app OR vendor's own very odd (read broken) requirements.
Then there's the "exclude the following directories from virus scanning" requirements.
CBob
Jul 6th 2011
1 decade ago
Unfortunately, patches are just as good as the software they fix, and have unintended side effects like breaking applications. And as we know, it is all about the apps!
So, scarce resources will be applied to the big problems, like how to allow cool device of the day to work with exchange. Issues like what version of qmail is running where and whether or not it should be patched will always get fiftg billing or less on the priority scale ... Until something breaks.
So, in conclusion. Testing patches properly is risky. Failure to do so means that when some app breaks, it is a sin for which there may be no forgiveness. Instead, make the cool new toys work. Anything that breaks doing that will be forgiven. Wait for the dull stuff to break on its own. That is the cmm level 1 patching process in brief.
Moving to a hygienic system management process involves discipline and management interest, too characteristics that are often hard to find in the chaotic world of systems administration, imho.
Joe h
Jul 6th 2011
1 decade ago