Putting all of Your Eggs in One Basket - or How NOT to do Layoffs
The recent story about Jason Cornish, a disgruntled employee of pharmaceutical company Shionogi is getting a lot of attention this week. In a nutshell, he resigned after a dispute with management, and was kept on as a consultant for a few months after.
The story then goes that he logged into the network remotely (ie - VPN'd in using his legitimate credentials), then logged into a "secret vSphere console" (I'd call "foul" on that one - there would be no reason to have a "secret" console - my guess is he used the actual corporate vCenter console or used a direct client against ESX, which you can download from any ESX server, so he had rights there as well) then proceeded to delete a large part of the company infrastructure (88 servers in the story I read). The company was offline for "a number of days", and Jason is now facing charges.
This diary isn't about the particulars of this case, it's much more of a common occurrence than you might think. We'll talk a bit about what to do, a bit about what NOT to do, and most important, we'd love to hear your insights and experiences in this area.
First of all, my perspective ...
Separation of duties is super-critical. Unless you are a very small shop, your network people shouldn't have your windows domain admin account, and vice versa. In a small company this can be a real challenge - if you've only got 1 or two people in IT, we generally see a single password that all the admins have. Separation of duties is simple to do in vmWare vSphere - for instance, you can limit the ability to create or delete servers to the few people who should have that right. If you have web administrators or database administrators who need access to the power button, you can give them that and ONLY that.
Hardening your infrastructure is also important. Everything from Active Directory to vSphere to Linux have a "press the enter key 12 times" default install. Unfortunately, in almost all cases, this leaves you with a single default administrator account on every system, with full access to everything. Hardening hosts will generally work hand-in-hand with separation of duties, in most cases the default / overall administator credentials are left either unused or deleted. In the case of network or virtual infrastructure, you'll often back-end it to an enterprise directory, often Active Directory via LDAP (or preferably LDAPs), Kerberos or RADIUS. This can often be a big help if you have audits integrated into your change control process (to verify who made a particular change, or to track down who made an unauthorized change)
HR processes need to be integrated with IT. This isn't news to most IT folks. They need to know when people are hired to arrange for credentials and hardware. But much more important, IT needs to be involved in termination. They need to collect the gear, revoke passwords and the like, in many cases during the exit interview. When an IT admin is layed off, fired or otherwise terminated, it's often a multi-person effort to change all the passwords - domain admin credentials, passwords for local hosts, virtual infrastructure admins, and the myriad of network devices (routers, switches, firewalls, load balancers, etc). If you've integrated your authentication back to a common directory, this can be a very quick process (delete or disable one account). In this case, a known disgruntled employee was kept on after termination as a consultant with admin rights. You would think that if HR as aware of this, or any corporate manager knew of it for that matter, that common sense would kick in, and the red flags would be going up well before they got to the point of recovering a decimated infrastructure. Yea, I know the proverb about common sense not being so common, but still ....
Backups are important. It's ironic that I'm spelling this out in the diary adjacent to the one on the fallout from the 2003 power outage where we talk about how far we've come in BCP (Business Continuity Planing), but it's worth repeating. Being out for "a number of days" is silly in a virtual environment - it should be *easy* to recover, that's one of the reasons people virtualize. It's very possible, and very often recommended, that all servers in a virtual infrastructure (Hyper-V, XEN, vmWare, KVM whatever), be imaged off to disk each day - the ability and APIs for this are available in all of them. The images are then spooled off to tape, which is a much slower process. This would normally mean that if a server is compromised or in this case deleted, you should be able to recover that server in a matter of minutes (as fast as you can spin the disks). This assumes that you have someone left in the organization that knows how to do this (see the next section).
Don't give away the keys. Organizations need to maintain a core level of technical competancy. This may seem like an odd thing for me to say (I'm a consultant), but you need actual employees of the company who "own" the passwords, and have the skills to do backups, restores, user creation, all those core business IT tasks that are on the checklist of each and every compliance regulation. In a small shop, it's common for IT to give consultants their actual administrative credentials, but it's much more common these days to get named accounts so that activity can be tracked, these accounts are often time limited either for a single day or the duration of the engagement.
I'd very much like to see a discussion on this - what processes do you have in place, or what processes have you seen in other organizations to deal with IT "root level" users - how are they brought on board, how are they controlled day-to-day, and how are things handled as they leave the organization? I'm positive that I've missed things, please help fill in the blanks !
If I'm off-base on any of my recommendations or comments above, by all means let me know that too !
===============
Rob VandenBrink
Metafore
Comments
If you accept the premise that your greatest risk comes from the person(s) you hire -- even the "gruntled" ;) -- then allowing an individual who left on less than genial terms to continue consulting is insane! Proper termination procedures will ensure prompt removal of access rights and a general cleanup such as you mentioned.
Drew
Aug 17th 2011
1 decade ago
Hiring the person who left back in as a consultant is a common "solution" to these - fixing one bad idea with another bad idea. We're seeing this week what the risk is of this approach.
Rob VandenBrink
Aug 17th 2011
1 decade ago
MTodd
Aug 17th 2011
1 decade ago
No way, not a chance - way too much risk to me, and they should have thought things through before deciding I was expendable. I want nothing to do with that company ever again, and I'm certainly not going to take the risk of something going wrong while I have access and getting blamed. That may even be the case here, who knows?
So, stupidity on both sides here, but mostly the company's - he wouldn't have been disgruntled in the first place if they hadn't fired him, then they compounded the error by using him after that. He erred by accepting the work, but he may have done so because he wanted to use the access to get them back.
Don't, just don't. How much will this mess cost that company? I bet a lot more than it would have cost to find a consultant rather than use their ex-employee.
Fritz
Aug 17th 2011
1 decade ago
Establishing the infrastructure that limits access without using superuser takes time and effort. Separation of duties to use the accounts requires bodies. All of the above takes money.
Superuser access provides the most privileges with the least amount of time, effort, bodies and money. You now trust the watch-keeper to tell you the time. Who is watching the watch-keeper?
Unconditional trust is a risk. To mitigate risk requires and investment.
The balancing act of investing money across low risk off occurring verses large impact or high risk of occurring verses small impact is a balancing act of risk management decision making.
It can be unnerving if you reflect on the number of IT people who have grown-up in basements of home soothed by the gentle purring of their server fans while they honed the computer skills so valued by companies and stunted social and emotional skills needed by society.
I will speculate the problem of watch-keepers out of sync happens more often than disclosed in the media. If it was a rampant problem the time, effort, bodies and money need to establish the infrastructure to support the unconditional risk would be a solution in the toolboxes of MBA that manage companies instead of “are you done yet”.
colporteur
Aug 18th 2011
1 decade ago
it could mean that he SSH'd into the servers, and used the command line to shutdown and delete things.
He was a consulant... it could very well be that he was in a position of trust and needed this access.
Suggestions like enforcing some kind of specific assignment 'separation of duties' are burdensome with not a clear proven benefit.
Separation of duties implies imposing limits on each team members' area of responsibilities, which means more employees are required
If someone's duty is to troubleshoot fix, or preen a VMware environment, then as a result, they're in a position of trust, and you're still screwed if they misbehave at any time.
The only assured saving grace is to have good backups....
Mysid
Aug 19th 2011
1 decade ago
Jason
Aug 19th 2011
1 decade ago
Moriah
Aug 19th 2011
1 decade ago