I have a client who's done the right thing, they've broken out their test environment from their production environment.  The production environment is in a colocation facility, and uses a different firewall.  The test environment is in the office location, and shares the office subnet and the office firewall.  So sort-of the right thing, they're moving in the right direction - I would have given the test lab it's own firewalled DMZ subnet.

About two years ago, one of the server admins asked the office firewall administrator to open port 3389 (RDP) to a test box, so that they could continue their build at home.  Not a great solution - I would have told him to VPN in and do it without changing the firewall - but it was done, the build got done and life moved on.

Unfortunately, the firewall change was not documented, was not remembered and was not backed out.

Fast forward 2 years.  The two folks from 2 years back have both moved on to other positions and/or companies, and a new server admin is building a new Hyper-V server in the test environment.  They're just about to deploy to producion when he notices RDP connections to it from our friends in China.  Yes, that undocumented change had come home to roost!

So, after we did the post mortem, what did they learn?

• There's no fixing a compromised hypervisor - NFO (Nuke from Orbit) - repartition the RAID Array and starting over is always the best advice.
• Hypervisors don't need a GUI - they shouldn't be RDP'ing into that box for admin in the first place.
• DOCUMENT ALL FIREWALL CHANGES.  HAVE A CHANGE CONTROL PROCESS.  Happily, they've got a formal change control process now.  On the firewall, there's an assessment step on all changes, to decide if the requested change is a good idea in the first place (open RDP was a singularly BAD idea).
• Finally, they now run a basic NMAP scan (all addresses in the range, all ports) of the office environment from the colo, and the results are run through diff, comparing it for changes against yesterday's results.  This client is lucky in this regard because they have 2 separate locations that can scan each other, but in a more typical situation, the folks responsible for security might do this from their laptop, scanning from home after work or before driving in each day.

You'd be surprised what a full port scan might find - those issues we're stuck with on open ports on home firewalls (https://isc.sans.edu/forums/diary/Scans+Increase+for+New+Linksys+Backdoor+32764+TCP+/17336 and https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 for instance) would have been caught a long time ago if more folks scanned their infrastructure from the untrusted outside network!  Mind you, typically home users never patch their firewalls anyway, so all those open PNP and other backdoor ports are with us for the long haul now.

Do you regularly scan your firewall from the outside?  Does your scan highlight changes, or are you looking for just vulnerabilities (using Nessus or similar) rather than changes?   Let us know in our comment form below.

===============
Rob VandenBrink
Metafore