Splunk published an advisory about a critical security vulnerability in deployment server, which is a component that comes installed (but not fully used) with every Splunk Enterprise installation. However, due to it being very useful, almost every organization I’ve seen (and I’m a Splunk person really) uses it – which makes this even more dangerous.

So what is the vulnerability about?

When you use deployment server, it allows you to create configuration bundles that can be automatically downloaded by Splunk Universal Forwarder (SUF) agents (or other Splunk Enterprise instances such as heavy forwarders). These configuration bundles can, among plain text configuration files also contain binary packages (most commonly used for specific connectors).

The administrator of a deployment server controls which SUF can download what – this can be done by IP addresses, DNS names or architecture. And since those bundles can contain binary files, once fetched by a SUF, as you can probably guess, the SUF will happily execute it. And by default, most SUF agents will run as SYSTEM on Windows …

The published vulnerability (the exploit has not been published yet, and we have not seen any information about exploitation at the time of posting this diary) allows an attacker, that has compromised a single SUF in an organization to abuse the vulnerability and presumably push a new configuration bundle to every other SUF in the organization. As I wrote above, since SUF can download and execute binaries, and as it quite often runs as SYSTEM, this can be translated to pwning a whole organization. Eeek!

What can we do?

Splunk’s answer currently is to update to 9.0, which fixes the vulnerability. However, v9.0 is literally 2 days old so if you decide to go this way be careful!
There are no fixes for any of the older versions! Which means that you are almost certainly affected.

The only solution I am aware of at the moment (and thanks to Boris Kresoja and Alan Osmanagic for testing – Splunk guys working with me) is to disable deployment server so it is down, and bring it up only if you need to push configuration updates. Any bundle that has been already downloaded to a SUF will continue working.

An easy way to do this is to run the following command:

$ /opt/splunk/bin/splunk disable deploy-server

Keep in mind that you need to restart Splunk after the command above – if you don’t, all active connections to deployment server will continue to work.
Beside this we recommend that you install deployment server on a standalone Splunk Enterprise instance, where you can upgrade it to v9.0 (with less risk). It looks as the rest can stay as it is in that case; I’ll update the diary if we get new information.

--
Bojan
@bojanz
NFIGO IS