There is more information available as to what Big Data really mean. In the type of business that most of us deal with daily, you are likely swamped by huge amount of structured or unstructured data that is entirely, partially or not at all collected. Some of the reasons to collect that data are for competitive advantages, network security, operational issues, etc. but the power of that information is really to make timely decisions.

A study conducted by Forester in 2011 estimated "[...] that firms effectively utilize less than 5% of available data. Why so little? The rest is simply too expensive to deal with."[1] That still leaves 95% of untouched and unanalyzed data. Depending who you ask what is big data to you; you may get different answers such as volume, speed, variety, type and quality. Depending of the size of the network, you may lack the storage capacity to ingest and process everything the network is capable of generating and made difficult choices on what is more important to collect to make those timely decisions.

I think in order to make significant gain in collecting Big Data, there is a need comprehensive approach to managing data, how it is analyzed to gain information intelligence. That means choosing the right data, turning structured or unstructured into a common format (i.e. CEF is a widely supported format), reduce data footprint by keeping and aggregating a single copy of similar data (deduplication) and an archiving policy for old data.

We would like to hear from you. Are you currently evaluating what to do with your data and base on your current security posture, do you think the data you currently collect is enough to get valuable insight as to what is going on inside your network?

Note: If interested in sharing, Stephen Northcutt is currently looking for SEIM/SIEM success stories.

[1] http://blogs.forrester.com/brian_hopkins/11-09-30-big_data_will_help_shape_your_markets_next_big_winners
[2] https://protect724.hp.com/docs/DOC-1072
[3] https://www.linkedin.com/pulse/seimsiem-success-story-request-stephen-northcutt?trk=mp-reader-card

-----------

Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot edu