Today I came a across a great post about distributed security alerting by Ryan Huber who works in the security team over at Slack. The post is almost 2 weeks old now but I’m thankful for @lady_nerd and @nickmalcolm for bringing it to my attention as the underlying concepts are identical to what we’ve been striving towards at ThisData for a while now.
In Ryan’s post he talks about various options on dealing with a rapidly increasing number of security alerts and the very real problem of alert fatigue. You see, what tends to happen in IT departments is that they get thousands of system-generated events, anomalies and alerts every day. Inspecting every one of these events becomes impossible so they start to filter and skip which often leads to attackers slipping through the cracks and breaching a system.
There have been so many stories like this over the past few years with one of the more prominent ones being the massive Target retail chain getting over 40M debit and credit card numbers stolen back in 2013 due to IT ignoring security alerts.
So how do we reduce alert fatigue?
Ryan came up with a great idea on how to distribute the load of checking alerts and make it easier for the security team to save time by bringing confirmed threats to the surface.
What if you send alerts directly to the person(s) involved and ask them whether the event was good or bad? What if you also do it the moment you detect something strange? This means you will be asking exactly the right person whether they did something within moments of seeing the activity. Now we’re onto something!" - Ryan Huber
The idea is beautifully simple, it increases the chances of catching the real threats and reduces time wasted by IT or security teams researching false positives.
In Slack’s case they have applied this concept to monitoring internal systems. Over at ThisData we focused on building a service for developers that identifies users, and has an automated feedback loop to confirm with those users if we suspect an action was not performed by them.
Was this you?
We call this process “Was this you?” notifications and we focus on making those notifications simple, informed and easy to understand. Often the people receiving them are not tech savvy so there is no need to freak them out with jargon when we can simply say, “we noticed you signed in from Mars for the first time, was this you?”.
The process is simple but it works and the most important thing is that the time between the incident and the response is dramatically reduced. In data breaches every minute that goes by can cost a lot, so the faster you can respond the better.
We launched this service in January so that other companies and apps can build intelligent security workflows that help protect their users, automate their response and reduce time spent investigating false alarms. We knew this was a great way to improve app security, but to be honest, we didn’t expect the number of confirmed alerts as we’ve seen so far.
Our customers typically listen for webhooks from us, so as soon as a user responds with “it wasn’t me!” their app can kill the user session or whatever else they need to do to contain the threat. We also push all of these events through to other channels - which brings me full circle back to Slack - and this makes it super easy to keep track of all of the alerts, good, bad or ugly in one place.
So thanks Ryan, for writing about distributed alerting and bringing more attention to a topic that we’re so passionate about.