Why Outlier Detection Won't Save You

by John Haldeman, Enterprise Architect

Now that it's been out for awhile, I feel like it's time for some hard truths on Guardium Outlier Detection. Before I get into some of it's limitations, I want say that it does indeed work, and it is indeed useful and powerful. This post is meant to serve as a healthy discussion on the practical problems of using it. To be clear, I would rather use the function than not - ie: it provides more value than what it costs to deal with the issues below.

1) Abnormal non-malicious stuff happens all the time

This is one of the "top two" issues with the outlier analysis of this sort. The fact of the matter is unusual things happen in databases all the time. DBAs are in particular very random. Day to day they may touch objects in a database that they never have before. There are so many system objects and combinations of things they could access that a lot of what outliers will show you is the DBA doing their jobs.

That being said, it's not unhelpful to have this information, it's just going to be reported to you constantly (I also hesitate to white list DBA work). On top of this, pretty much everyone reports on the activity from DBAs already, so now you have to process and review the same data - twice (once in outliers, once in your privileged user reports).

Of course you can manage this, but never succumb to the thinking that outliers is going to solve all the problems with activity monitoring review and whitelist management.

2) High risk behavior is likely happening all the time (and as a result are trained into outliers)

True story: Alice is a business analyst in charge of reviewing records related to healthcare claims. She has modest technical skills and knows enough to be able to run a database query on a daily basis from her workstation under her desk to transfer her claims data from a remote database that you monitor to a local access database for easier review. That access database is consequently on a share that has READ granted to EVERYONE.

This is trained into outliers as "normal" - it is normal - but clearly it's tremendously risky behavior. Outliers help you analyze and respond to certain risky behaviour, but not all of it. My bet is that in your database environment risky behavior is occurring all the time.

3) It doesn't tell you anything about what you don't log

This may seem obvious, but outliers performs analysis on data already captured by Guardium. If your policy prevents the SQL constructs from being captured for certain sessions, outliers is not going to tell you anything about those constructs. It's also a best practice not to log constructs for everything in most environments, and so a lot of effort still has to go into managing what's logged. Outleirs doesn't absolve you from that responsibility.

4) If it tells you what's new, and you start to log something new, it's tells you

This tautological statement boils down to thinking about what's being logged when and where. If you change your policy and start logging a wide variety of new events, outliers is going to throw a large amount of new hits your way at least in the short term. If you are in the habit of frequently changing your policy, this will happen a lot.  

Additionally, outlier models are local to the collectors which means in the event of a failover, it's not going to have the history for the server that's failing over to the new collector. This also generates a lot of new events for you to process.

Again, this isn't a critique about outliers, but I think it's important to think about it differently than maybe how it's discussed sometimes. Outliers is another data point you get to manage, analyze and whitelist. The developerworks article I link to at the top of this blog posting have some great strategies for performing that management. Outliers doesn't make life easier. It makes Guardium more valuable because it's giving you an additional data point to analyze. it doesn't make the other data points you need to analyze go away.