It's important to remember there are different types of analytics, both of which are important for any organization. Real-time analytics are what I would consider to be tactical. If your site is getting slammed you need to know about it - right now! And then you need to act. If you only focus on the RT analytics though it's easy to get stuck in local minimums, miss the forest through the trees, or simply not realize you're measuring the wrong things.
There are also strategic analytics. When looking at strategy, sufficient time periods are generally required. This type of analysis guides decisions in terms of months, quarters, and years. Strategic analytics are what drive the business long term. They provide the decions makers the data and visibility they need to solidify their vision.
Finally, I like to think of analytic data in 3 distinct iterative stages:
First is the data. This is the raw data spinning off the multitude of systems in any company. In any slightly complex system there are many datapoints that can be captured and counted. Companies often have entire systems just to capture the data, and this is before any analysis can occur.
Next is the information stage. In this stage all of that raw data is turned into some sort of reporting. It's still a lot of data, but is in a format that can start to be analyzed and acted upon (tactical reporting). In the past I've used various data warehousing solutions to execute on this step.
Finally, the organization turns the information into long term knowledge. This step occurs when the data has been analyzed and used to drive champion/challenger, A/B testing, etc... and those results are rolled back into the organization (vision and long term organizational changes). This knowledge also helps the organization discover what other datapoints need to be found and/or captured and the process starts again.
Different analytics are important at different stages of a company's development. A place like Microsoft might place little to no value in RT analytics most of the time. They tend to move too slowly to be able to do anything in real time. OTOH, a struggling startup trying to grow from 500 to 1000 users might look almost exclusively at the RT analytics.
I totally agree with the sentiment in the blog post. As someone who has built and now compulsively checks a real time dashboard however, I'll enumerate one case where I've found real time really valuable.
I'm the product owner of a social game, and we are blessed with:
1) high traffic
2) tools that make it easy to test sales, promotions, contests in real time
3) a team that is capable of doing multiple meaningful releases a week reliably
The result of this is that we end up making meaningful changes to our product on a daily basis. With our high traffic, I can reliably tell you within one hour (on the outside) whether a promotion, contest or feature is having its intended impact, especially once I:
1) Benchmark against recent averages (and take into account the volatility in the metric I'm measuring... everything is framed as how many standard deviations above or below recent averages)
2) Compare against recent trends.
3) Compare against my expectations/hypothesis. After a couple years of looking at realtime KPIs move in response to changes, I've gotten pretty good at forecasting results.
Obviously, I also look at metrics at a daily level and track weekly cohorted metrics out for months to make sure strategically we're maximizing for a global maxima and not a local one. But there are a lot of pretty realtime businesses where tactics are extremely important that could really benefit from realtime dashboards.
Great post - if anything, I think this problem goes way beyond just real-time dashboards to extend to the vast majority of analysis done today (both by web and more traditional companies).
The statement that particularly resonated for me was [figure out] "if you’re looking at stats now because you’re curious and impatient, or because those stats will actually drive business decisions", but I'd take it one step further.
Analysis and stats are incredibly valuable when:
1) they are applied to a real business decision that is tied to actual value
2) you are willing to change what you are doing based on the result
3) you don't already know the answer
4) you have sufficient confidence in the result to act on it.
We need to get more used to stopping and thinking before we start analysis by laying out the decision we need help making, the different paths we're willing to take, and the amount of confidence we'll need to change our decisions. I could definitely be a lot more disciplined about it.
(I actually wrote a blog post last week on this same topic that lays out the above criteria in more detail and might be useful - though that was a reaction to the profusion of ads out there calling Big Data and Analytics "hot", that ignore how they actually drive real value. Blog post is here: http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...)
There has been a lot of interesting discussions on growth hacking here on HN lately. This article raises the important point that long term growth has to be the goal as opposed to increase in sign-ups or page views. Feeding the Internet troll, for example, may not always lead to long term growth.
This is why I chose not to provide real-time analytics for Instahero[1]. Apart from being very constraining, implementation-wise, it simply does not provide that much business value.
Much in the same way that looking at an A/B test invalidates it, it's better if you let your metrics converge before you look.
There are also strategic analytics. When looking at strategy, sufficient time periods are generally required. This type of analysis guides decisions in terms of months, quarters, and years. Strategic analytics are what drive the business long term. They provide the decions makers the data and visibility they need to solidify their vision.
Finally, I like to think of analytic data in 3 distinct iterative stages:
First is the data. This is the raw data spinning off the multitude of systems in any company. In any slightly complex system there are many datapoints that can be captured and counted. Companies often have entire systems just to capture the data, and this is before any analysis can occur.
Next is the information stage. In this stage all of that raw data is turned into some sort of reporting. It's still a lot of data, but is in a format that can start to be analyzed and acted upon (tactical reporting). In the past I've used various data warehousing solutions to execute on this step.
Finally, the organization turns the information into long term knowledge. This step occurs when the data has been analyzed and used to drive champion/challenger, A/B testing, etc... and those results are rolled back into the organization (vision and long term organizational changes). This knowledge also helps the organization discover what other datapoints need to be found and/or captured and the process starts again.