When I meet with people running analytics in their enterprises, they almost always claim: "We’ve got everything that our business needs!". Each time I recall the famous quote that "640 kb should be enough for anybody". Things often get even more funny if some execs representing business departments are in the same room. One common thing they usually say at this point is "Then why is it that everything we ask for is 6 months and 3 million dollars?".
Surely, both sides have a lot of sensible arguments. The Analytics Team will say that deploying any new environment based on multiple sources or complex data requires a proper requirements analysis, data modelling and fixing, setting up ETL processes and, on top of that, enabling some kind of analytics for those who have not spent their lives on SQL and UML models.
Business, bursting into tears at this point, often cries out "But we do NOT know what questions we will want to ask! We also HAVEN’T got as much time to make decisions as we need to answer all the questions we already have".
Now, the big question is: Who is right and who is wrong?
Or maybe let’s formulate the question anew and change the rules of the game: What makes both approaches correct?
Analytics Departments are spending their lives doing heavy lifting in the data world, and their results are clearly visible in data warehouses, ETLs and all the other parts of infrastructure invisible to business. But the infrastructure is there.
Business doesn’t see the capabilities of infrastructure and is frustrated with the time they have to wait, as they have already moved most of their processes into some form of agile operations, combined with a lean/kaizen approach. They know that "fast beat big" and their competition knows that too.
“Waterfall development methodology, which follows a seemingly reasonable pattern of gathering requirements, designing a product, and building and testing. The problem is that requirements are almost always wrong. The Waterfall method does not allow for mid-course corrections, and the beginning-to-end cycle may span a year or even many years.”
If you do not want your analytics stack to become extinct in arduous efforts to continuously improve the infrastructure you have built, prototyping is the missing link between your business and your analytics. Without the ability to enable users to check things quickly, to model and remodel their hypotheses as well as to rapidly verify possibilities, you’re just growing bigger and bigger until one day a meteorite falls down that’ll make you extinct. Nobody will care though, because the world will be populated by those who have learned how to adapt and evolve faster.
Of course, adapting the prototyping approach for small and simple data requires almost no effort and is already here, with the most powerful analytics tool – Excel. However, when it comes to enterprise data, this gap has been filled only with armies of data scientists and hoods claiming that the opensource file storing technology can save us all, making ties even more frustrated with growing costs and increasing divergence between business and insights.
Nothing has to be thrown away though, and you don’t have to choose sides either. Adopting prototyping for enterprise data analysis simply eliminates the problem:
What benefits could be reaped of adopting prototyping and an agile approach to enterprise analytics? Well, ask any of the business representatives if they ever felt that investing energy in trying to get full data for their decision-making processes is a waste of time? How many times have people decided it’s not worth spending their time and company money on insights?
What if we eliminate these barriers by technology? There’s one perfect example: Google. Nobody is scared of googling insights anymore and you know how much it has changed our efficiency and access to insights. Enabling open access to silo'd enterprise data will be the next big step, even though it seems hard today.
Once we know something, we find it hard to imagine what it was like not to know it.
Chip & Dan Heath, Authors of Made to Stick, Switch