Because of the way they are deployed, less than 10% of employees regularly use business intelligence (BI) tools. Forrester Research puts the fraction at between 5% and 9%. And of that user base, anecdotes abound of daily reports taking several days just to look through.
BI is plagued with two different kinds of backlogs: firstly, the almost unmentionable problem of implementation and integration delays and secondly, the human backlog of painstakingly searching through reports looking for insight and trends.
“BI works exceptionally well at turning database rows into useful insight,” says Anton van Heerden, GM at Altech ISIS. “But the explosion of data and the limited deployment among employees has put pressure on those users who have to try and make sense of the reports.”
Van Heerden doesn’t go far as to say that big data has damaged BI beyond repair but he does think that companies need a new approach when thinking of mining big data.
“Today’s BI deployments need a human, preferably with some kind of domain expertise, looking at the reports and seeing what to make of them,” he says. “But as data scales up, simply throwing hardware and faster networks at the problem won’t solve the underlying issue: there’s too much data. Even automation and reliance on industry-specific solutions can only go so far.”
He says companies should be looking to formally trained statistical analysis of the output from automated intelligent systems built into the back-end infrastructure.
“But analysts with the proper qualifications are hard to come by – and expensive. Also, no-one quite knows yet where to pay the complexity costs: do you make sure all your sources of information are clean first?
“Do you do it at the extract-transform-load (ETL) stage? Or do you leave it unstructured and introduce complexity in the backend to make sense of it? And of course different industries will have different answers to these questions.”
The challenge needs an entirely new approach he thinks and it must be company-wide and driven from the top down.
“At Altech ISIS we have found that executive buy-in of a big data strategy is essential. Changing company culture – which is what it comes down to – is useless to attempt without the backing of at least one C-level executive. Secondly, companies need to see how their data is siloed. If there’s too much concentration of ownership, then it needs to be flattened as much as possible.
“Not everyone can – or should – have a statistics degree in the company, but everyone should be able to back up their day-to-day decisions with data. That means promoting a data culture where everyone has access to what they need and can trust the results. And finally, everyone should be encouraged to look at outside data for insight: public records, market research or even sensor information.”