Companies have spent millions pursuing the hype and promise of business intelligence (BI), but what exactly does BI promise to deliver and why is it prone to failure? asks Bryn Davies, MD of InfoBlueprint.
A recent survey of 2 000 BI users and consultants by the Business Application Research Centre (BARC) says: “We regularly find that the received wisdom in this industry doesn't match what the majority of real-world users are finding.” Clearly, something is amiss.
BI has become the be-all and end-all of anything and everything to do with the effective delivery of information. And that’s the problem: the focus has been too much on effective delivery, with scant regard for the complexities surrounding the goods being delivered – data.
A typical BI project starts with understanding what information must be displayed, how and to whom, to enable effective analysis and decision making. But an obstacle instantly arises: Most large organisations have many disparate applications spread across the enterprise, which have evolved independently to suit that area of the business. That means a huge diversity of underlying data: Different formats, different codes and varying degrees of detail and quality.
As a result implementing BI usually requires performing some serious data gymnastics within data warehouse structures and in Extract Transform Load (ETL) processes. One of the major functions of most ETL processes is data scrubbing: Making sure, for example, that the description of the country’s provinces is consistently represented, as opposed to “Northern Cape” in one application, “N. Cape” in another, and “Noord Kaap” in a third. Even this simple example highlights a fundamental problem: data has rarely been subject to sound and consistent governance.
Data scrubbing is a difficult and delicate process, and it never gets any easier. Quite the reverse, in fact. Every time an underlying database is changed, the data scrubbing process needs to be changed as well. More importantly, the rate of data creation and collection in organisations is growing all the time. Combined with the growing demand for real-time BI, data scrubbing will increasingly become a bottleneck in the delivery of trusted information.
What is the solution? Clearly, the less work that has to be done during the flow of data through a BI system, the more easily the system itself will be able to respond to business changes and growing data volumes. At heart data scrubbing is reactive, always tackling the symptoms rather than the underlying causes of the disease. To get proactive, companies need to focus their attention squarely on the data itself.
But even data is not the ultimate culprit; rather it is the product of business processes and subject to day to day management – or not. Here is the crux of the matter: We have failed to treat data as the useful and valuable asset it is. We have given little regard to how it is governed, who owns it, who is responsible for its quality, how it is made up, where it comes from and where it goes, and what it actually all means. Indeed, the simplest of terms, such as “customer”, can have multiple perceived meanings across functional boundaries. Try and get a data scrubbing routine to deal with that.
It is time to treat our data as we do our other valuable assets. That means inventorying it, agreeing on its meaning (use the term “metadata” if you want), organizing it (building and harmonizing corporate data models) and looking after its quality.
BI has come a long way – but it still bears a great cost in the daily heavy lifting we have to do, using expensive people and technology, to “fix” the data problems that never seem to go away. Data quality has long been recognised as a threat to BI because bad data can cause defective or sub-optimal decision-making.
However, questionable data also creates significant ongoing effort and hidden costs in the BI factory, is an unnecessary risk for BI projects, decreases ROI and is an unwelcome threat to business success.
All the sophisticated BI technology we have bought to pound our data into shape and make it look pretty is just a delivery vehicle. Let’s start focusing on the very essence that BI delivers – data – and we will finally be able to claim that BI has indeed fulfilled its promise.