One of the biggest challenges that Business Intelligence (BI) analysts face is data debt – the accumulated cost of shortcuts taken with data quality and consistency over the years, which have created long-term problems.
Among these issues are the time and sheer effort it takes to build most BI reports. A report that should, at face value, take a few days to write may actually take weeks or months, because locating and preparing relevant data to support the report can be immensely time consuming.
By Gary Allemann, MD of Master Data Management
Under pressure to meet deadlines, BI developers take their own short cuts, building more data debt. Addressing data debt by acknowledging data management issues, and then taking steps to correct them, will go a long way toward delivering requested BI reports quicker, and with more trust.
Data first, always
Without a change in approach, BI analysts will remain trapped on a hamster wheel of trying to deliver reports for business timeously, without the required support. Data issues need to be addressed systematically, but this is often easier said than done. There is still a disconnect between business and IT, and without taking steps to change this, data will never be the priority it should be.
We expect information to be a business differentiator. Yet, in practice, only 25% of workers feel they use data effectively in their jobs, and only 21% of workers feel confident in their overall data literacy skills. It doesn’t matter how much data your business collects if your staff and decision-makers are not equipped to use it effectively.
At one level, decision makers need to be educated to understand the impact of data debt, and the hidden costs associated with it – for example, the costs incurred in finding and preparing data for analytics. At the same time, we need to invest in data literacy training for operational staff and “data capture” clerks, because if the people responsible for creating and storing the data do not understand the potential inherent problems, then there will never be a change.
BI teams could also consider adopting the DataOps methodology – an agile approach to data that leverages tools to bridge the gap between data scientists, data analysts, engineers and business stakeholders – an important step in activating data to deliver business value.
How do we do things differently?
It is clear that something needs to change. Without business awareness of the issues and hidden costs, nothing will ever change. It is in everyone’s best interest to address the underlying culture that builds data debt. Data that is hard to find, inconsistent and inaccurate not only impacts analytics teams’ ability to effectively deliver insight, but also creates operating inefficiencies that have a broad impact across the business.
The reality is that the vast majority of the effort that goes into creating business reports should be accessible and repeatable, but in most cases, it is not. More than 80% of the data stored in most businesses is known as dark data – data that is not understood at all and therefore inaccessible for analytics. Murky data, which is data that can only be used with considerable input from a subject matter expert, further compounds the problem.
Collaboration is key
Addressing this data challenge is not just about going back and cleaning legacy data, but about ensuring that, going forward, we are not adding to our data debt. Finding ways to collaborate and build capacity going forward, cataloguing past efforts to make things easier in the future, and ensuring that everything is effectively documented, will work to dramatically improve productivity.
Data needs to be easy to find, easy to use and understandable, which means that BI developers need to build context around that data, as they uncover it.
With this context in place, data sets and reports become more accessible to the organisation as a whole, unlocking advanced analytical abilities that allow data stories to be developed. This is the work that adds value and helps businesses develop trust in the data, where it comes from, what it means and that it delivers on what is needed. This is how BI reports can finally become quicker and easier to deliver.