A data lake is an enterprise resource that gives business units, functions and departments unprecedented freedom and flexibility to gather, analyse, and use the data they most need.
In essence, business data lakes hold the keys to meeting the fast-growing business appetite for new combinations of data and to putting big data analytics to work across the enterprise. Servaas Venter, Country Manager of EMC Southern Africa, believes data lakes will drive big changes.
“You can look at business data lakes in three ways: as one place to put all the data you may want to use; as a platform for big data analytics; and to help resolve the long-standing tension between the corporate push to get standard data into warehouses and used consistently, and the business unit need for local views and combinations of data that get implemented in all those Excel spreadsheets,” he says.
“Organisations and individuals have been generating enormous amounts of data for a long time. Only lately have we had technologies and methods for dealing with it with relative ease. Data lakes help put big data to work for the enterprise.”
Applications of data lakes include those that need to analyse vast amounts of newly-generated or combined data, for example, genomic analysis or predictive models for when and where power grids will fail.
Venter points out the opportunities around customer insight and experience. “You can pull together everything you know about your customers and everything they tell you – customer profiles, purchase history, sales and call centre interactions, the social media data where customers are speaking for themselves,” he says. “Analyse all that data together, and you can design and deliver a more compelling experience – even shape the experience in realtime.”
CIOs will be particularly interested in information systems security applications. Controls like firewalls and authentication are insufficient to protect an enterprise against all of today’s threats, external or internal. CIOs need to be able to notice and analyse the behaviours of people or programs that have or appear to have valid credentials.
“An organisation that puts all its system logs and network activity into a data lake can get better and faster at spotting anomalies, which leads to a faster, more targeted response. IT management can then close the loop by using the intelligence generated from the data lake to build predictive models of when and where problems are most likely to occur,” Venter says.
Organisations opting to implement data lakes need to recognise that data lakes will become at least an order of magnitude bigger than the largest corporate data repositories today. Fortunately, data lakes have a far more favourable cost structure than conventional databases, where it can be cost prohibitive, not to mention technologically cumbersome, to do big data analytics.
“For business leaders, the biggest challenge may be deciding what to do with data lakes because there are so many opportunities. The constraints around how much data an organisation can work with have effectively been removed. That opens up endless possibilities for doing new things, doing old things better, and doing things extremely fast. We’re limited by our imaginations, not the technology,” Venter says.
He recommends that CIOs recognise that data lakes are not on the horizon – they’re here today. The technology integration of flexible, scalable data storage with big data analytics is complex, so investing in data lakes as a platform as a service solution is wise.
“The purpose of data lakes is to enable business people and organisations to work with far more data of interest, do better and faster analytics, decide and act in real time, and generate greater insight and value. To the business, the data lake is a service, and success is measured in how the service is consumed and converted into other forms of business value,” Venter says.
“Finally, you can be ambitious. Most enterprises are just scratching the surface of what they can do with big data and analytics. The constraints on data use really have been removed. So work with your business executive partners, pick an interesting opportunity or two, be creative, and exceed your ambitions.”