Back in the dark ages – say 1985 or so – processors were slow and expensive, and so was memory. Remember when 1Mb of RAM was a big deal? In those days computer programming was largely a matter of writing around hardware constraints. Relational databases were optimised for data capture and storage, which made efficient use of the available hardware but gave little credit to the needs of those who actually had to get the data out again in a usable format.
When it came to business intelligence and the analysis of large quantities of information, these hardware and database design constraints led programmers to an approach that involved pre-calculating as much as possible and storing the results in data structures or “cubes” that could then be queried.
This “online analytical processing” or OLAP got the job done, but at a high cost. The OLAP cubes were difficult and time-consuming to build, and users couldn’t explore their data outside of the structure imposed by the cubes. If the report you were looking at led to a question that required cutting the data another way, too bad. You could choose to live without the answer, or spend long hours wrestling with spreadsheets to get something you could live with.
To make matters worse, cubes need to be re-populated and re-calculated daily, a process which can take several hours each time. All of this ties up scarce IT resources that could be better used somewhere else.
Fortunately, memory and processing power have both come a long way since then. Moore’s Law – roughly put, that processing power doubles every two years – has held true, and contemporary programmers have access to vastly greater processing resources than they did 20 years ago. At the same time the cost of memory has plummeted: the price of a Gigabyte of RAM dropped from $1 000.00 in 2000 to $100.00 in 2005 – and should drop to less than $10.00 by 2010.
All that relatively cheap processing power and memory, with the help of modern compression algorithms, makes for a radically different approach to business intelligence. Instead of pre-calculating OLAP cubes, it’s now possible to load an entire database into memory and perform live calculations, giving users unprecedented speed and flexibility. A patented technology called Associative Query Logic builds connections and associations between separate bits of data the same way the human mind does, allowing for the first time the possibility of a truly intuitive relationship to large bodies of structured data.
Let’s say you’re reviewing your sales figures for this month. In-memory processing means it’s possible, with just a single click, to compare those figures to last month’s, the same month last year and the year before, or to get a graph for the past three years. You notice that one rep’s sales have decreased dramatically, and another single click gets you a list of just that rep’s accounts. A quick glance shows that one client placed no orders this month, and another single click gets you that client’s entire order history – right down to the level of the individual invoice.
Without in-memory processing, it could take you weeks to discover that you’d lost a major client. With in-memory processing, you can review each day’s sales figures the next day and pick up problems immediately. With the client’s entire history in front of you, it becomes much easier to investigate the problems, and to fix them.
This is the future of business intelligence – in fact, of any work that involves analysing and understanding large bodies of data. With the memory and processing power that enable in-memory BI now available, it seems mad to go to the trouble and expense of doing all the pre-processing required by older OLAP technology – especially when it yields results that are far less flexible.
The good news in this for business users is that they get far better access to information they need to do their jobs effectively. The good news for IT departments is that writing reports, possibly one of the most boring jobs in the fields, will soon be a thing of the past – which frees talented people up to do more exciting things.
OLAP-based BI was a great solution to the constraints we faced in 1985, but that was a long time ago. In-memory BI is on the right side of Moore’s law and will continue to grow in power and flexibility as 64-bit computing becomes commonplace and memory prices continue to fall.