The amount of data being created continues to grow at almost unbelievable rates – Zettabytes and Yottabytes are set to become commonplace – and will soon overtake the ability of companies and individuals to store it.
The amount of data being created and stored continues to grow at unprecedented rates, says EMC's executive vice-president for global marketing and customer quality Frank Hauck.
EMC and IDC this week unveiled results from an annual study into the digital universe, which found that data grow in 2007 exceeded previous predictions by 10%, and will probably outstrip 2008 expectionations by about 20%.
While companies used to talk about Megabytes and Gigabytes of data, today's conversations are more likely to revolve around Terabytes and even Petabytes, he says.
Following soon after these iterations are Exabytes and finally Yottabytes.
EMC already has a handful of customers with storage arrays totalling multiple Petabytes, says Hauck, so even larger data quantities are on the horizon.
He illustrates the relative data quantities, with a single byte represented by a grain of sand: A Megabyte would be a spoonful of sand; a Gigagbyte a block of sand 16 400 cubic centimetres; a Terabyte a sandbox 7,4 square metres and 30cm deep; a Petabyte a beach 68km long, 30,5m wide and 30cm deep; an Exabyte would be the same beach stretching from Swaziland to East London; and Zettabyte would be the same beach stretching along the whole South African coastline; and a Yottabyte would be enough sand to bury South Africa to a depth of 134m.
Hauck explains that the data explosion is driven by both the user footprint – the data a person generates and stores for himself – and the user shadow – information that is generated and stored about him.
New technologies like multimedia and entertainment are helping to fuel the data explosion – and individual consumers are going to be responsible for about 70% of new data.
However, organisations will have the task of storing, securing and managing it.
Companies will soon have to look for a manager show can be to information what the CFO is to money, and will have to start developing an information governance framework, says Hauck.
Companies face other storage challenges as well – security and governance are just two of the most pressing.
Hauck points out that the storage industry has certainly contributed to the data explosion by making plentiful storage available at ever-decreasing costs.
However, the time is rapidly approaching where the amount of data being created will simply outstrip the amount of storage available to keep it.
When this happens, companies and individuals will have to start making hard decisions about what they need to keep, and in what format.
"People are going to have to change the way they store their data," says Hauck.
For instance, a large proportion of the data currently held in online storage could well be archived – it would still be available but on slower and cheaper arrays.
Data management will also become a lot more important, says Hauck, especially to eliminate the vast amount of duplication that currently occurs on most systems.
Different types of storage architectures are also coming on to the market, which will give users more choices of storage types depending upon the speed and price points they are comfortable with.
On the technological front, solid state storage is enjoying a comeback, although it was relegated for many years as old technology.
Even flash drives today have aggressive roadmaps, says Hauck, and will help to contribute to further growth in the storage world.