Organisations are under pressure to achieve a faster time-to-value for implementing analytics projects, and this is driving migration to the cloud.

This is among the findings of an IDG Research MarketPulse survey, “Optimising Business Analytics by Transforming Data in the Cloud” released by Matillion and IDG Research, exposing the challenges companies face and the strategies they use to prepare data for BI and analytics.

The top takeaways of the research include:

Enterprises struggle to use data volumes for actionable insights.

* On average, data volumes are growing at a rate of 63% a month;

* 12% report that their data volumes are growing at 100% or more per month;

* More than 20% of those surveyed report drawing from 1 000 or more data sources; and

* Responsibility for exploiting data is divided between IT groups (55%) and business units (45%).

Enterprises are tapping the cloud to simplify and scale data management.

* While less than one-quarter (23%) of respondents have completely centralized their BI and analytics teams, virtually all plan to tap the cloud for data management, migrate or continue to migrate data to the cloud over the next 24 months;

* 90% have already placed some data in CDWs;

* 37% of organisational data is in CDWs, 35% is in on-premises data warehouses, and 25% is in offsite, non-CDWs;

* Three CDWs dominate: Amazon Redshift (used by 54% of the respondents), Google BigQuery (50%), and Snowflake (26%); and

* The top reason enterprises migrate their data to cloud platforms is faster time-to-value for implementing analytics projects.

Despite broad and growing use of CDWs, they aren’t a panacea; CDW adoption alone does not address all data analytics needs.

* More than 90% said it is challenging to make data available in a format usable for analytics; and

* Respondents cited several obstacles slowing their data analytics projects, including a lack of necessary data granularity; manual coding of data pipelines; and difficulty connecting with multiple data sources.

Part of the challenge is in how enterprises are using legacy ETL processes in a modern data environment.

* Many struggle with the sequence and process of data transformation; more than one-third (37%) manually code data into the necessary format before loading it into BI and analytics tools;

* Only 28% load data into the cloud and then use the cloud platform to automate the transformation process; and

* Data portability (45%) and scalability (46%) top the list of perceived benefits of a modern approach to data transformation.

“Without the right data management strategy, growing data volumes go from a competitive advantage to a company-wide struggle to make it useful,” says Matthew Scullion, CEO of Matillion. “Our research underscores the demand and the urgency to use the flexibility and scalability of the cloud – not just to store data, but also to accelerate insights through data transformation.”