Large organisations in the US and Europe are already achieving significant value from their investment in data streaming.
This is one finding from a new research report by Conduktor that questioned 200 senior IT and data executives at large companies with annual revenues of $50-million or more. It found the top three areas in which respondents say they have derived the most benefits from using data streaming are:
- Enhancing customer experiences with faster, personalized services;
- Improving real-time decision making and responsiveness; and
- Detecting and preventing fraud or security threats.
Other areas in which organisations are seeing value from data streaming include optimizing operations (such as supply chain or logistics) through faster data insights, automating business processes and workflows, and reducing operational costs through faster issue detection and resolution.
Despite those benefits, the majority (86%) of respondents struggle with integration across different data streaming platforms. An even higher proportion of respondents (98%) said that they are concerned about losing access to valuable data when consolidating streaming platforms.
Conduktor advises that the best way to avoid losing access to valuable data when operating or consolidating data streaming platforms is to involve these three teams from the very beginning:
- The governance team, to ensure that lineage tracking and metadata are moved in terms of ownership, data contracts, monitoring, and documentation.
- The platform team, to build bridges including proxies, dual writes, and shadow consumers during the transition to maintain continuity.
- The security team, to manage Identity and Access Management (IAM), Role-based Access Control (RBAC) and Attribute-based Access Control (ABAC) policy migration, plus firewalls and approvals.
Nicolas Orban, CEO of Conduktor, says: “Organisations risk data loss because of connectivity gaps caused by networking or security constraints; changing schemas and data quality rules; missing metadata and context from migrations; and a lack of trust in the quality, performance, or security of the new platform.
“Without a common control plane, the spread of streaming platforms can quickly create chaos.”
According to Dataintelo, the global market size for streaming data processing system software was valued at approximately $9,5-billion in 2023 and is projected to reach around $23,8-billion by 2032, reflecting a compound annual growth rate (CAGR) of 10.8% over the forecast period.
Dataintelo says: “The surge in the need for real-time data processing capabilities, driven by the exponential growth of data from various sources such as social media, IoT devices, and enterprise data systems, is a significant growth factor for this market.”