Management Summary “Data Black Holes” Survey

The existence of data silos is nothing new. Data-producing applications were once isolated systems. They were built to at least partially automate a specific subtask of a business process. The transactional data was stored in isolated data sets and initially served only one purpose, namely, to document the transaction that had taken place.Over time, enterprises realized that data is worth more. Utilization of operational data for enterprise management helps to gain insights into the current state of the business and supports fact-based decision-making. This has been important for decades. However, the operational data stored in data silos was not suitable for this task. Many companies therefore built a data warehouse to consolidate their operational data silos.Which data architecture is right for the data-driven enterprise remains a subject of ongoing debate.In the age of digitalization, more extensive data and analytics requirements have emerged for which the data warehouse was not sufficiently designed. Data-based insights are being used to automate decisions. The goal is to make business processes faster, more efficient and less vulnerable to risk. Analytics-driven insights are also expected to drive business innovation. Thus, alternative data architecture concepts have emerged, such as the data lake and the data lakehouse. Which data architecture is right for the data-driven enterprise remains a subject of ongoing debate.This study evaluates current thinking around data silos and addresses several questions:What are the implications of data silos for the data-driven enterprise?What are the main challenges companies are facing due to data silos?Which approaches are being adopted to break down the barriers of these data and knowledge silos?Participants around the world were polled, with most responses coming from Europe. We examined the current approaches of companies of different sizes from various industries. For deeper insight, we also analyzed the answers according to data management maturity. Participants were asked to rate the skills and competencies in the handling of data in their company compared to their main competitors. This allowed us to gain a better understanding of what “best-in-class” companies are doing to overcome their data silos in comparison to “laggards” (see demographics).We hope this survey contributes to your company’s efforts to overcome your existing data black holes. Please do not hesitate to contact us if you have any questions.1. Data black holes: the high cost of supposed flexibilityData warehouse, data lake, data lakehouse: the main paradigm for provisioning data for analytics is a centralized approach. However, business users often resort to individual data preparation because centralized data sets do not meet their needs, they trust their own data sets more, or it enables them to get results faster. Consequently, despite all attempts at centralized data provisioning, most companies are unable to reduce the number of data silos they have. On the contrary, their proliferation continues to increase. As the number of silos continuously grows, finding and understanding the right data and preparing it for a specific purpose becomes even more complex and time-consuming.RecommendationDecentralized data preparation is both an opportunity and a risk. Avoid an uncontrolled proliferation of data silos. Create transparency about decentralized data preparation processes and where they negatively impact business efficiency and effectiveness. Identify how you can optimize central data provisioning processes to increase the usability of the data provided.2. Data silos prevent digital transformationHaving key personas tied up with elementary data problems instead of working on the digital future of the company is the biggest challenge caused by the state of current data landscapes. This is a vicious circle. Their capacity is tied up in maintaining existing data problems instead of solving them. In turn, they do not have the time to work on data-driven evolutions, both for themselves and to support others. This limits business agility and velocity as well as the innovation of the data-driven enterprise.RecommendationEstablish business responsibility for data and its strategic development. Make sharing knowledge about data and its use in the company mandatory and create corresponding processes. In the digital enterprise, distributed data sets are not only a matter of course, but a necessity. Therefore, logical, cross-functionalaccountability for data is indispensable.3. Architecture and technology help balance centralized and decentralized data requirementsPredominantly centralized data architecture is not the silver bullet of the digital future. Physical data silos will neither be avoided nor completely dissolved. Therefore, an exclusively centralized approach to data integration is doomed to fail. Instead, distributed data must be linkable as needed. Architecture and technology play a significant role here. They help overcome the limitations of physically distributed data sets more efficiently by supporting intuitive and on-demand exploration, preparation and (advanced) analysis of data.RecommendationPlan a logical architecture that ensures combinability of physically distributed data. Populate this architecture with smart technologies such as data cataloging, data virtualization and AI-augmented data preparation that help overcome the boundaries of physical data sets.4. Organizational silos weigh heavier than data silos – overcoming them is a cultural journeyUnified business terminology, not centralized data management, is the main success factor for the digital enterprise. Sharing data, analyses and data-based insights across business units only delivers value if they are understandable to others. However, this requires a uniform (data) language. But because business units have been managed as silos for many years, these language barriers are primarily organizational. Data silos are often just a consequence of this. Accordingly, a lack of communication is the most encountered cultural challenge when it comes to improving the data landscape.RecommendationWhen it comes to ending the data silo dilemma, change requires bold actions. Shaping data strategy and goals and transforming the behavior of people in the company will not succeed without executive vision and leadership. Secure the support of your company management for your data strategy and convince them that they must actively lead the digital transformation. Employees need guidance based on clear strategic goals and what becoming a data-driven business means in practice.