Infographic: The Future of Data Architecture

Has the Data Warehouse Had its Day?

Data architecture is a topic that is as relevant today as ever. It is widely regarded as a matter for data engineers, not business domain experts. But is it really? Statements from countless interviews with our customers reveal that the data warehouse is seen as a “black box” by many and understood by few business users. Therefore, it is not clear why the costly and apparently flexibility-inhibiting data warehouse is needed at all. We even know companies that are cutting back their investments in a “single point of truth” for these reasons.

Participants around the world were surveyed, with most responses coming from Europe. We examined the current architecture approaches of companies of different sizes from various industries. Participants were asked to rate the skills and competencies in the handling of data in their company compared to their main competitors. This allowed us to gain an understanding of how “best-in-class” companies are shaping their data architecture in comparison to “laggards”.

In general, central data & analytics teams determine the data architecture for analytical data, decoupled from the landscape of operational data sources. Our survey shows that they continue to pursue clearly centralized architecture concepts. Indeed, this is what the data warehouse, data lake and data lakehouse have in common, regardless of the differences in their detail. However, is this best practice, which has basically lasted for three decades, still suitable for meeting the challenges of the increasingly distributed data landscapes of today’s digital enterprises?

Our survey results show that the opinions of different roles in the organization differ considerably in many cases. It is primarily the data & analytics experts in the central teams who adhere to the central approach. They see it as proven and successful.

Among executives and managers, as well as process and data & analytics experts in business units, the picture is often different. The question arises: Should future data architecture be determined more centrally from the perspective of data engineers or more decentrally from the perspective of domain experts? Or is a federated approach the best way forward?

We hope that our survey results will provide you with valuable insights and help you find the right solution for the future design of your company’s data landscape. We welcome your feedback at any time.

Global survey

This study was based on the findings of a worldwide online survey conducted in March and April 2022. The survey was promoted within the BARC panel, as well as via websites and newsletter distribution lists. A total of 268 people took part, representing a variety of different roles, industries and company sizes.

Key takeaways

• Data & analytics users are surprisingly patientAt first glance, users appear to be quite satisfied with their existing data landscape. But in fact, line-of-business managers and team members tend to be more critical about the future viability of their data landscape than the data & analytics experts in central teams. They are frequently dissatisfied with flexibility and extensibility and also criticize the lack of comprehensibility. Both executives and business users say that existing data is hard to find. All groups agree that existing analyses are not easy for business consumers to understand.
• Business user empowerment must finally gain momentumProviding self-service analytics tools to the business has picked up speed. Only a minority finds that the tools available to them are too technical. The limiting factor is rather the data landscape. Because data warehouses frequently do not meet business needs, data for analysis must be tediously gathered from various sources. Nevertheless, initiatives in the direction of a data catalog including suitable data documentation and self-service data preparation are not widespread.
• Business domain experts must take responsibility for data productsCreating useful and understandable data & analytics products that deliver business value requires business domain expertise. While central data & analytics teams have very strong technology and data engineering knowledge, their business domain expertise is not so strong. This is why there is an urgent need for business domain experts to take responsibility for data products. This should not be limited to analytical data products but must include source-system-oriented data products that provide the original source data as well.
• Centralized data architecture concepts have served their timeMost companies continue to apply tried-and-tested concepts such as the data warehouse design paradigm when redesigning their data landscape. Best-in-class companies favor the data lakehouse approach. However, executives and business users in particular criticize the fact that centralized approaches cannot prevent the emergence of further data silos. But there are also signs of a change in thinking. Companies are seeking to improve their source data to streamline their data pipelines and to build a better basis for data virtualization.
• Evolving a data culture is about people, not just buying technologyCompanies are aware that organizational measures are needed above all to further develop their data culture. But in practice, only a minority is implementing such measures. Initiatives that support self-service analytics are the most widespread. This is good, but not enough. Strengthening cross-domain data collaboration requires both harmonized master and reference data, and an understandable and shared view of data regardless of system silos. This can only be achieved if business process owners and experts, the data producers, are actively involved.

Five recommendations

• The most important measure in favor of better data utilization is to create transparency. The data landscape of most companies is complex. Business users are already confronted with this issue today, despite the data warehouse. This will not change with a new platform or a new architecture. Help is required to navigate this complexity. At the same time, central data & analytics teams must gain more insight into the concerns and needs of business users.
• Best-in-class companies realize that it is important to invest in enabling business users to work with data as and when required. For selected data experts in the business, this will include data preparation. They already do this, but with inappropriate support, for example, in spreadsheets and without the right data documentation. Central data & analytics teams must embrace this reality and shape it more consciously in the future.
• Investments in data must deliver business value. Business domain expertise is required to assess where this is appropriate and how it should look. Plan for a gradual transformation, because it goes hand in hand with a cultural change and learning process. Central data teams become advisors to the business, provide support with their technology and data engineering expertise and consult on central data governance issues.
• As the quantity and volume of distributed data sources increase, data architecture concepts that are exclusively centrally oriented quickly reach their limits. Therefore, design your future data architecture with extended capabilities. Work towards a federated overall architecture in which data virtualization and distributed data pipelining and orchestration are options and create the framework for this.
• Establishing a data culture is a long-term process. It requires a change in mindset and a willingness to look outside the box. You need to bring data producers and consumers together to enable the exchange of perspectives and objectives. Technology such as a data catalog can effectively support data collaboration. But it is ultimately the business users who, through their contributions, bring the technology to life. It takes cross-domain collaboration to find the right measures to improve the data landscape.

Infographic of the key findings

You can access the full study here.

The Data Management Survey

Nutzen Sie Software für Data Intelligence, Data Catalogs & Marketplaces, Cloud Data Platforms, Data Observability & Quality, Data Warehouse Automation oder Data Product Engineering? Dann würden wir gerne Ihre Erfahrungen dazu hören!

Nehmen Sie jetzt an der weltweit größten Umfrage zum Thema Datenmanagement teil:

• Sie erhalten eine Zusammenfassung der Umfrageergebnisse
• Sie stellen sicher, dass Ihre Erfahrungen in die Auswertung einfließen
• Sie nehmen an der Verlosung eines von zehn Amazon-Gutscheinen im Wert von 50 € teil.

Jetzt teilnehmen!