Infographic: Data Warehouse Modernization – Why it is important and how it can benefit organizations
Modernizing the Data Warehouse: Challenges and Benefits
Data warehousing is getting on in years. Concepts and architectures have been applied more or less unchanged since the 1990s. However, data warehousing and BI applications are only considered moderately successful. Users criticize their inadequate benefits, quality, efficiency and effectiveness.
Especially in times of rapidly changing markets, decision-support systems should promote the quickest possible knowledge growth. Advanced analytics and new ways of working with data also create new requirements that surpass the traditional concepts.
Many companies are therefore forced to put these concepts to the test. But what are the right measures to make the data warehouse and BI fit for the future? What role do technology and IT infrastructure play?
Can the basic nature of the data be proactively improved? What challenges must be expected?
The following insights came from a global BARC survey into the current status of data warehouse modernization. Participants came from companies of different sizes, industries and locations around the world, with 77 percent based in Europe. People from BI and analytics teams, business units, IT, corporate management and external consultant teams took part.
A time-consuming development process and restricted support of self-service BI are the major drivers for modernizing the data warehouse
The most pressing issue for 44 percent of survey participants is a lack of agility in the data warehouse development process. It is noteworthy that business users in particular consider the inability to provide required data and the lack of user acceptance as even more important than enhanced self-service. In particular executives (31 percent) and business intelligence/analytics teams (30 percent) agree that software licenses are too expensive in general. Interestingly, neither poor compliance capabilities nor insufficient technology or functionality are seen as major challenges.
The primary modernization approach is data warehouse/ETL automation, which helps promote broad usage of the data warehouse but can only partially improve efficiency in data management processes
Data warehouse/ETL automation as the most common approach corresponds with the most commonly deployed technologies, which are data integration and data warehouse automation tools. Benefits include broad data warehouse usage and the ability to handle a larger variety of data to support enhanced analytic requirements.
However, an automation approach alone is of limited usefulness when data management processes are inefficient. The data landscape and the data integration tasks to be solved are often too complex.
Leaders rely less on data mart deployment than on lean, flexible architectures and usable data based on cloud services, a complementary data lake, data governance, data hubs and data catalogs
Leaders choose different approaches and technologies from those of followers and laggards. Leaders obviously strive towards a leaner architecture and a flexible infrastructure. They focus less on data marts, tend to extend the data warehouse with a data lake and utilize analytic databases and real-time processing more frequently.
They are opting for cloud data services more frequently. While there is agreement that documentation promotes broader and more efficient use of the data warehouse, the wider adoption of data catalogs by leaders seems to indicate a missed opportunity.
The evolution towards agile, interoperable data services is driving improvements in the ability to meet new requirements
As companies evolve from an integrated data warehouse infrastructure to an enhanced, service-oriented data architecture, they learn to adapt to new and changing requirements. Companies that have embarked on this journey are using lesser-known advanced approaches and technologies. For example, varied data types are handled with new data modeling approaches supported by appropriate tools. Data virtualization and data hubs make it possible to access data sources faster and more flexibly.
Investing in Hadoop, Spark and NoSQL technologies pays for itself by improving the ability to handle a good variety of data with microservices, API management and containerization, improving deployment flexibility.
The transformation from a complex and inefficient data landscape with inadequate data governance to a holistic and agile data architecture requires clear management support and the development of specific data skills
The top four challenges companies face in modernizing their data warehouse environment are primarily related to organization: processes are not agile enough, there is a lack of skills in the business and IT areas and weak data governance results in growing complexity. However, changes in these areas cannot really be accomplished without management support, which is also lacking. Data must become a C-level priority. We must invest consistently in data and analytics literacy, but this also entails a cultural change.
The most important trends aim at more efficient, effective and agile data management based on data governance, automation and self-service
Empowering an organization to be data-driven requires efficient, effective and agile data and analytics processes. This requires usable, quality-assured data. Recurring tasks must be automated, and this applies to data provisioning as well as reporting and analysis. Creative work with data must be supported with user-friendly self-service, which includes providing a data catalog to make data easy to find and use. Machine learning can help to automate manual tasks.
Infographic of the key findings
You can access the full study here.
