Advanced and Predictive Analytics: An Introduction

“More than just reporting” – this is the mantra for many BI executives nowadays. Advanced and predictive analytics in particular offer huge potential in terms of return on investment in BI systems and data inventory.

What is Advanced and Predictive Analytics?

Advanced analytics describes data analysis that goes beyond simple mathematical calculations such as sums and averages, or filtering and sorting. Advanced analyses use mathematical and statistical formulas and algorithms to generate new information, to recognize patterns, and also to predict outcomes and their respective probabilities.

Predictive analytics is a sub-division of advanced analytics and focuses on the identification of future events and values with their respective probabilities.

Advanced and predictive analytics can be applied to a variety of use cases, ranging from classic customer value and success forecasts, the prevention of contract termination or price, sales and demand forecasts to new tasks such as the prediction of machine failures, social media monitoring and evaluation, and predictive policing.

Advanced analytics enables both “optimization” and “innovation”. It can support the improvement of existing processes, for example, in the form of more precise sales planning (and therefore production and purchasing planning). Also, new insights from advanced analyses can highlight potential new business or even make new products and services possible.

The following tasks can be addressed using advanced analytics:

• Segmentation: creation of groups based on similarities between objects;
• Association: identification of the frequency of joint occurrences and, if necessary, the derivation of rules such as “From A and B (usually) follows C”;
• Classification: for example, of hitherto unclassified elements;
• Correlation analysis: identification of relationships between element properties;
• Forecast: derivation of future values.

Why Advanced and Predictive Analytics Is Becoming So Important

The current age of information and digitalization has brought with it new technologies and methods for improving business operations and maintaining competitive advantage:

• New big data technologies enable cost-effective storage, processing and analysis of large amounts of data;
• Modern and intuitive user interfaces allow more user groups to draw insights and make informed decisions; and
• Advanced analytics software enables better analysis, and analysis of relationships and future events.

Since modern techniques and technologies to accelerate or otherwise improve decisions or processes along the value chain are now widely available, it is important to carefully evaluate how advanced analytics can be used within your company in order to keep pace with the competition.

How Important Is Advanced and Predictive Analytics for BI Professionals?

Generally, most companies see advanced and predictive analytics as one of the more important BI trends in 2017. However, there are a few differences in viewpoint across various user and company types.

Best-in-class companies and organizations in North and South America lead the way when it comes to predictive and advanced analytics.

On the other hand, the trend is much less important in telecommunications companies and the German-speaking region of Central Europe.

Importance of Advanced and Predictive Analytics in 2017 (n=2,661)

Overall, the data shows that the trend for advanced and predictive analytics is on the rise. This is especially true in best-in-class, Eastern European companies and organizations from the United Kingdom and Ireland, as well as the telecommunications industry.

Surprisingly, advanced and predictive analytics is seen as less important by companies from the telecommunications industry this year.

Challenges

An important characteristic of advanced analytics projects is the comparatively high risk of failure.

Traditional BI reports only show data. If the data is correct, then reports are highly likely to be reliable as well, as most modern BI environments are now quite mature and their reporting methods and concepts have reached a high level of sophistication.

However, there is less guarantee that advanced analysis will deliver the results expected.

Today, a large number of standard algorithms and methods are available for specific use cases (e.g., customer classification), and new ones are constantly being developed. Finding the most appropriate one for a dataset depends largely on the abilities of the user and the software used.

Furthermore, algorithms can also fail due to lack of data (e.g., the customer classification model). If an advanced analysis project shows that no results can be found, it should be aborted and the next project started.

Furthermore, users of advanced analytics methods should have expertise in working with probabilities. While classic BI reports almost always produce the correct numbers, the probabilities generated by advanced analyses have to be interpreted.

The quality of a sales forecast or customer classification must therefore not only be noted and communicated for every single analysis, but also continuously monitored and optimized.

Technologies and Tools for Advanced and Predictive Analytics

Companies can choose from a variety of software tools to perform advanced analytics.

In principle, there are four distinct classes of software:

• Use-case-specific solutions that offer easy access to sometimes complex statistical and mathematical analysis methods;
• Business intelligence solutions extended by advanced analytics functions;
• Advanced analytics platforms and traditional data mining software;
• Proprietary developments (e.g., using the extensive and license-free “R libraries”)

In addition to the algorithms and procedures required for the application, use-case-specific solutions also offer business user-friendly interfaces. These conceal complicated models and their corresponding calculations from the user.

Using pre-defined logic in the form of task-specific modules, software providers are making their systems as user-friendly as possible. For example, business departments needing to carry out simple, regular analyses such as shopping cart evaluations, customer classifications or price optimizations have a range of tools with predefined templates, content and corresponding user interfaces to choose from.

However, when business requirements become more complex (e.g., there are not three, but thirty-three characteristic values to be included in a data mining model), the picture becomes altogether different.

Also, if the model has to be adapted (e.g., new variables are generated), the limitations of use-case-specific solutions become apparent. The models must then be further developed, which is no trivial task, by experts with the necessary skills. Functional and easy-to-use “advanced analytics platforms” support this process of finding the right data mining model.

Products historically based mainly on business intelligence (i.e. visualizing and evaluating data using dashboards, reports or analyses), are increasingly being equipped with advanced analysis functions to visualize not only the current state of affairs, but also to calculate complex statistical models or predictions. The power of these functions varies considerably. Most of these BI products have a form editor in their reporting and analysis tools.

This allows the user to store simple mathematical calculations in reports or analyses. As well as determining the minimum or maximum value within an analysis, the report designer can, for example, also store a calculation which uses exponential smoothing to determine a forecast value such as, for example, sales in the coming months.

However, business intelligence software has limitations in more complex predictive analytics tasks: modeling and implementation of more advanced application-specific tasks, neural networks, decision trees and text mining usually require external software (e.g., advanced analytics platforms).

BARC Recommendations

• Advanced analytics brings business benefits. Try it! It is a good idea to at least evaluate advanced analytics, and this does not have to involve a lot of effort. Existing business intelligence or CRM tools often include advanced functions for (sales) forecasts or classifications. Open source data mining tools are freely available and can be used in conjunction with sample models for short-term experiments.
• Companies can draw from an extensive pool of technologies, procedural models, techniques, use cases and organizational concepts to implement advanced analytics. However, the variety and sheer quantity of tools available make it difficult to start quickly and purposefully, so it is important to address some basic, fundamental questions when getting started: Which tools could be used to perform our advanced analytics tasks? What’s the best software approach for us? Use-case-specific solutions, modern BI tools, generic advanced analytics platforms, or a custom development? Which employees/roles should be used/trained? How important should advanced analytics become to the company? How would the advanced analytics team fit into your organization in terms of the way it works together with the BI team (e.g., BICC), business departments and IT? Which use cases can bring added value to the company and be implemented with little effort?
• Companies at the very beginning of their advanced analytics journey should take the opportunity to learn from others: attend lectures, ask questions at trade fairs, learn on the job by running your first advanced/predictive analytics projects with a joint team of internal staff and external data scientists.
• Failure is acceptable in advanced analytics – it just means you have to rethink! Compared to typical IT projects, advanced analysis requires a more exploratory approach. As in a laboratory, a wide variety of hypotheses can be tested and many of those tests will yield unsatisfactory results. Indeed, it is possible that for some ideas either no model can be found, the effort to determine the model exceeds its value, or the necessary data is simply not available. Approaching a project with a defined timescale, budget and result is therefore not always effective.
• Data is the foundation and fuel of advanced and predictive analytics projects. Make sure there’s a sufficient quantity, quality and availability of data. BARC surveys show that (often expensive) staff employed to deal with the search and evaluation of a suitable advanced analytics model are often held back by poor data quality and unattached or poorly documented data sources. For this reason, companies often value data preparation and processing competence more than statistical/mathematical understanding. To make it easier for users to work, IT managers should push for better equipment. Many employees benefit from good data quality and easy-to-use, well documented data in their day-to-day operational work as well as in decision-making processes. Evaluate modern concepts such as visual data integration solutions, metadata management tools, data virtualization and business glossaries to ensure transparency and traceability. Check to see if Hadoop technology is worth the effort to keep large amounts of data at a low cost and at the same time make it accessible to users.
• Learn to document and communicate complicated facts in a comprehensible manner. Companies often have problems communicating content to their report recipients in simple BI reports (e.g., pie charts distorting the quantities they’re displaying; missing legends; or ill-defined target, actual or plan values). Findings from advanced analytics are typically even more difficult to communicate as data is often presented in at least three new dimensions (e.g., forecast values, probabilities, possible models). Use visualization standards as developed, for example, by Rolf Hichert or Edward Tufte. And create a special role for someone with skills in visualizing and communicating information and analysis results throughout the company (a “Data Artist”).
• Optimize your advanced analytics organization, tools and processes step by step. Advanced analytics lives by its dynamism and flexibility. It requires employees from business departments, IT, statisticians and project coordinators to work together in virtual or physical teams. Prepare yourself to assess various technologies such as in-memory databases and processing, Hadoop, NoSQL, self-service data integration, and data virtualization. The agility required here can make cloud computing an attractive option. Conduct structured vendor evaluations or proof-of-concepts with established software providers and promising start-ups. If you recognize the importance of advanced analytics in your company, a strong interweaving and interaction between different departments is inevitable. The various business units should then be able to work together with their data, software, stakeholders and processes in the most integrated environment possible to reduce information losses and achieve results quickly.

Ongoing examination and comparison of one’s own technology, strategy and organization with current concepts and methods will enable a future-oriented positioning of your company in the market.