About The Analytics & AI Survey
The survey was previously published as The BI & Analytics Survey. The new name reflects the AI-focused criteria we have added, as AI functionality becomes an increasingly important part of BI & analytics software.
Based on a sample of over 700 survey responses, The Analytics & AI Survey 27 offers an unsurpassed level of user feedback on 17 leading business intelligence & analytics solutions.
Our research covers issues ranging from the purchase cycle right through to deployment, including critical information on performance levels, scalability and problems encountered.
Respondents
Countries
BI & Analytics Products
Years
Components of The Analytics & AI Survey
Instead of delivering one long document covering all aspects of The Analytics & AI Survey, we have split the information into several smaller documents. By providing the raw data via a web-based tool – The Analytics & AI Survey Analyzer – users have the ability to carry out their own analysis of the survey results. The Analytics & AI Survey 27 is divided into several documents, as listed below.
These documents do not need to be read in sequence. The Results and the Vendor Performance Summaries can be read independently.
Sample & Methodology
The value of a survey like this depends on having a sufficiently large, well-distributed and unbiased sample. This section describes the characteristics of the people who took part in the study.
The Analytics & AI Survey 27 has the largest sample of any survey of business intelligence & analytics users available on the market. The Analytics & AI Survey has a rule that, as far as possible, only sub-samples containing 20 or more data points should be reported.
We apply increasingly stringent data cleansing rules, using a number of different tests. We remove all suspect data that purports to be from user sites.
Most surveys are conducted or sponsored by an organization based in, and focused on, one country. However, business intelligence and analytics is a worldwide market and we wanted, as far as possible, to capture a large international sample. This not only presents a more accurate global picture but also allows international variations to be analyzed.
The largest business intelligence and analytics markets are the United States, Germany, France and the United Kingdom, so The Analytics & AI Survey 27 was produced as a collaboration between organizations in each of these countries, and in partnership with publishers and vendors in these and other countries.
Sample size and make-up
Hundreds of thousands of people around the world were invited to participate in The Analytics & AI Survey 27, using social media as well as dozens of email lists, magazines and websites.
As in previous years, the questionnaire offered different sets of questions for vendors and users (or consultants answering on behalf of users). This seems to produce better quality data as in the past some vendors pretended to be users when they saw they were not being asked relevant questions.
Participants from last year who indicated that they would like to be part of our panel received a pre-filled questionnaire with their answers from last year’s questions. They were asked to update their responses, and then to answer the new questions in this year’s survey.
The results of the online data collected are shown in the table, with the numbers of responses removed also displayed.
The number of responses is split between users, consultants and vendors. Vendors answered a different set of questions to those answered by users.
| Responses | |
|---|---|
| Total responses | 859 |
| Filtered during data cleansing | -82 |
| Remaining after data cleansing (total answering questions) | 777 |
| Not yet considered buying | -24 |
| Total answering questions | 753 |
| Responses | |
|---|---|
| Users | 593 |
| Consultants | 95 |
| All users | 688 |
| Vendors/Resellers | 65 |
Organization sizes by headcount
Business intelligence and analytics products are most commonly found in midsize and large organizations. 86% of the responses we received came from users in companies with more than 100 employees.
Vertical markets
We asked all respondents their company’s industry sector. The chart shows the results of this question and only includes data from respondents who answered product-related questions in The Analytics & AI Survey.
Having dominated the list for several years, manufacturing is still the best represented industry in The Analytics & AI Survey.
Products in The Analytics & AI Survey
At least 20 user reviews are necessary to be included in the detailed product analysis. In this year’s Analytics & AI Survey we analyze 17 BI & analytics tools (or groups of products) in detail.
When grouping and describing the business intelligence & analytics solutions featured in The Analytics & AI Survey, we did not strictly follow the naming conventions that the vendors use. In some cases, we combined various products to make analysis more convenient. Note that the names we use in The Analytics & AI Survey are our own and are not always the official product names used by the vendors.
One of the key reasons for this is that the products we analyze are not necessarily the latest version of the tool. Vendors will often change a product’s name between versions, making it difficult to have a single official name for several versions of the same product.
Another reason is that we sometimes bundle related products into a single group to increase the sample size, even if the vendor prefers to view them as distinct for marketing reasons. In both these cases, the point is not to challenge the naming conventions of the vendor, but simply to reduce the complexity of the survey findings for the convenience of the reader. In some cases, we also shorten the names of the products to improve the formatting of charts.
We asked respondents explicitly about their experiences with products from a predefined list, with the option to nominate other products. This list is updated each year and is based on the sample size of the products in the previous year, as well as additional new products in the BI & analytics market. In cases where respondents said they were using an ‘other’ product, but from the context it was clear that they were actually using one of the listed products, we reclassified their data accordingly.
We solicited responses on all surviving products with more than a minimal response in last year’s survey, plus a few others whose numbers had potentially grown to the point where there would be enough data to be analyzed.
The table shows the products included in the detailed analysis.
| Product name | Respondents |
|---|---|
| Domo | 20 |
| enventa BI | 22 |
| GoodData | 20 |
| IBM Cognos Analytics | 22 |
| InfoZoom | 29 |
| Microsoft Power BI | 117 |
| MyReport | 38 |
| Oracle Analytics Cloud | 21 |
| Phocas | 24 |
| Pyramid | 23 |
| Qlik Sense | 34 |
| SAP Analytics Cloud | 64 |
| SAS Viya | 20 |
| Strategy One | 21 |
| Tableau | 28 |
| TARGIT | 29 |
| Zoho Analytics | 22 |
The Peer Groups
The Analytics & AI Survey 27 features a range of different types of BI and analytics tools so we use peer groups to help identify competing products. They are used to ensure similar BI & analytics tools are compared against each other.
The groups are essential to allow fair and useful comparisons of BI & analytics tools that are likely to compete. The peer groups are primarily based on the results from The Analytics & AI Survey, how customers say they use their products and our knowledge of those products.
We categorize the different BI & analytics products into peer groups, based on three criteria: implementation size, usage scenario and global presence.
Peer groups act as a guide to the reader to help make the products easier to understand and to show why individual products return such disparate results. They are not intended to be a judgment of the quality of the products. Most products appear in more than one peer group. These peer groups are used in a consistent way in our analysis as well as in The Analytics & AI Survey Analyzer.
The KPIs
The goal of the KPIs is to help the reader spot winners and losers in The Analytics & AI Survey 27 using well designed dashboards packed with concise information. The Analytics & AI Survey includes 5 aggregated KPIs, which can be absorbed at a glance. It also includes a set of 25 normalized KPIs, which we refer to as ‘root’ KPIs for each of the 17 products. The ‘aggregated’ KPIs are aggregations of these root KPIs.
This year we have calculated a set of KPIs for each of the 5 peer groups. The values are normalized on the whole sample. Peer groups are used to enable fair and useful comparisons of products that are likely to compete.
The KPIs all follow these simple rules:
- Only measures that have a clear good/bad trend are used as the basis for KPIs.
- KPIs may be based on one or more measures from The Analytics & AI Survey.
- Only products with samples of at least 15 – 30 (depending on the KPI) for each of the questions that feed into the KPI are included.
- Each KPI is measured on a scale from 0 (lowest possible value) to 10 (highest possible value).
- In some instances, adjustments are made to account for extreme outliers.
KPIs are only calculated if the samples have at least 15-30 data points (this varies depending on the KPI) and if the KPI in question is applicable to a product. Therefore some products do not have a full set of root KPIs. It is important to exclude KPIs based on small (and therefore not representative) samples to ensure that the graph scales are not distorted by outlier KPIs.
Measuring business benefits – the BBI and best-in-class companies
Business benefits are the real reason for carrying out any BI & analytics project and The Analytics & AI Survey has been studying them directly for years. We ask respondents the extent to which they realize a list of benefits with their BI tools.
For each potential benefit, respondents are asked to indicate the level of achievement, if any, with five levels. We use a weighted scoring system to derive a composite score for each of the possible benefits, based on the level of benefit achieved. We call this the BBI (Business Benefits Index).
This rating system is the basis of the most important index in The Analytics & AI Survey. It is a dimensionless number with an arbitrary value, but as long as the weighting system remains constant it can be used for comparisons between segments of the sample, such as the sample for individual products or regions, to name just two.
Throughout The Analytics & AI Survey, we use a classification of best-in-class companies and laggards. Best-in-class companies comprise the top 10 percent of companies, based on their achievement of business benefits, while laggards are defined as the 10 percent of companies achieving the lowest level of business benefits. This classification enables us to examine correlations between best-in-class/laggard companies and high (or low) KPI scores, and to identify what the most successful companies do differently to laggards.
For a more detailed description of the KPIs and how we calculated them, see our Sample, Products, Methodology and KPIs PDF.
| Aggregated KPIs | Root KPIs |
|---|---|
| Business Value |
Business Benefits Implementation Success Implementation Speed Price to Value |
| Customer Satisfaction |
Vendor Support Implementer Support Recommendation Product Satisfaction Sales Experience Ecosystem Satisfaction |
| Functionality |
Data Connectivity Data Preparation Semantic & Metric Layer Dashboards & Reports Analytics Insight Delivery Embedded Analytics |
| User Experience |
Ease of Use Self-Service Flexibility Performance & Scalability |
| AI Experience |
AI Assistance AI Transparency AI Insights GenAI Assistance |