Predicting the 2019 MQ
The die is cast. Last month, Gartner selected 16 vendors to include in its 2019 Magic Quadrant for Data Science and Machine Learning. Now, as Gartner prepares to publish the report early next year, I think it will be fun to make some predictions about how each vendor will fare.
Some ground rules. I’m not going to talk about DataRobot, my employer. Nor will I discuss the accuracy or value of Gartner’s assessments. People can agree or disagree with Gartner, but many people trust their analysis. Vendors invest a significant amount of time participating in the MQ; at a minimum, they believe there is a segment of customers who trust Gartner completely.
Several factors make this task hard:
— Gartner has information that is not available to the public. Vendors brief Gartner about their vision under nondisclosure. A firm that plans a major shift in strategy might disclose it to Gartner but not to the public. Gartner surveys customers.
— Gartner’s evaluation criteria are not static. Each year, Gartner adds to the list of product features and functions it uses to assess current offerings. Participating vendors can influence Gartner’s view of the future. Thus, a vendor that submits the same materials from one year to the next could see a marked decline in its rating if Gartner believes that the market as a whole has moved forward.
In making these predictions, I’m going to focus on the areas Gartner says are challenges for each vendor, and see how well the vendor has addressed these challenges since the last report.
Many of my predictions will be wrong, for the reasons cited above. But the only way to avoid prediction error is to avoid predictions.
The Gartner MQ
First, let’s talk about how Gartner evaluates vendors.
The horizontal dimension, Completeness of Vision (CoV), depends on Gartner’s assessment of a vendor’s market understanding and product strategy. In practice, a high score on this dimension means that the vendor’s view of the world aligns with Gartner’s view of the world. Vendors who work actively with Gartner’s analysts and act on Gartner recommendations tend to do well on CoV.
Vendors who pivot strategically and make dramatic changes (such as acquisitions or major product launches) can increase the CoV score markedly. For example, when Microsoft launched Azure Machine Learning Studio and acquired Revolution Analytics three years ago, it moved dramatically to the right on the MQ, from Niche Player to Visionary.
The vertical dimension is Ability to Execute (AtE). Product features affect this dimension, but a vendor’s product score is just one of several factors. Others include a vendor’s viability (which is relatively easy to assess) and results of the customer survey (which is not.) Vendors with strong customer experience scores from the survey tend to keep them from year to year (and vice-versa), but there are exceptions to that rule of thumb. Dataiku, for example, dropped precipitously on AtE from 2017 to 2018, primarily due to concerns surfaced in the customer survey.
Since a vendor’s product score accounts for only part of the overall AtE score, minor product enhancements do not produce dramatic movement from year to year. The most predictable improvements in AtE stem from major new products introduced relatively late in Gartner’s MQ cycle. When this happens, Gartner factors the product into a vendor’s CoV, but the product does not impact AtE until a minimum number of customers use it, typically in the following year. Gartner discloses these situations, so a careful reading of the MQ report provides information we can use to predict a vendor’s rating in the following year.
For example, H2O.ai released Driverless AI to Alpha last July. Since H2O.ai had no production customers for the product, its features did not impact H2O.ai’s product and AtE scores, but Gartner factored the release into H2O.ai’s CoV rating. This year, the product is generally available and can affect H2O.ai’s product score and AtE (if it has a sufficient number of customers.)
Vendor Assessments and Predictions
According to Gartner, Alteryx’s strong upward movement on Ability to Execute (AtE) in 2018 reflects its revenue growth, customer acquisition, and the IPO. The strategy behind the Yhat acquisition positively affected its Completeness of Vision (CoV.) For 2019, the Yhat acquisition is already baked into Alteryx’ CoV score. Alteryx hasn’t materially addressed the deficiencies Gartner cited in machine learning and enterprise readiness.
Prediction: Alteryx will remain a leader, and will maintain its position on AtE, but could decline slightly on CoV.
In 2018, H2O markedly improved its position on both dimensions. The improved CoV reflects the company’s development of Driverless AI and work with GPU acceleration. Since these innovations are already reflected in Gartner’s assessment, so it’s not likely that H2O will improve on CoV in 2018.
H2O’s November funding round and partnership with NVIDIA may drive an improved AtE rating. Driverless AI is a distinct product, it may or may not affect H2O’s dot position, as it is unlikely that there are enough reference customers for the product to meet Gartner’s inclusion criteria.
Prediction: H2O will improve its rating on AtE based on General Availability of Driverless AI and continued progress with GPU acceleration.
KNIME improved markedly on CoV in 2018, retaining its position in the Leader quadrant. KNIME hasn’t materially addressed the challenges cited by Gartner in the 2018 report.
Prediction: KNIME will hold its position on AtE based on solid customer satisfaction. The company hasn’t materially addressed the challenges Gartner cited in the 2018 report and could decline on CoV.
In 2018, RapidMiner declined somewhat on AtE. Product improvements in Release 8 and 9 are interesting, but they don’t address any of the concerns Gartner surfaced in the MQ. The company struggles to raise new funding, which may impair Gartner’s assessment of its viability.
Prediction: RapidMiner will continue to decline on AtE, but not by enough to fall out of the Leader quadrant.
For 2018, Gartner writes that “SAS remains a Leader, but has lost some ground in terms of both Completeness of Vision and Ability to Execute. The Visual Analytics suite shows promise because of its Viya cloud-ready architecture, which is more open than prior SAS architecture and makes analytics more accessible to a broad range of users. However, a confusing multiproduct approach has worsened SAS’s Completeness of Vision, and a perception of high licensing costs has impaired its Ability to Execute. As the market’s focus shifts to open-source software and flexibility, SAS’s slowness to offer a cohesive, open platform has taken its toll.”
Prediction: SAS seems to be singing from a new hymnal, which may be enough to boost its rating on CoV. SAS’ multi-product approach and pricing model are baked into its business model, and will not change. However, strong customer satisfaction offsets these concerns. SAS remains a leader.
Databricks announced support for Azure in November 2017, too late for Gartner to consider it in the 2018 MQ. The company has announced no other major moves or enhancements.
Prediction: Databricks will improve on AtE (reflecting the Azure expansion.)
Dataiku declined markedly on both dimensions in 2018. Releases 4.3 and 5.x of Data Science Studio address some concerns surfaced by Gartner in the 2018 report. Release 4.3 addresses issues in model deployment. Release 5.x delivers containerized workloads and support for deep learning.
Prediction: Dataiku will improve on AtE.
Domino Data Lab
In 2018, Gartner highlighted Domino’s weakness in data access, preparation, exploration, and visualization. These are inherent in Domino’s architecture and approach, and unlikely to change.
Prediction: Domino will hold its position.
IBM fell out of the Leaders quadrant in 2018, declining on both dimensions. Gartner raised concerns about IBM’s complex and confusing mix of brands, poor customer experience, and slow improvements to SPSS. IBM is structurally unable to address these issues. The proliferation of brands reflects the many different fiefdoms within IBM, who seem to be unable to coordinate or agree on a unified strategy. Customer experience issues stem from IBM’s ongoing headcount reduction and outsourcing of customer service to low-cost venues. And IBM is simply unwilling to make significant investments in SPSS, preferring instead to develop new products under the Watson brand.
In 2018, Gartner evaluated SPSS only; Data Science Experience did not qualify for inclusion. IBM has subsequently rebranded DSX as IBM Watson Studio and rolled some SPSS features into it. This new product may or may not meet Gartner’s requirements for inclusion. Forrester likes Watson Studio a lot; personally, I think it lacks coherence. In any case, IBM Watson Studio is available in IBM Cloud only, which Gartner will see as a serious limitation.
Prediction: IBM will continue to decline on CoV. IBM Watson Studio is a modest improvement over last year’s product but is unlikely to have a sufficient number of customers to influence IBM’s AtE rating. IBM remains a Visionary.
Microsoft’s position in the MQ has not changed significantly in three years. Gartner attributes its score on CoV to “low scores for market responsiveness and product viability, as Azure Machine Learning Studio’s cloud-only nature limits its usability for the many advanced analytic use cases that require an on-premises option.”
Gartner has criticized Microsoft for at least two consecutive MQs for lacking an on-premises capability. This is puzzling because Microsoft clearly has such a capability in Microsoft Machine Learning Server, the latest version of the Revolution Analytics code base acquired in 2015. Gartner implies that it would have considered this product, but that Microsoft did not release the latest version until after the cutoff date for the MQ.
Prediction: Microsoft will articulate a unified vision for Machine Learning Server and Azure Machine Learning, and submit both products for evaluation, which will improve the company’s score on both dimensions. Microsoft’s recent acquisition of Bonsai will improve its CoV score.
Gartner notes that Mathworks’ CoV “is limited by its focus on engineering and high-end financial use cases, largely to the exclusion of customer-facing use cases like marketing, sales and customer service.” This is not likely to change.
Prediction: Mathworks will hold its position.
TIBCO entered this MQ when it acquired Statistica in 2017; subsequently, TIBCO acquired Alpine Data. For 2018, Gartner notes that “in terms of Ability to Execute, this Magic Quadrant evaluates only TIBCO’s ability with the Statistica platform. Other acquisitions by TIBCO contribute only to its Completeness of Vision.” Since TIBCO’s CoV score is less than that which Statistica achieved on its own, TIBCO’s acquisitions did not contribute at all to its CoV.
TIBCO has rebranded Alpine as Spotfire Data Science, but actual integration of the software is “still in its early stages” as Gartner puts it. Alpine scored near the bottom on AtE in 2016, the last time it appeared on its own. Under Dell ownership, Statistica made it into the Leader quadrant in the 2016 MQ but fell back into the Challenger quadrant in for 2017 and 2018.
Prediction: Until TIBCO can integrate the Alpine code base with Statistica and Spotfire, the acquisition has no impact on TIBCO’s AtE score. If TIBCO can articulate a coherent vision for the acquired products, its CoV will improve.
2018 Niche Players
Anaconda holds a weak position in the Niche quadrant and has made no significant moves since last year.
Prediction: Anaconda remains a Niche player.
Datawatch acquired Angoss in January 2018. Datawatch owns Monarch Panopticon, which appeared in the 2016 MQ for BI but not since then.
Prediction: Datawatch is a bottom feeder, and is unlikely to participate in the MQ.
SAP entered the MQ when it acquired KXEN in 2013. Since then, it has steadily declined on AtE, which is somewhat surprising for an organization with SAP’s deep pockets. Gartner notes that SAP Leonardo was not a factor in the company’s AtE rating and that the Leonardo vision seems decoupled from the existing product. In any case, Leonardo also was not a factor in SAP’s CoV rating, which increased only slightly.
SAP’s vision, customer experience, and product development issues appear to be structural, and unlikely to change.
Prediction: SAP will continue to decline in AtE. The Leonardo rollout will not improve SAP’s CoV position for this MQ.
Gartner writes that Teradata’s “lack of cohesion and ease of use on the data science development side have impaired both its Ability to Execute and its progress on the Completeness of Vision axis. It remains a Niche Player.” Teradata recently retired Aster, a core component of its data science offering.
Teradata executives talked a lot about machine learning at its recent event. Teradata talking about machine learning is like Mel Gibson talking about his boobs. He doesn’t have any, and nobody would care if he did.
Prediction: Teradata will be kicked out of the MQ entirely.
There are likely to be two spots open in the MQ for new entrants. In 2018, Gartner cited the following vendors for Honorable Mention:
- Big Squid
- Pitney Bowes
Of these, Amazon has the best chance to join the MQ.
- Amazon actively participates in other Gartner MQs, with some success. Sagemaker and AWS AI/ML services are a powerful combination. AWS has a good track record in articulating a vision that satisfies with Gartner’s view of the world. If AWS wants to jump in, they could qualify almost as well as Databricks.
- Big Squid is not ready for the big leagues.
- DataScience.com had a decent product but struggled commercially. The acquisition by Oracle makes the company viable, but Oracle’s rebranded version of the product will not be available until June 2019.
- FICO lacks the broad industry appeal required for this MQ
- Google Cloud DataLab is Google’s equivalent to Sagemaker, and not nearly as good. Google’s automated machine learning is still in Alpha release, and it supports image recognition only. Google doesn’t seem to care much about competing in Gartner MQs.
- Megaputer dropped out of the MQ in 2017 and isn’t likely to drop back in.
- Pitney Bowes is just mailing it in at this point.
Cloudera is another potential new entrant. Cloudera did not meet Gartner’s inclusion criteria for the 2018 MQ but might qualify for the 2019 MQ. Cloudera Data Science Workbench is not as good as Domino Data Lab and Databricks, but better than Anaconda. If Cloudera enters, it will be a Niche Vendor.