Revolution R Enterprise – Thomas Dinsmore's Blog

Microsoft Buys Revolution Analytics

On Friday, January 23, Microsoft announced an agreement to acquire Revolution Analytics. Coverage of the announcement in the media is extensive, with stories by TechCrunch, Wired, ZDNet, VentureBeat and many others (here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here, here and here.)

Microsoft did not disclose the negotiated purchase price; Revolution’s total capitalization is around $40 million. Given Revolution’s scale of operations, the acquisition will have minimal impact on Microsoft’s near-term revenue and profit.

Many analysts follow Microsoft, but few have heard of Revolution Analytics, and most seem to be stumped by this move. An example:

Question: What is the significance of Microsoft acquiring Revolution Analytics?

Answer: I am not sure.

Microsoft gets four things with this deal:

Instant credibility with the growing open source analytics community
Consulting and support skills to help enterprise customers adopt R
A capable engineering organization (conveniently located in Seattle)
Software bits that should integrate well with the Microsoft stack

In addition to its primary offering, Revolution R Enterprise, Revolution distributes Revolution R Open, an enhanced free distribution of open source R; and Revolution R Cloud an elastic offering on the AWS Marketplace. Revolution R Open is equivalent in many respects to Oracle R Distribution, which is also compiled with the Intel Math Kernel Libraries. Revolution R Plus is commercially supported, and includes additional software bits for enterprise integration; this product is comparable to Oracle R Enterprise.

Revolution Analytics’ other key software assets include ScaleR, a distributed out-of-memory back end with a strong R interface; DeployR, a component that supports enterprise deployment of web-based applications; and DevelopR, a Windows-based IDE.

While the IDE has a number of useful features, it requires significant investment to compete effectively with RStudio, which has won the hearts and minds of R users. Upgrading software simply to make it competitive with a “free” competitor strikes me as a dubious commercial move; it seems more likely that Microsoft will add an R capability to the Visual Studio suite.

Revolution’s ScaleR back end enables R users to leverage a platform for distributed analytics. ScaleR already runs on Windows Server HPC clusters, which should make integration with Azure a straightforward matter. This is important for Microsoft, since Azure Machine Learning currently maxes out at around 10Gb.

ScaleR’s integration with Hadoop currently runs through MapReduce; competing best-in-class Hadoop analytics (such as Spark, H2O, Skytree and SAS) run in memory for better performance. Microsoft’s deep pockets give Revolution the means to make this product competitive.

2015: Predictions for Big Analytics

First, a review of last year’s predictions:

(1) Apache Spark matures as the preferred platform for advanced analytics in Hadoop.

At the New York Strata/Hadoop World conference in October, if you took a drink each time a speaker said “Spark”, you would struggle to make it past noon. At my lunch table, every single person said his company is currently evaluating Spark. There are few alternatives to Spark for advanced analytics in Hadoop, and the platform has arrived.

(2) “Co-location” will be the latest buzzword.

Few people use the word “co-location”, but thanks to YARN, vendors like SAS and Skytree are now able to honestly position their products as running “inside” Hadoop. YARN has changed the landscape for analytics in Hadoop, so that products that interface through MapReduce are obsolete.

(3) Graph engines will be hot.

Graph engines did not take off in 2014. Development on Apache Giraph has flatlined, and open source GraphLab is quiet as well. Apache Spark’s GraphX is the only graph engine for Hadoop under active development; the Spark team recently promoted GraphX from Alpha to production. However, with just 10 out of 132 contributors working on GraphX in Release 1.2, the graph engine is relatively quiet compared to the SQL, Machine Learning and Streaming modules.

(4) R approaches parity with SAS in the commercial job market.

As of early 2014, when Bob Muenchin last updated his job market statistics, SAS led R in job postings, but R was closing the gap rapidly.

Linda Burtch of Burtch Works is the nation’s leading executive recruiter for quants and data scientists. I asked Linda what analytic languages hiring managers seek when they hire quants. “My clients are still more frequently asking for SAS, although many more are now asking for either SAS or R,” she says. “I also recommend to my clients who ask specifically for SAS skills to be open to those using R, and many will agree after the suggestion. ”

(5) SAP emerges as the company most likely to buy SAS.

After much hype about the partnership in late 2013, SAS and SAP issued not a single press release in 2014. The dollar’s strength against the Euro makes it less likely that SAP will buy SAS.

(6) Competition heats up for “easy to use” predictive analytics.

Software companies target the “easy to use” analytics market because it’s larger than the expert market and because expert analysts rarely switch. Alpine, Alteryx, and Rapid Miner all gained market presence in 2014; Dell’s acquisition of Statsoft gives that company the deep pockets they need for a makeover. In easy to use cloud analytics, StatWing has added functionality, and IBM Watson Analytics emerged from beta.

Four out of six ain’t bad. Now looking ahead:

(1) Apache Spark usage will explode.

While interest in Spark took off in 2014, relatively few people actually use the platform, which appeals primarily to hard-core data scientists. That will change in 2015, for several reasons:

The R interface planned for release in Q1 opens the platform to a large and engaged community of users
Alteryx, Alpine and other easy to use analytics tools currently support or plan to support Spark RDDs as a data source
Databricks Cloud offers an easy way to spin up a Spark cluster

As a result of these and other innovations, there will be many more Spark users in twelve months than there are today.

(2) Analytics in the cloud will take off.

Yes, I know — some companies are reluctant to put their “sensitive” data in the cloud. And yet, all of the top ten data breaches in 2014 defeated an on-premises security system. Organizations are waking up to the fact that management practices are the critical factor in data security — not the physical location of the data.

Cloud is eating the analytics world for three big reasons:

Analytic workloads tend to be lumpy and difficult to predict
Analytic projects often need to get up and running quickly
Analytic service providers operate in a variable cost world, with limited capital for infrastructure

Analytic software options available in the Amazon Marketplace are increasing rapidly; current options include Revolution R, BigML and YHat, among others. For the business user, StatWing and IBM Watson Analytics provide compelling independent cloud-based platforms.

Even SAS seeks to jump on the Cloud bandwagon, touting its support for Amazon Web Services. Cloud devotees may be disappointed, however, to discover that SAS does not offer elastic pricing for AWS, lacks a native access engine for RedShift, and does not support its Hadoop interface with EMR.

(3) Python will continue to gain on R as the preferred open source analytics platform.

The Python versus R debate is at least as contentious as the SAS versus R debate, and equally tiresome. As a general-purpose scripting language, Python’s total user base is likely larger than R’s user base. For analytics, however, the evidence suggests that R still leads Python, but that Python is catching up. According to a recent poll by KDNuggets, more people switch from R to Python than the other way ’round.

Both languages have their virtues. The sheer volume of analytic features in R is much greater than Python, though in certain areas of data science (such as Deep Learning) Python appears to have the edge. Devotees of each language claim that it is easier to use than the other, but the two languages are at rough parity by objective measures.

Python has two key advantages over R. As a general-purpose language, it is a better tool for application development; hence, for embedded analytic applications (such as recommendation engines, decision engines and online scoring), Python gets the nod over R. Second, Python’s open source license is less restrictive than the R license, which makes it a better choice for commercial use. There are provisions in the R license that scare the pants off some company lawyers, rightly or wrongly.

(4) H2O will continue to win respect and customers in the Big Analytics market.

If you’re interested in scalable analytics but haven’t checked out H2O, you should. H2O is a rapidly growing true open source project for distributed analytics; it runs in clusters, in Hadoop and in Amazon Cloud; offers an excellent R interface together with Java and Scala APIs; and is accessible from Tableau. H2O supports a rich and growing machine learning library that includes Deep Learning and the only available distributed Gradient Boosting algorithm on the market today.

While the software is freely available, H2O offers support and services for an attractive price. The company currently claims more than two thousand users, including reference customers Cisco, eBay, Nielsen and Paypal.

(5) SAS customers will continue to seek alternatives.

SAS once had an almost religious loyalty from its customers. This is no longer the case; in a recent report published by Gartner, surveyed executives reported they are more likely to discontinue use of SAS than any other business intelligence software. While respondents rated SAS above average on sales experience and average on product quality, SAS fared poorly in measures of usability and ease of integration. While the Gartner survey does not address pricing, it’s fair to say that no vendor can command premium prices without an outstanding product.

While few enterprises plan to pull the plug on SAS entirely, many are limiting growth of the SAS footprint and actively developing alternatives. This is especially marked in the analytic services industry, which tends to attract people with the skills to use Python or R, and where cost control is important. Even among big banks and pharma companies, though, SAS user headcount is declining.

Distributed Analytics: A Primer

Can we leverage distributed computing for machine learning and predictive analytics? The question keeps surfacing in different contexts, so I thought I’d take a few minutes to write an overview of the topic.

The question is important for four reasons:

Source data for analytics frequently resides in distributed data platforms, such as MPP appliances or Hadoop;
In many cases, the volume of data needed for analysis is too large to fit into memory on a single machine;
Growing computational volume and complexity requires more throughput than we can achieve with single-threaded processing;
Vendors make misleading claims about distributed analytics in the platforms they promote.

First, a quick definition of terms. We use the term parallel computing to mean the general practice of dividing a task into smaller units and performing them in parallel; multi-threaded processing means the ability of a software program to run multiple threads (where resources are available); and distributed computing means the ability to spread processing across multiple physical or virtual machines.

The principal benefit of parallel computing is speed and scalability; if it takes a worker one hour to make one hundred widgets, one hundred workers can make ten thousand widgets in an hour (ceteris paribus, as economists like to say). Multi-threaded processing is better than single-threaded processing, but shared memory and machine architecture impose a constraint on potential speedup and scalability. In principle, distributed computing can scale out without limit.

The ability to parallelize a task is inherent in the definition of the task itself. Some tasks are easy to parallelize, because computations performed by each worker are independent of all other workers, and the desired result set is a simple combination of the results from each worker; we call these tasks embarrassingly parallel. A SQL Select query is embarrassingly parallel; so is model scoring; so are many of the tasks in a text mining process, such as word filtering and stemming.

A second class of tasks requires a little more effort to parallelize. For these tasks, computations performed by each worker are independent of all other workers, and the desired result set is a linear combination of the results from each worker. For example, we can parallelize computation of the mean of a distributed database by computing the mean and row count independently for each worker, then compute the grand mean as the weighted mean of the worker means. We call these tasks linear parallel.

There is a third class of tasks, which is harder to parallelize because the data must be organized in a meaningful way. We call a task data parallel if computations performed by each worker are independent of all other workers so long as each worker has a “meaningful” chunk of the data. For example, suppose that we want to build independent time series forecasts for each of three hundred retail stores, and our model includes no cross-effects among stores; if we can organize the data so that each worker has all of the data for one and only one store, the problem will be embarrassingly parallel and we can distribute computing to as many as three hundred workers.

While data parallel problems may seem to be a natural application for processing inside an MPP database or Hadoop, there are two constraints to consider. For a task to be data parallel, the data must be organized in chunks that align with the business problem. Data stored in distributed databases rarely meets this requirement, so the data must be shuffled and reorganized prior to analytic processing, a process that adds latency. The second constraint is that the optimal number of workers depends on the problem; in the retail forecasting problem cited above, the optimal number of workers is three hundred. This rarely aligns with the number of nodes in a distributed database or Hadoop cluster.

There is no generally agreed label for tasks that are the opposite of embarrassingly parallel; for convenience, I use the term orthogonal to describe a task that cannot be parallelized at all. In analytics, case-based reasoning is the best example of this, as the method works by examining individual cases in a sequence. Most machine learning and predictive analytics algorithms fall into a middle ground of complex parallelism; it is possible to divide the data into “chunks” for processing by distributed workers, but workers must communicate with one another, multiple iterations may be required and the desired result is a complex combination of results from individual workers.

Software for complex machine learning tasks must be expressly designed and coded to support distributed processing. While it is physically possible to install open source R or Python in a distributed environment (such as Hadoop), machine learning packages for these languages run locally on each node in the cluster. For example, if you install open source R on each node in a twenty-four node Hadoop cluster and try to run logistic regression you will end up with twenty-four logistic regression models developed separately for each node. You may be able to use those results in some way, but you will have to program the combination yourself.

Legacy commercial tools for advanced analytics provide only limited support for parallel and distributed processing. SAS has more than 300 procedures in its legacy Base and STAT software packages; only a handful of these support multi-threaded (SMP) operations on a single machine; nine PROCs can support distributed processing (but only if the customer licenses an additional product, SAS High-Performance Statistics). IBM SPSS Modeler Server supports multi-threaded processing but not distributed processing; the same is true for Statistica.

The table below shows currently available distributed platforms for predictive analytics; the table is complete as of this writing (to the best of my knowledge).

Distributed Analytics Software, May 2014

Several observations about the contents of this table:

(1) There is currently no software for distributed analytics that runs on all distributed platforms.

(2) SAS can deploy its proprietary framework on a number of different platforms, but it is co-located and does not run inside MPP databases. Although SAS claims to support HPA in Hadoop, it seems to have some difficulty executing on this claim, and is unable to describe even generic customer success stories.

(3) Some products, such as Netezza and Oracle, aren’t portable at all.

(4) In theory, MADLib should run in any SQL environment, but Pivotal database appears to be the primary platform.

To summarize key points:

— The ability to parallelize a task is inherent in the definition of the task itself.

— Most “learning” tasks in advanced analytics tasks are not embarrassingly parallel.

— Running a piece of software on a distributed platform is not the same as running it in distributed mode. Unless the software is expressly written to support distributed processing, it will run locally, and the user will have to figure out how to combine the results from distributed workers.

Vendors who claim that their distributed data platform can perform advanced analytics with open source R or Python packages without extra programming are confusing predictive model “learning” with simpler tasks, such as scoring or SQL queries.