My roundup of the Spark Summit Europe is here.
Two important events this week:
- H2O World starts today and runs through Wednesday at the Computer History Museum in Mountain View CA. Yotam Levy summarizes here and here.
- Open Data Science Conference meets November 14-15 at the Marriott Waterfront in SFO
Five backgrounders and explainers:
- At HUG London, Apache’s Ufuk Celebi delivers a nice intro to Flink.
- On the Databricks blog, Yesware’s Justin Mills explains how his team migrates Spark applications from concept through prototype through production.
- On Slideshare, Alpine’s Holden Karau delivers an overview of Spark with Python.
- Chloe Green wakes from a three year slumber and discovers Spark.
- On the Cloudera Engineering blog, Madhu Ganta explains how to build a CEP app with Spark and Drools.
Third quarter financials drive the news:
(1) MapR: We Grew 160% in Q3
MapR posts its biggest quarter ever.
(2) HDP: We Grew 168% in Q3
HDP loses $1.33 on every dollar sold, tries to make it up on volume. Stock craters.
(3) Teradata: We Got A Box of Steak Knives in Q3
Teradata reports more disappointing sales as customers continue to defer investments in big box solutions for data warehousing. This is getting to be a habit with Teradata; the company missed revenue projections for 2014 as well as the first and second quarters of this year. Any company can run into headwinds, but a management team that consistently misses targets clearly does not understand its own business and needs to go.
Full report here.
(4) “B” Round for H2O.ai
Machine learning software developer H2O.ai announces a $20 million Series B round led by Paxion Capital Partners. H2O.ai leads development of H2O, an open source project for distributed in-memory machine learning. The company reports 25 new support customers this year.
(5) Fuzzy Logix Lands Funds
In-database analytics vendor Fuzzy Logix announces a $5 million “A” round from New Science Ventures. Fuzzy offers a library of analytic functions that run in a number of high-performance databases and in HiveQL.
(6) New Optimization Package for Spark
On the Databricks blog, Aaron Staple announces availability of Spark TFOCS, an optimization package based on the eponymous Matlab package. (TFOCS=Templates for First Order Conic Solvers.)
(7) WSO2 Delivers IoT App on Spark
IoT middleware vendor WSO2 announces Release 3.0 of its open source Data Analytics Server (DAS) platform. DAS collects data streams and applies batch, real-tim or interactive analytics; predictive analytics are in the roadmap. For streaming data sources, DAS supports java agents, javascript clients and 100+ connectors. The software runs on Spark and Lucene.
(8) Hortonworks: We Aren’t Irrelevant
On the Hortonworks blog, Vinay Shukla and Ram Sriharsha tout Hortonworks’ contributions to Spark, including ORC support, an Ambari stack definition for Spark, tighter integration between Hive and Spark, minor enhancements to ML and user-facing documentation. Looking at the roadmap, they discuss Magellan for geospatial and Zeppelin notebooks. (h/t Hadoop Weekly).
(9) Apache Drill Delivers Fast SQL-on-Laptop
On the MapR blog, Mitsutoshi Kiuchi offers a case study in how to run a silly benchmark.
Comparing the functionality of Drill and Spark SQL, Kiuchi argues that Drill “supports” NoSQL databases but Spark does not, relegating Spark’s packages to a footnote. “Support” is a loaded word with open source software; technically, nothing is supported unless you pay for it, in which case the scope of support is negotiated as part of the SLA. It’s also worth noting that MongoDB developed Spark’s interface to MongoDB (for example), which provides a certain amount of confidence.
Kiuchi does not consider other functional areas, such as security, YARN support, query fault tolerance, the user interface, metastore management and view support, where Drill comes up short.
In a previously published performance test of five SQL engines, Spark successfully ran nine out of eleven queries, while Drill ran eight out of ten. On the eight queries both engines ran, Drill was slightly faster on six. For this benchmark, Kiuchi runs three queries on his laptop with a tiny dataset.
As a general rule, one should ignore SQL-on-Hadoop benchmarks unless they run industry standard queries (e.g. TPC) with large datasets in a distributed configuration.
Thomas Dinsmore's Blog 