Models can be trained by data scientists in Apache Spark using R or Python, saved using MLlib, and then imported into a Java-based or Scala-based pipeline for production use.

While R is a newcomer to Spark, it already has a solid number of users compared to the other languages that Spark supports, including Python, Java, and Scala. “Give it a year. I definitely think it’s ...

For instance, with Apache Spark having been written in Scala and optimized for running Scala or Java programs, this often left R and Python developers out in the cold.

This monolithic architecture creates dependencies between the Spark code that people develop using whatever language (Scala, Java, Python, etc.) and the Spark cluster itself. Those dependencies, in ...