Every year, an increasing number of distributed systems to manage data are introduced to the industry. Among them, Spark and Hadoop have emerged as the most successful ones. This article discusses these two systems and tries to find out which one is better.
What’s Hadoop?
Hadoop is a general-purpose form of distributed processing that consists of several components. The Hadoop Distributed File System (HDFS), YARN and MapReduce are some very important components of Hadoop. Even though this system is entirely built in Java, it is accessible through many other languages including Python. An SQL like interface which allows running queries on HDFS, Hive is another important feature of Hadoop.
What’s Spark?
Spark is a relatively new project developed in 2012. It enables us to process data in parallel across a cluster. The major difference with Hadoop is that it works in-memory. Spark can process data in RAM using a concept called RDD or Resilient Distributed Dataset. It also comes with several APIs. Even though the original interface was written in Scala, based on the heavy usage by data scientists, R and Python endpoints were also provided.
Now let’s take a look at these platforms in different perspectives such as performance, cost and machine learning.
Performance
It is found that spark can run 100 times faster in-memory and ten times faster on disk than Hadoop. Especially when it comes to machine learning applications such as Naive Bayes and K-means, Spark is much faster. Following are the crucial reasons behind the better performance of Spark.
While running a selected part of a MapReduce task, Spark is not limited by the input-output concerns. It enables faster operation in applications.
The DAGs of spark permits optimization between each step. So, there would be performance tuning during the process which is not present in Hadoop.
However, in situations where the spark is running on YARN, the performance is found to be reduced. Also, sometimes it could lead to RAM overhead memory leaks. So, in a batch processing use-case, Hadoop is the more efficient system.
Costs
Since both Spark and Hadoop are open-source Apache projects, you can potentially use them with zero installation cost. However, there are other costs such as maintenance, hardware purchase and costs of supporting team. We know that the Hadoop requires more memory on disk and spark requires more RAM. In that sense, spark clusters are more expensive to set up. Also, since it is a new system, the experts of Spark would be rarer and more expensive.
Machine Learning Capabilities
Spark comes with a machine learning library, MLLib to use for the iterative machine learning applications. It includes regression and classification. Also, you can build machine learning pipelines with hyperparameter tuning using it.
Hadoop makes use of Mahout to process data. It has clustering, batch based collaborative filtering, and classification. Lately, it is being phased out in favor of Samsara. It is a Scala-backed DSL language and allows you to build your own algorithms.
Conclusion
It is sure that these two are the most prominent distributed systems available today for data processing. Between them, Hadoop is mainly recommended for disk-heavy operations while Spark is more flexible. However, the in-memory processing architecture Spark is more expensive than that of the Hadoop. So pointing out one as better than the other is not easy. It varies under different circumstances.
