A powerhouse processing engine.
Rating: 9 out of 10
IncentivizedUse Cases and Deployment Scope
We use Apache Spark for cluster computing in large-scale data processing, ETL functions, machine learning, as well as for analytics. Its primarily used by the Data Engineering Department, in order to support the data lake infrastructure. It helps us to effectively manage the great amounts of data that come from our clusters, ensuring the capacity, scalability, and performance needed.
Pros
- Speed: Apache Spark has great performance for both streaming and batch data
- Easy to use: the object oriented operators make it easy and intuitive.
- Multiple language support
- Fault tolerance
- Cluster managment
- Supports DF, DS, and RDDs
Cons
- Hard to learn, documentation could be more in-depth.
- Due to it's in-memory processing, it can take a large consumption of memory.
- Poor data visualization, too basic.
Likelihood to Recommend
Well suited for: large datasets, fault tolerance, parallel processing, ETL, batch processing, streaming, analytics, graphing, or machine learning. Mostly any kind of large-scale processing, since it will save you a lot of time (days of processing). Less appropriate for: smaller datasets, you are better off using pandas or other libraries.