Apache Spark Professional, Scientific, and Technical Services Reviews & Insights

Score9.2 out of 10

161 Reviews and Ratings

Community insights

TrustRadius Insights for Apache Spark are summaries of user sentiment data from TrustRadius reviews and, when necessary, third party data sources.

Pros

Great Computing Engine: Apache Spark is praised by many users for its capabilities in handling complex transformative logic and sophisticated data processing tasks. Several reviewers have mentioned that it is a great computing engine, indicating its effectiveness in solving intricate problems.

Valuable Insights and Analysis: Many reviewers find Apache Spark to be useful for understanding data and performing data analytical work. They appreciate the valuable insights and analysis capabilities provided by the software, suggesting that it helps them gain deeper understanding of their data.

Extensive Set of Libraries and APIs: The extensive set of libraries and APIs offered by Apache Spark has been highly appreciated by users. It provides a wide range of tools and functionalities to solve various day-to-day problems, making it a versatile choice for different data processing needs.

Apache Spark Reviews

7 Reviews

Professional, Scientific, and Technical ServicesInformation Technology & Services5Management Consulting1Marketing & Advertising1

Lightning Fast In-Memory Cluster Computing Framework

Rating: 10 out of 10

Incentivized

August 30, 2022

Use Cases and Deployment Scope

Earlier we were using RDBMS like Oracle for retail and eCommerce data. We faced challenges such as cost, performance, and a huge amount of transactions coming in. After a lot of critical issues we migrated to delta lake. Now, we are using Apache Spark Streaming to deal with all real-time transactions. For batch data as well, we are pretty much handling TBs of data using Apache Spark.

Pros

Realtime data processing
Interactive Analysis of data
Trigger Event Detection

Cons

Machine Learning
GraphX Lib
True Realtime Streaming

Likelihood to Recommend

Well suited for batch processing and provides performance improvement through optimization techniques. Data Streaming is getting better with Apache Spark Structured Streaming. Out of memory issues and Data Skewness problems when data is not properly organized. Integration with BI tools such as Tableau could be better.

Riyaz Khan

Staff Engineer in Information Technology at Nagarro (Information Technology & Services, 10,001+ employees)

Vetted Review

View profile

Epic Computation Engine Framework

Rating: 9 out of 10

Incentivized

November 8, 2020

Use Cases and Deployment Scope

Apache Spark is being used by our organization for writing ETL applications. It enables us to ingest thousands of records of data to database tables.

Pros

Great computing engine for solving complex transformative logic
Useful for understanding data and doing data analytical work
Gives us a great set of libraries and api to solve day-to-day problems

Cons

High learning curve
Complexity
More documentation
More developer support
More educational videos

Likelihood to Recommend

Apache Spark is suited for big data applications when there is a need for performing analysis, streaming data work, and ETL work.

Chetan Munegowda

Software Engineer in Information Technology at SemanticBits (Information Technology and Services, 201-500 employees)

Vetted Review

View profile

A powerhouse processing engine.

Rating: 9 out of 10

Incentivized

September 19, 2020

Use Cases and Deployment Scope

We use Apache Spark for cluster computing in large-scale data processing, ETL functions, machine learning, as well as for analytics. Its primarily used by the Data Engineering Department, in order to support the data lake infrastructure. It helps us to effectively manage the great amounts of data that come from our clusters, ensuring the capacity, scalability, and performance needed.

Pros

Speed: Apache Spark has great performance for both streaming and batch data
Easy to use: the object oriented operators make it easy and intuitive.
Multiple language support
Fault tolerance
Cluster managment
Supports DF, DS, and RDDs

Cons

Hard to learn, documentation could be more in-depth.
Due to it's in-memory processing, it can take a large consumption of memory.
Poor data visualization, too basic.

Likelihood to Recommend

Well suited for: large datasets, fault tolerance, parallel processing, ETL, batch processing, streaming, analytics, graphing, or machine learning. Mostly any kind of large-scale processing, since it will save you a lot of time (days of processing). Less appropriate for: smaller datasets, you are better off using pandas or other libraries.

Verified User

Engineer in Information Technology (Information Technology & Services company, 11-50 employees)

Vetted Review

Apache Spark -- The best big data solution

Rating: 8 out of 10

Incentivized

January 12, 2020

Use Cases and Deployment Scope

We were working for one of our products, which has a requirement for developing an enterprise-level product catering to manage a vast amount of Big data involved. We wanted to use a technology that is faster than Hadoop and can process large scale data by providing a streamlined process for the data scientists. Apache Spark is a powerful unified solution as we thought to be.
The main problem that we identified in our existing approach was that it was taking a large amount of time to process the data, and also the statistical analysis of the data was not up to the mark. We wanted a sophisticated analytical solution that was easy and fast to use. With using Apache Spark, the processing was made 5 times faster than earlier, giving rise to pretty good analytics. With Spark, across a cluster of machines, the data abstraction was achieved by using RDDs.

Pros

DataFrames, DataSets, and RDDs.
Spark has in-built Machine Learning library which scales and integrates with existing tools.

Cons

The data processing done by Spark comes at a price of memory blockages, as in-memory capabilities of processing can lead to large consumption of memory.
The caching algorithm is not in-built in Spark. We need to manually set up the caching mechanism.

Likelihood to Recommend

1. Suitable where the requirement for advanced analytics is prominent.
2. When you want big data to be processed at a very fast pace.
3. For large datasets, Spark is a viable solution.
4. When you need fault tolerance to be at a precision, go for Spark.

Spark is not suitable:
1. If you want your data to be processed in real-time, then Spark is not a good solution.
2. When you need automatic optimization, then Spark fails at that point.

Yogesh Mhasde

Technical Manager in Information Technology at Rishabh Software Private Limited (Information Technology & Services, 501-1000 employees)

Vetted Review

Want to save dollars, resources and time processing big data, switch to Apache Spark

Rating: 9 out of 10

Incentivized

March 27, 2019

Use Cases and Deployment Scope

We sold a data science product to one of the leading US-based e-commerce firms. Suddenly, their data started growing at a very fast rate. The product, at this stage, was based on R programming. With such huge data, the product started taking a lot of time. We then started thinking of an alternative to R, to process multiplying big data such as this client has. We eventually came across Apache Spark. With the permission of the client, we started switching the codes from R to Apache Spark. It took a very long time to learn and code in Spark, but it was worth the effort. The R codes, which were taking days of time to run, came down to a few hours.

Pros

Very good tool to process big datasets.
Inbuilt fault tolerance.
Supports multiple languages.
Supports advanced analytics.
A large number of libraries available -- GraphX, Spark SQL, Spark Streaming, etc.

Cons

Very slow with smaller amounts of data.
Expensive, as it stores data in memory.

Likelihood to Recommend

If your data is very huge, I recommend converting the underlying technology into Apache Spark. This will save you a lot of time and effort in the near future due to your growing data. The Apache Spark scalability feature also means it handles all the future data related processing.

Verified User

Engineer in Engineering (Management Consulting company, 51-200 employees)

Vetted Review

Apache Spark, the be all End All.

Rating: 9 out of 10

Incentivized

March 27, 2018

Use Cases and Deployment Scope

Spark was/is being used in myriad of ways. With Kafka, using Spark Streams to grab data from kafka queue into our hdfs environment. SparkSQL used for analysis of data for those not familiar with spark. Using Spark for data analysis as well and for main workflow process. Using spark over mapreduce. Using Spark for some machine learning algo's with the data.

Pros

Machine Learning.
Data Analysis
WorkFlow process (faster than MapReduce).
SQL connector to multiple data sources

Cons

Memory management. Very weak on that.
PySpark not as robust as scala with spark.
spark master HA is needed. Not as HA as it should be.
Locality should not be a necessity, but does help improvement. But would prefer no locality

Likelihood to Recommend

Spark is great as a workflow process and extract transform layer process tool. Is really good for machine learning especially for large datasets that can be processed in split file paralallelization.
Spark streaming is scalable for close to real-time data workflow process.
what it's not good for, is smaller subset of data processing.

Anson Abraham

Data Czar in Engineering at Envisagenics, Inc. (Marketing and Advertising, 51-200 employees)

Vetted Review

View profile

A useful replacement for MapReduce for Big Data processing

Rating: 8 out of 10

Incentivized

September 12, 2016

Use Cases and Deployment Scope

We are learning core Apache Spark + SparkSQL and MLLib, while creating proof-of-concepts as well as providing solutions for clients. It addresses the needs of quickly processing large amounts of data, typically located in Hadoop.

Pros

Scale from local machine to full cluster. You can run a standalone, single cluster simply by starting up a Spark Shell or submitting an application to test an algorithm, then it quickly can be transferred and configured to run in a distributed environment.
Provides multiple APIs. Most people I know use Python and/or Java as their main programming language. Data scientists who are familiar with NumPy and SciPy can quickly become comfortable with Spark, while Java developers would best served using Java 8 and the new features that it provides. Scala, on the other hand, is a mix between the Java and Python styles of writing Spark code, in my opinion.
Plentiful learning resources. The Learning Spark book is a good introduction to the mechanics of Spark although written for Spark 1.3, and the current version is 2.0. The GitHub repository for the book contains all the code examples that are discussed, plus the Spark website is also filled with useful information that is simple to navigate.

Cons

For data that isn't truly that large, Spark may be overkill when the problem could likely be solved on a computer with reasonable hardware resources. There doesn't seem to be a lot of examples for how a Spark task would otherwise be implemented in a different library; for instance scikit-learn and NumPy rather than Spark MLlib.

Likelihood to Recommend

On the plus side, Spark is a good tool to learn to apply to various data processing problems.

As described in the Cons - Spark may not be needed unless there is truly a large amount of data to operate on. Other libraries may be better suited for the same task.

Jordan Moore

Staff Consultant in Information Technology at Avalon Consulting, LLC (Information Technology and Services, 51-200 employees)

Vetted Review

View profile

Loading Reviews List....

A useful replacement for MapReduce for Big Data processing

Rating: 8 out of 10

Incentivized

September 12, 2016

Use Cases and Deployment Scope

Pros

Scale from local machine to full cluster. You can run a standalone, single cluster simply by starting up a Spark Shell or submitting an application to test an algorithm, then it quickly can be transferred and configured to run in a distributed environment.
Provides multiple APIs. Most people I know use Python and/or Java as their main programming language. Data scientists who are familiar with NumPy and SciPy can quickly become comfortable with Spark, while Java developers would best served using Java 8 and the new features that it provides. Scala, on the other hand, is a mix between the Java and Python styles of writing Spark code, in my opinion.
Plentiful learning resources. The Learning Spark book is a good introduction to the mechanics of Spark although written for Spark 1.3, and the current version is 2.0. The GitHub repository for the book contains all the code examples that are discussed, plus the Spark website is also filled with useful information that is simple to navigate.

Cons

For data that isn't truly that large, Spark may be overkill when the problem could likely be solved on a computer with reasonable hardware resources. There doesn't seem to be a lot of examples for how a Spark task would otherwise be implemented in a different library; for instance scikit-learn and NumPy rather than Spark MLlib.

Likelihood to Recommend

Jordan Moore

Staff Consultant in Information Technology at Avalon Consulting, LLC (Information Technology and Services, 51-200 employees)

Vetted Review

View profile