Apache Spark Reviews & Insights

Score9.2 out of 10

161 Reviews and Ratings

Community insights

TrustRadius Insights for Apache Spark are summaries of user sentiment data from TrustRadius reviews and, when necessary, third party data sources.

Pros

Great Computing Engine: Apache Spark is praised by many users for its capabilities in handling complex transformative logic and sophisticated data processing tasks. Several reviewers have mentioned that it is a great computing engine, indicating its effectiveness in solving intricate problems.

Valuable Insights and Analysis: Many reviewers find Apache Spark to be useful for understanding data and performing data analytical work. They appreciate the valuable insights and analysis capabilities provided by the software, suggesting that it helps them gain deeper understanding of their data.

Extensive Set of Libraries and APIs: The extensive set of libraries and APIs offered by Apache Spark has been highly appreciated by users. It provides a wide range of tools and functionalities to solve various day-to-day problems, making it a versatile choice for different data processing needs.

Apache Spark Reviews

10 Reviews

Mid-sized Companies (51-1,000 employees)

Apache Spark: Lightning-Fast Distributed Computing with a Learning Curve

Rating: 10 out of 10

Incentivized

August 18, 2023

Use Cases and Deployment Scope

If you are working on large and big scale data with analytics - don't go further without the use of Apache Spark! One of the projects that I was involved in using Apache Spark was a Recommendation Systems based project. My area or domain of research expertise is also Recommendation Systems. The deployment of a RecSys along with the use of Apache Spark - functionalities like scalability, flexibility of using various data sources along with fault-tolerant systems - are very easy. The built-in machine learning library MLlib is a boon to work. We don't require any other libraries.

Pros

Fault-tolerant systems: in most cases, no node fails. If it fails - the processing still continues.
Scalable to any extent.
Has built-in machine learning library called - MLlib
Very flexible - data from various data sources can be used. Usage with HDFS is very easy

Cons

Its fully not backward compatible.
It is memory-consuming for heavy and large workloads and datasets
Support for advanced analytics is not available - MLlib has minimalistic analytics.
Deployment is a complex task for beginners.

Likelihood to Recommend

Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks.

Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.

Ananth Gouri

Assistant Professor in Engineering at The National Institute of Engineering, Mysuru (Education Management, 501-1000 employees)

Vetted Review

View profile

Epic Computation Engine Framework

Rating: 9 out of 10

Incentivized

November 8, 2020

Use Cases and Deployment Scope

Apache Spark is being used by our organization for writing ETL applications. It enables us to ingest thousands of records of data to database tables.

Pros

Great computing engine for solving complex transformative logic
Useful for understanding data and doing data analytical work
Gives us a great set of libraries and api to solve day-to-day problems

Cons

High learning curve
Complexity
More documentation
More developer support
More educational videos

Likelihood to Recommend

Apache Spark is suited for big data applications when there is a need for performing analysis, streaming data work, and ETL work.

Chetan Munegowda

Software Engineer in Information Technology at SemanticBits (Information Technology and Services, 201-500 employees)

Vetted Review

View profile

Apache Spark -- The best big data solution

Rating: 8 out of 10

Incentivized

January 12, 2020

Use Cases and Deployment Scope

We were working for one of our products, which has a requirement for developing an enterprise-level product catering to manage a vast amount of Big data involved. We wanted to use a technology that is faster than Hadoop and can process large scale data by providing a streamlined process for the data scientists. Apache Spark is a powerful unified solution as we thought to be.
The main problem that we identified in our existing approach was that it was taking a large amount of time to process the data, and also the statistical analysis of the data was not up to the mark. We wanted a sophisticated analytical solution that was easy and fast to use. With using Apache Spark, the processing was made 5 times faster than earlier, giving rise to pretty good analytics. With Spark, across a cluster of machines, the data abstraction was achieved by using RDDs.

Pros

DataFrames, DataSets, and RDDs.
Spark has in-built Machine Learning library which scales and integrates with existing tools.

Cons

The data processing done by Spark comes at a price of memory blockages, as in-memory capabilities of processing can lead to large consumption of memory.
The caching algorithm is not in-built in Spark. We need to manually set up the caching mechanism.

Likelihood to Recommend

1. Suitable where the requirement for advanced analytics is prominent.
2. When you want big data to be processed at a very fast pace.
3. For large datasets, Spark is a viable solution.
4. When you need fault tolerance to be at a precision, go for Spark.

Spark is not suitable:
1. If you want your data to be processed in real-time, then Spark is not a good solution.
2. When you need automatic optimization, then Spark fails at that point.

Yogesh Mhasde

Technical Manager in Information Technology at Rishabh Software Private Limited (Information Technology & Services, 501-1000 employees)

Vetted Review

Want to save dollars, resources and time processing big data, switch to Apache Spark

Rating: 9 out of 10

Incentivized

March 27, 2019

Use Cases and Deployment Scope

We sold a data science product to one of the leading US-based e-commerce firms. Suddenly, their data started growing at a very fast rate. The product, at this stage, was based on R programming. With such huge data, the product started taking a lot of time. We then started thinking of an alternative to R, to process multiplying big data such as this client has. We eventually came across Apache Spark. With the permission of the client, we started switching the codes from R to Apache Spark. It took a very long time to learn and code in Spark, but it was worth the effort. The R codes, which were taking days of time to run, came down to a few hours.

Pros

Very good tool to process big datasets.
Inbuilt fault tolerance.
Supports multiple languages.
Supports advanced analytics.
A large number of libraries available -- GraphX, Spark SQL, Spark Streaming, etc.

Cons

Very slow with smaller amounts of data.
Expensive, as it stores data in memory.

Likelihood to Recommend

If your data is very huge, I recommend converting the underlying technology into Apache Spark. This will save you a lot of time and effort in the near future due to your growing data. The Apache Spark scalability feature also means it handles all the future data related processing.

Verified User

Engineer in Engineering (Management Consulting company, 51-200 employees)

Vetted Review

Very useful application for Big Data processing and excellent for large volume production workflows

Rating: 10 out of 10

Incentivized

August 28, 2018

Use Cases and Deployment Scope

Apache Spark is being used by the whole organization. It helps us a lot in the transmission of data, as it is 100 times faster than Hadoop MapReduce in memory and 10 times faster in disk, as we work with Java this application. It allows native links for Java programming languages, and as it is compatible with SQL, is completely adapted to the needs of our organization, because of the large amount of information that we use. We highly prefer Apache Spark since it supports in-memory processing to increase performance of big data analysis applications.

Pros

It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.
It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times
Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Cons

Increase the information and trainings that come with the application, especially for debugging since the process is difficult to understand.
It should be more attentive to users and make tutorials, to reduce the learning curve.
There should be more grouping algorithms.

Likelihood to Recommend

It is suitable for processing large amounts of data, as it is very easy to use and its syntax is simple and understandable. I also find it useful to use in a variety of applications without the need to integrate many other processing technologies, and it is very fast and has many machine learning algorithms that can be used for data problems. I find it less appropriate for data that is not so large, as it uses too many resources.

Carla Borges

Consultor Tecnico - Java Developer and Php Developer. in Engineering at Consultec-TI (Computer Software, 51-200 employees)

Vetted Review

View profile

Apache Spark, the be all End All.

Rating: 9 out of 10

Incentivized

March 27, 2018

Use Cases and Deployment Scope

Spark was/is being used in myriad of ways. With Kafka, using Spark Streams to grab data from kafka queue into our hdfs environment. SparkSQL used for analysis of data for those not familiar with spark. Using Spark for data analysis as well and for main workflow process. Using spark over mapreduce. Using Spark for some machine learning algo's with the data.

Pros

Machine Learning.
Data Analysis
WorkFlow process (faster than MapReduce).
SQL connector to multiple data sources

Cons

Memory management. Very weak on that.
PySpark not as robust as scala with spark.
spark master HA is needed. Not as HA as it should be.
Locality should not be a necessity, but does help improvement. But would prefer no locality

Likelihood to Recommend

Spark is great as a workflow process and extract transform layer process tool. Is really good for machine learning especially for large datasets that can be processed in split file paralallelization.
Spark streaming is scalable for close to real-time data workflow process.
what it's not good for, is smaller subset of data processing.

Anson Abraham

Data Czar in Engineering at Envisagenics, Inc. (Marketing and Advertising, 51-200 employees)

Vetted Review

View profile

Use Apache Spark to Speed Up Cluster Computing

Rating: 7 out of 10

Incentivized

January 23, 2018

Use Cases and Deployment Scope

In our company, we used Spark for a healthcare analytical project, where we need to do large-scale data processing in a Hadoop environment. The project is about building an enterprise data lake where we bring data from multiple products and consolidate. Further, in the downstream, we will develop some business reports.

Pros

We used to make our batch processing faster. Spark is faster in batch processing than MapReduce with it in memory computing
Spark will run along with other tools in the Hadoop ecosystem including Hive and Pig
Spark supports both batch and real-time processing
Apache Spark has Machine Learning Algorithms support

Cons

Consumes more memory
Difficult to address issues around memory utilization
Expensive - In-memory processing is expensive when we look for a cost-efficient processing of big data

Likelihood to Recommend

Well suited:
1. Data can be integrated from several sources including click stream, logs, transactional systems
2. Real-time ingestion through Kafka, Kinesis, and other streaming platforms

Verified User

Employee in Engineering (Hospital & Health Care company, 501-1000 employees)

Vetted Review

Apache Spark Should Spark Your Interest

Rating: 9 out of 10

Incentivized

December 13, 2017

Use Cases and Deployment Scope

At my current company, we are using Spark in a variety of ways ranging from batch processing to data analysis to machine learning techniques. It has become our main driver for any distributed processing applications. It has gained quick adoption across the organization for its ease of use, integration into the Hadoop stack, and for its support in a variety of languages.

Pros

Ease of use, the Spark API allows for minimal boilerplate and can be written in a variety of languages including Python, Scala, and Java.
Performance, for most applications we have found that jobs are more performant running via Spark than other distributed processing technologies like Map-Reduce, Hive, and Pig.
Flexibility, the frameworks comes with support for streaming, batch processing, sql queries, machine learning, etc. It can be used in a variety of applications without needing to integrate a lot of other distributed processing technologies.

Cons

Resource heavy, jobs, in general, can be very memory intensive and you will want the nodes in your cluster to reflect that.
Debugging, it has gotten better with every release but sometimes it can be difficult to debug an error due to ambiguous or misleading exceptions and stack traces.

Likelihood to Recommend

If you are running a distributed environment and are running applications that make use of batch processing, analytics, streaming, machine learning, or graphing then I cannot recommend Spark enough. It is easy to get going, simple to learn (relative to similar technologies), and can be used in a variety of use cases. All while giving you great performance.

Verified User

Engineer in Engineering (Computer Software company, 51-200 employees)

Vetted Review

Sparkling Spark

Rating: 10 out of 10

Incentivized

June 26, 2017

Use Cases and Deployment Scope

It's being replaced as the traditional ETL tool and we are using Apache Spark for data science solutions.

Pros

It makes the ETL process very simple when compared to SQL SERVER and MYSQL ETL tools.
It's very fast and has many machine learning algorithms which can be used for data science problems.
It is easily implemented on a cloud cluster.

Cons

The initialization and spark context procedures.
Running applications on a cluster is not well documented anywhere, some applications are hard to debug.
Debugging and Testing are sometimes time-consuming.

Likelihood to Recommend

It's well suited for ETL, data Integration, and data science problems of large data sets. It's not at all suitable for small data sets which can be done on desktops and laptops using the Python tool.

Sunil Dhage

Big Data Analyst in Information Technology at PSL Group (Pharmaceuticals, 51-200 employees)

Vetted Review

View profile

A useful replacement for MapReduce for Big Data processing

Rating: 8 out of 10

Incentivized

September 12, 2016

Use Cases and Deployment Scope

We are learning core Apache Spark + SparkSQL and MLLib, while creating proof-of-concepts as well as providing solutions for clients. It addresses the needs of quickly processing large amounts of data, typically located in Hadoop.

Pros

Scale from local machine to full cluster. You can run a standalone, single cluster simply by starting up a Spark Shell or submitting an application to test an algorithm, then it quickly can be transferred and configured to run in a distributed environment.
Provides multiple APIs. Most people I know use Python and/or Java as their main programming language. Data scientists who are familiar with NumPy and SciPy can quickly become comfortable with Spark, while Java developers would best served using Java 8 and the new features that it provides. Scala, on the other hand, is a mix between the Java and Python styles of writing Spark code, in my opinion.
Plentiful learning resources. The Learning Spark book is a good introduction to the mechanics of Spark although written for Spark 1.3, and the current version is 2.0. The GitHub repository for the book contains all the code examples that are discussed, plus the Spark website is also filled with useful information that is simple to navigate.

Cons

For data that isn't truly that large, Spark may be overkill when the problem could likely be solved on a computer with reasonable hardware resources. There doesn't seem to be a lot of examples for how a Spark task would otherwise be implemented in a different library; for instance scikit-learn and NumPy rather than Spark MLlib.

Likelihood to Recommend

On the plus side, Spark is a good tool to learn to apply to various data processing problems.

As described in the Cons - Spark may not be needed unless there is truly a large amount of data to operate on. Other libraries may be better suited for the same task.

Jordan Moore

Staff Consultant in Information Technology at Avalon Consulting, LLC (Information Technology and Services, 51-200 employees)

Vetted Review

View profile

Loading Reviews List....

Very useful application for Big Data processing and excellent for large volume production workflows

Rating: 10 out of 10

Incentivized

August 28, 2018

Use Cases and Deployment Scope

Pros

It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.
It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times
Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Cons

Increase the information and trainings that come with the application, especially for debugging since the process is difficult to understand.
It should be more attentive to users and make tutorials, to reduce the learning curve.
There should be more grouping algorithms.

Likelihood to Recommend

Carla Borges

Consultor Tecnico - Java Developer and Php Developer. in Engineering at Consultec-TI (Computer Software, 51-200 employees)

Vetted Review

View profile