Apache Spark Reviews & Insights

Score9.2 out of 10

161 Reviews and Ratings

Community insights

TrustRadius Insights for Apache Spark are summaries of user sentiment data from TrustRadius reviews and, when necessary, third party data sources.

Pros

Great Computing Engine: Apache Spark is praised by many users for its capabilities in handling complex transformative logic and sophisticated data processing tasks. Several reviewers have mentioned that it is a great computing engine, indicating its effectiveness in solving intricate problems.

Valuable Insights and Analysis: Many reviewers find Apache Spark to be useful for understanding data and performing data analytical work. They appreciate the valuable insights and analysis capabilities provided by the software, suggesting that it helps them gain deeper understanding of their data.

Extensive Set of Libraries and APIs: The extensive set of libraries and APIs offered by Apache Spark has been highly appreciated by users. It provides a wide range of tools and functionalities to solve various day-to-day problems, making it a versatile choice for different data processing needs.

Apache Spark Reviews

13 Reviews

Enterprises (1,001+ employees)

Apache Spark is still a valid DE tool

Rating: 9 out of 10

Incentivized

December 28, 2024

Use Cases and Deployment Scope

We use Apache Spark on a daily basis as the main computation engine for updating most critical and non-critical data pipelines. We mostly work with batch processing but there are instances for using Spark Streaming as well. The scope is for all analysis pipelines, machine learning datasets and several operational use cases.

Pros

Parallel processing
Configurability
Usage with other tools

Cons

More ready-to-use solutions for tweaking the Apache Spark configs
Reduce the creation of UDFs for Pyspark by implementing transformations directly

Likelihood to Recommend

Based on my personal experience, Apache Spark is great when you have the need for highly parallelized jobs and have the time and resources to adapt the configurations for your jobs: for this reason I would not recommend it for companies that do not have a strong group of data engineers that can support other data roles to process data in their company.

Verified User

Employee in Engineering (Consumer Services company, 1001-5000 employees)

Vetted Review

Lightning Fast In-Memory Cluster Computing Framework

Rating: 10 out of 10

Incentivized

August 30, 2022

Use Cases and Deployment Scope

Earlier we were using RDBMS like Oracle for retail and eCommerce data. We faced challenges such as cost, performance, and a huge amount of transactions coming in. After a lot of critical issues we migrated to delta lake. Now, we are using Apache Spark Streaming to deal with all real-time transactions. For batch data as well, we are pretty much handling TBs of data using Apache Spark.

Pros

Realtime data processing
Interactive Analysis of data
Trigger Event Detection

Cons

Machine Learning
GraphX Lib
True Realtime Streaming

Likelihood to Recommend

Well suited for batch processing and provides performance improvement through optimization techniques. Data Streaming is getting better with Apache Spark Structured Streaming. Out of memory issues and Data Skewness problems when data is not properly organized. Integration with BI tools such as Tableau could be better.

Riyaz Khan

Staff Engineer in Information Technology at Nagarro (Information Technology & Services, 10,001+ employees)

Vetted Review

View profile

Apache Spark is the next generation of big data computing.

Rating: 9 out of 10

Incentivized

April 18, 2022

Use Cases and Deployment Scope

We need to calculate risk-weighted assets (RWA) daily and monthly for different positions the bank holds on a T+1 basis. The volume of calculations is large: more than millions of records per day with very complicated formulas and algorithms. In our applications/projects, we used Scala and Apache Spark clusters to load all data we needed for calculation and implemented complicated formulas and algorithms via its DataFrame or DataSet from the Apache Spark platform.

Without adopting the Apache Spark cluster, it would be pretty hard for us to implement such a big system to handle a large volume of data calculations daily. After this system was successfully deployed into PROD, we've been able to provide capital risk control reports to regulation/compliance controllers in different regions in this global financial world.

Pros

DataFrame as a distributed collection of data: easy for developers to implement algorithms and formulas.
Calculation in-memory.
Cluster to distribute large data of calculation.

Cons

It would be great if Apache Spark could provide a native database to manage all file info of saved parquet.

Likelihood to Recommend

For a large volume of data to be calculated, Apache Spark is the go-to; for intermediate or small volumes of data sets, Apache Spark is an option.

Steven Li

Senior Software Developer (Consultant) in Information Technology at Morgan Stanley (Banking, 10,001+ employees)

Vetted Review

View profile

good solution for long and narrow data

Rating: 9 out of 10

Incentivized

May 20, 2021

Use Cases and Deployment Scope

We are building a model and due to the size of the data, we chose to use Apache Spark for the feature generation. The usage of the tool is limited within my department and one another department. The two departments need to deal with long dataset and the other departments does not need that.

Pros

quick
utilized CPU cores
trendy

Cons

lack of support
memory hungry
slow on wide data

Likelihood to Recommend

I would recommend Apache Spark to the colleague if that person is working with long but narrow dataset. This would be a great tool to help the person fully utilize the CPU cores and speed up the work process. However, I would not recommend this tool if the dataset is wide not not very large.

Verified User

Analyst in Professional Services (Financial Services company, 10,001+ employees)

Vetted Review

Apache Spark - your go to technology for distributed data processing

Rating: 9 out of 10

Incentivized

May 3, 2021

Use Cases and Deployment Scope

We are using Apache Spark in Digital - Data teams to build data products and help business teams to take data-driven decisions.
We use Apache Spark to source that from different source systems, process it, and store it in the data lake.
Once the data is in data lake, we use spark for data cleansing and data transformation as per business requirements
Once the data is transformed, then we will insert it into the final target layer in the data warehouse.

Pros

Spark is very fast compered to other frameworks because it works in cluster mode and use distributed processing and computation frameworks internally
Robust and fault tolerant
Open source
Can source data from multiple data sources

Cons

No Dataset API support in python version of spark
Apache Spark job run UI can have more meaningful information
Spark errors can provide more meaningful information when a job is failed

Likelihood to Recommend

Specific scenarios where Apache Spark is well suited:
1. real-time processing of streaming data
2. processing unstructured data, semi-structured data, and structured data from multiple sources
3. avoid vendor lock-in and cloud platform lock-in while developing products

Surendranatha Reddy Chappidi

Senior Data Engineer in Information Technology at A.P. Moller - Maersk (Logistics & Supply Chain, 10,001+ employees)

Vetted Review

View profile

Apache Spark in Telco

Rating: 10 out of 10

Incentivized

April 30, 2021

Use Cases and Deployment Scope

Apache Spark is being widely used within the company. In Advanced Analytics department data engineers and data scientists work closely in machine learning projects to generate value. Spark provides unified big data analytics engine which helps us easily process huge amount of data. We are using Spark in projects like churn prediction, network analytics.

Pros

Machine learning on big data
Stream processing
Lakehouse with Delta

Cons

Indexing
Mllib
Streaming

Likelihood to Recommend

Apache Spark is very good for prosessing large amount of data but not that good if you need many joins or low latency. With combination of delta engine performance improved alot. Especially having ACID support, time travel features and consistent view for simultaneous read and writes it’s now ready for next level.

Verified User

Engineer in Information Technology (Telecommunications company, 10,001+ employees)

Vetted Review

Great open source tool for data processing

Rating: 9 out of 10

Incentivized

December 13, 2019

Use Cases and Deployment Scope

We do use Apache Spark for cluster computing for our ETL environment, data and analytics as well as machine learning. It is mainly used by our data engineering team to support the entire Data Lake foundation. As we have huge amounts of information coming from multiple sources, we needed an effective cluster management system to handle capacity and deliver the performance and throughput we needed.

Pros

Cluster management for ETL.
Data processing engine for our data lake.

Cons

You still need Hive or other HDFS to store information.
Security is behind compared to MapReduce.

Likelihood to Recommend

Spark is a one-size-fits-all data processing platform. You can run batch and in-motion streams, you can use for ETL, machine learning or even graphs. You do not have multiple tools, so it makes your TCO and management tasks way easier. As every new platform, has room to grow: storage and security are the main opportunities we found.

Verified User

Executive in Information Technology (Consumer Goods company, 10,001+ employees)

Vetted Review

Apache Spark Review

Rating: 7 out of 10

Incentivized

February 27, 2019

Use Cases and Deployment Scope

We used Apache Spark within our department as a Solution Architecture team. It helped make big data processing more efficient since the same framework can be used for batch and stream processing.

Pros

Customizable, it integrates with Jupyter notebooks which was really helpful for our team.
Easy to use and implement.
It allows us to quickly build microservices.

Cons

Release cycles can be faster.
Sometimes it kicked some of the users out due to inactivity.

Likelihood to Recommend

It is beneficial to use Apache Spark if:

You are working with big data, preprocessing data before machine learning
Building simple microservices and creating PoC. It makes it easier to create REST and simple web APIs.
If you need great customer service, Apache Spark would be a great choice since they provide it 24/7.

Verified User

Analyst in Information Technology (Computer Networking company, 1001-5000 employees)

Vetted Review

Apache Spark - defacto for big data processing/analytics

Rating: 9 out of 10

Incentivized

December 14, 2018

Use Cases and Deployment Scope

Used as the in memory data engine for big data analytics, streaming data and SQL workloads. Also, in the process of trying it out for certain machine learning algorithms. It basically processes data for analytical needs of the business and is a great tool to co-exist with the hadoop file systems.

Pros

in memory data engine and hence faster processing
does well to lay on top of hadoop file system for big data analytics
very good tool for streaming data

Cons

could do a better job for analytics dashboards to provide insights on a data stream and hence not have to rely on data visualization tools along with spark
also there is room for improvement in the area of data discovery

Likelihood to Recommend

Apache Spark is very well suited for big data analytics in conjunction with the hadoop file system and also does a good job of providing fast access to data in SQL workloads since it has an in memory data processing engine that can very quickly process data. In addition, it can also be used for streaming data processing.

Shiv Shivakumar

Acquisitions Leader in Information Technology at Abbott (Computer Software, 1001-5000 employees)

Vetted Review

View profile

Apache Spark: One stop shop for distributed data processing, machine learning and graph processing

Rating: 10 out of 10

Incentivized

July 20, 2018

Use Cases and Deployment Scope

We use Apache Spark across all analytics departments in the company. We primarily use it for distributed data processing and data preparation for machine learning models. We also use it while running distributed CRON jobs for various analytical workloads. I am familiar with a story where we contributed an algorithm to Spark open source which is on Random Walks in Large Graphs - https://databricks.com/session/random-walks-on-large-scale-graphs-with-apache-spark

Pros

Rich APIs for data transformation making for very each to transform and prepare data in a distributed environment without worrying about memory issues
Faster in execution times compare to Hadoop and PIG Latin
Easy SQL interface to the same data set for people who are comfortable to explore data in a declarative manner
Interoperability between SQL and Scala / Python style of munging data

Cons

Documentation could be better as I usually end up going to other sites / blogs to understand the concepts better
More APIs are to be ported to MLlib as only very few algorithms are available at least in clustering segment

Likelihood to Recommend

Apache Spark has rich APIs for regular data transformations or for ML workloads or for graph workloads, whereas other systems may not such a wide range of support. Choose it when you need to perform data transformations for big data as offline jobs, whereas use MongoDB-like distributed database systems for more realtime queries.

Nitin Pasumarthy

Software Engineer in Engineering at LinkedIn (Internet, 5001-10,000 employees)

Vetted Review

View profile

Loading Reviews List....