Apache Spark vs. Informatica PowerCenter (legacy)

Apache Spark has rich APIs for regular data transformations or for ML workloads or for graph workloads, whereas other systems may not such a wide range of support. Choose it when you need to perform data transformations for big data as offline jobs, whereas use MongoDB-like distributed database systems for more realtime queries.

Incentivized

Informatica Powercenter is the centerpiece of our overall enterprise data warehouse strategy. It's a critical enablement to ensure we can feed in multiple data stream and transform them into digestible data within our data warehouse. With its flexible capabilities and API availability, we were able to feed in industry standard data format as well as home grown data structure. Overall, we are very pleased with their capability and contribution to our data warehouse strategy.

Incentivized

• It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.
• It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times
• Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Incentivized

• Informatica has a wide range of support for databases. Pretty much every mainstream DBMS is compatible here.
• Designing ETL mappings and workflows is a very intuitive process, and takes minimal learning time and effort even for a beginner.
• Informatica's biggest strength is its sheer performance. It is unmatched in terms of handling large volumes of data.

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• One of the challenges of PowerCenter is the lack of integration between the components and functionality provided by PowerCenter. PowerCenter consists of multiple components such has the repository service, integration service, metadata service. Considerable time and resources were required to install and configure these components before PowerCenter was available for use.
• In order to connect to various data sources such as Netezza database or SAS datasets, PowerCenter requires the installation and configuration of separate plug-ins. We spent considerable time trouble-shooting and debugging problems while trying to get the various plug-ins integrated with PowerCenter and get them up and running as described in the documentation.
• PowerCenter works well with structured data. That is, it is easy to work with input and output data that is pre-defined, fixed, and unchanging. It is much more difficult to work with dynamic data in which new fields are added or removed ad-hoc or if data format changes during the data ingest process. We have not been as successful in using PowerCenter for dynamic data.
• One of the challenges of learning PowerCenter is that it is difficult to find documentation or publications that help you learn the various details about PowerCenter software. Unlike SAS Institute, Informatica does not publish books about PowerCenter. The documentation available with PowerCenter is sparse; we have learned many aspects of this technology through trial and error.

Incentivized

Capacity of computing data in cluster and fast speed.

Our team enjoys using Informatica and feels that it is one of the best ETL tools on the market.

If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used

Incentivized

The tool is very flexible and will meet most, if not all, of your data transformation needs. It is an expert-level tool, so building your knowledge-base and user-base (and keeping that base healthy!) is very important. But it will pay off with strong data management and the ability to leverage that data in ways you haven’t thought of yet. Bottom line, data is money, and PowerCenter helps you monetize your data.

Incentivized

Positives; - Multi-user development environment. - The speed of transformation. - Seamless integration with other Informatica products. Negatives; - There should be fewer windows, to maintain developers' focus while using. You probably need two big monitors when you start development with Informatica Power Center. - Oracle Analytical functions should be natively used. - E-LT support as well as ETL support.

Incentivized

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

Informatica power center is a leader of the pack of ETL tools and has some great abilities that make it stand out from other ETL tools. It has been a great partner to its clients over a long time so it's definitely dependable. With all the great things about Informatica, it has a bit of tech burden that should be addressed to make it more nimble, reduce the learning curve for new developers, provide better connectivity with visualization tools.

Incentivized

We used Surprise Kit for one of the other research works. It is more fine-tuned to Recommendation systems and their algorithms. Apache Spark has MLlib for majority of ML problems. Where as software like Surprse Kit - it suitable for a specific task of Recommendations only

Incentivized

Basically the two solutions have, more or less, the same functions and features.The difference, for me, is that ThreatQuotient make more features over the security and I think is oriented to a SOC enviroments. InformaticaExchange Connectors is oriented to the quality, integration and distribution of the data in order to ensure the reliability and access of data from different sources, as well as the integration in a single repository of enterprise data (External/internal)

Incentivized

• Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
• Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
• Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.

Incentivized

• PowerCenter has been instrumental in being the center of all data movement within the organization.
• It has also provided a foundation for which re-usability and scalability are the focus.
• Finding talent with experience and expertise in PowerCenter is far more likely due to its presence and market share.

Incentivized