Apache Airflow vs. Pentaho

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

Pentaho is very well suited to perform data extraction & data mining from various cloud storage & transform that data using various available data models. However, the software struggles when it comes to visualizing the extracted data in an appealing manner & can be difficult for end-users to get an understanding of data tables created using those models.

Incentivized

• Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
• Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
• Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.

Incentivized

• Integrate and synchronize with big data easily
• Import data from any sources and different databases
• Managing data in on-premise, hybrid and cloud environments.
• Compatibility and flexibility of the platform with any type of scenario and any business or industry
• Various tools in the software suite to transformation of data
• Simple interface appearance and creative UI graphics

• A local "dry run" or IDE plugin that can validate and simulate DAG execution without needing a full environment.
• Better feedback on DAG parse errors in the UI or CLI.
• Navigating large DAGs with hundreds of tasks can be slow and hard to understand visually.

Incentivized

• I think the relative obscurity of the tool is a downside, not as many developers, consultants or peers you can tap into.
• Lack of a solid user community held us back, looking at Power BI and Qlik, they have huge user communities that help each other out. Would have liked that here.
• Smaller company means smaller sales force, and the lack of a local presence made it hard to only interact online with the account rep. Other companies have someone local who often stops by with pre-sales developers to just pitch in free of charge when they have time.

Incentivized

I will use Pentaho until I find a better tool with a better, easier to use report designer client. For now, Pentaho has been the most powerful reporting tool for our clients because of its ability to connect to Odoo, integrate in Odoo (reports are accessible in Odoo) and the flexibility in report design and parameter integration

Incentivized

For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.

Incentivized

Even if Pentaho requires less technical skills to develop a pipeline or ETL project, its learning curve can be a bit slow since there are many ways to do the same thing as in any other platform. However, in Pentaho, some things can be confusing some moments for non-technical teams.

Incentivized

We are an Enterprise customer. They handle problems INSTANTLY when they are critical, including initiation an immediate WebEx screen share call when needed. Smaller/less-critical problems are handled within 1-2 days -- and NEVER fall off their radar, no matter how small. As needed, we can also leverage "professional services" from them -- much of which is included in our Enterprise contract. Finally, when a problem I have discovered turns out to be a bug..they create a JIRA for the fix, and make me a watcher. I love seeing notes come in showing me status updates of bugs filed because of something I found. They really are TOP-NOTCH.

Course Taken: DI1000 Pentaho Data Integration Fundamentals Setup A week before your class started, the instructor will start sending out class material and lab setup instructions. This is helpful so that you understand how the environment is laid out and can start reviewing the content. Ultimately it saved about a 1/2 day trying to setup with 10 other people online which was great! The Course The 3-day course was laid out like many other technical classes with 15-30 minutes instruction and 15-60 minutes of lab exercises. The instructor was very knowledgeable with the functionality from version to version and answered questions as we went along. I was amazed at some of the functionality that was available that I was not using at the time and quickly implemented changes to many existing transformations and jobs. The novice users seemed to catch on quickly and more experienced users explained how some of the functionality was used in their home environments. Towards the end there was enough time so that we were able to ask very directed questions about our own environments. Overall, I really found the class to be informative and deliver enough information to be dangerous. My skills improved and I was able to design better and efficient transformations for the HIE. Course Description: https://training.pentaho.com/instructor-led-training/pentaho-data-integration-fundamentals-di1000

Get the right people in before starting implementation. Start small and build as you go approach is time consuming and involves lot of rework. Evangalize within the organization the capabilities and limitations equally so that correct delivery expectations are set. Set expectations with the Customer that the tool cannot replace proprietary software in terms of stability/usability and that timelines could change given the new ness of the product.

Incentivized

Apache Airflow is suited for a much wider set of use cases compared to Databricks. You can run it anywhere, and there is also no vendor lock-in. With Airflow, we can utilize almost any compute engine. Same thing we want to do with Databricks. There might be some level of difficulty based on the support.

Incentivized

Perhaps Snowflake and SalesForce have some components which align with the Pentaho tools. The Pentaho tools have integrations with these technologies to add more value to the final users. Perhaps the only weakness I can honestly find in the Pentaho tools right now is the lack of a powerful web interface for data transformations. There is a web component from which you can access existing data transformations created with the Pentaho Data Integration tool. Still, the web component only allows visualization of the data transformation and remote execution. A complete web interface with remote execution would be excellent, and I'm sure that we might see something like this available at some point in the future.

Incentivized

• Most of the ETL processes were automated, cutting down on human labor.
• Apache Airflow's user interface (UI) was very informative and straightforward.
• Since ETL processes were providing data via airflow, we were able to gain a deeper comprehension of the data at hand.

Incentivized

• Pentaho has improved our overall business process.
• Pentaho has helped the Managers and Directors to analyze the numbers going up and down from time to time.
• We have a started a big project using Pentaho that is going to include all the business processes in the organization.

Incentivized