Apache Airflow vs. Camunda

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

Camunda Platform is well suited for scenarios where there are different stages in a business flow and the flow is driven by user action at each stage. For example placing of an order on an ecommerce platform. Depending on whether user was able to make the payment or not the workflow would go to dispatch or retry stage. Now the retry stage would trigger further actions like sending follow up emails etc. Likewise, dispatch stage would have a different set of actions. Since every order is important and we need to know where it stands, using Camunda Platform is imperative. Camunda Platform might not be a right choice where just a one off thing needs to be done. For example, uploading of product information by user or periodic processing of heavy images by a worker. These are all either one step processes or periodic automated processes where we can track the status without using a business modeler like Camunda Platform.

Incentivized

• Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
• Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
• Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.

Incentivized

• Task orchestration
• Freeware
• Graphical interface allows normal users to understand business process model diagrams more intuitively

Incentivized

• A local "dry run" or IDE plugin that can validate and simulate DAG execution without needing a full environment.
• Better feedback on DAG parse errors in the UI or CLI.
• Navigating large DAGs with hundreds of tasks can be slow and hard to understand visually.

Incentivized

• Documentation - It is usually daunting for beginners because of lack of good documentation
• Simplified setup on local machine for Camunda Server would help developers test changes quickly
• Easier migration from one deployment version of a Camunda process to another deployment version would help in making changes and deploying them faster.
• Heap memory management becomes issue at times resulting in stuck processes. This needs to be resolved.

Incentivized

For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.

Incentivized

Camunda provides pretty standard product support offerings.

Apache Airflow is suited for a much wider set of use cases compared to Databricks. You can run it anywhere, and there is also no vendor lock-in. With Airflow, we can utilize almost any compute engine. Same thing we want to do with Databricks. There might be some level of difficulty based on the support.

Incentivized

• Most of the ETL processes were automated, cutting down on human labor.
• Apache Airflow's user interface (UI) was very informative and straightforward.
• Since ETL processes were providing data via airflow, we were able to gain a deeper comprehension of the data at hand.

Incentivized

• We've built a good business around developing and deploying Camunda for our clients over the last 7 years.
• Our clients are finding it a cost-effective way to try before they buy.
• There's an over-reliance on big offshore outsourcing shops, which hurts ROI and success rates.