Scalable Scheduling Framework and Orchestration tool
Use Cases and Deployment Scope
We are using Apache Airflow as an orchestration tool in data engineering workflows in gaming product.
We are scheduling multiple jobs i.e hourly / daily / weekly / monthly.
We have a lot of requirement for dependent jobs i.e job1 should mandatory run before job2, and Apache Airflow does this work very swiftly, we are utilising multiple Apache Airflow integration with webhook and APIs. Additionally, we are doing a lot of jobs monitoring and SLA misses via Apache Airflow features
Pros
- Job scheduling
- Dependent job workflows
- Failure handling and rerun of workflows
Cons
- Better User Interface
Likelihood to Recommend
Dependent Job scheduling
Rerun mechanism of workflows
High availability deployment strategies
