Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.
N/A
Control-M
Score 9.3 out of 10
N/A
Control-M from BMC is a platform for integrating, automating, and orchestrating application and data workflows in production across complex hybrid technology ecosystems. It provides deep operational capabilities, delivering speed, scale, security, and governance.
For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.
Scenarios where it's well suited are where you're running jobs across platforms. We have processes on our mainframe that have to complete our daily batch processing and the controlling handles the scheduling of the daily batch processing. Then once that's done, it moves into executing stuff on servers like SQL loads and transferring files to load into SQL and then also running reporting on the SQL servers when the data's been loaded and is available. It makes for a really smooth transition of the overnight processing the bank has to go through. That's our main function that we use it for currently. It handles it very well.
Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.
Workload change manager is one of our favorite features of this product. It enforces standards, which is a huge benefit. Users don't just make crazy changes that cause issues with other jobs. It does great promotions from one environment to another, transforming all the data to match the next environment.
Certificate on all levels: Each agent uses different keystrokes for different functions, such as AI, web service calls, and MFT.
It's really hard to manage these keystrokes.
Trust stores, in case control-m, act as servers; it should be simple to implement certificates from a corporate CA.
Some functions must be performed in the fat client, while others can only be performed through the web interface. This should be streamlined as soon as possible.
Control-M integrates with DevOps toolchains to automate the deployment of applications and updates. It supports Jobs-as-Code, allowing developers to define workflows programmatically, which accelerates development cycles and improves deployment consistency1.These use cases highlight how Control-M can streamline operations, enhance efficiency, and support critical business functions across various domains.
For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.
Since it is GUI-based, it is not hard to use, although all new programs have a learning curve. The color scheme is easy to use to find the status of different Batch jobs. The check-in and check-out feature of job definitions is easy to use and avoids problems with users trying to use the same resource at the same time.
Secondary Instances: Control-M supports the installation of a secondary instance of the entire Control-M environment, Control-M/EM, or Control-M/Server.Automatic & Manual Failover: In case of a failure on the primary host, Control-M can automatically failover to the secondary host if using Oracle or MSSQL databases. Manual failover is also an option, enabling a controlled switch during planned maintenance.Fallback: After resolving the issue on the primary host, you can easily fall back to it, or even designate the secondary host as the new primary. Database Replication: For high availability, Control-M leverages database replication from the primary site to a disaster recovery site. While replication is essential, its implementation and maintenance are the user's responsibility.
good page load times, efficient report completion, and minimal impact on integrated systems. Specifically, the well-designed GUI contributes to a positive user experience, and the platform's ability to automate various stages of the workflow, including Big Data processes, is highlighted as a key strength. Fast Page Loads: Control-M is reported to have a responsive user interface with fast page load times, allowing users to quickly navigate and manage their workflows
He contactado varias veces con el soporte de BMC y ha sido bastante bueno, siempre han sabido darme una solución a lo que he pedido. Esta vez quiero hacer alguna nueva pregunta, pero no se si se me podrá contestar, ya que es algo que tal vez fuera de otro rango y no pertenezca a ellos.
Very knowledgeable instructors provide a hands-on, collaborative learning experience and can interact directly with instructors to develop our Control-M skills. This format allows for immediate feedback, in-depth discussions, and tailored guidance, leading to a deeper understanding of Control-M concepts and practical application. Face-to-face interaction fosters higher engagement and a more dynamic learning environment.
Simple and easy to use web based, well paced. Available any time. All online courses are simple and easy to access and use. Very practical everyday use scenarios and solutions. Incorporates software simulations, learning games, and built-in assessments to enhance comprehension and engagement. Online subscriptions are regularly updated with the latest product information, ensuring users have access to the most current knowledge.
As HA we have depend on the external DB, why don't we have HA feasibility with embedded DB. As with external DB, there are performance issues and fine tuning the DB. As if its embedded DB, Control-M it self take care of the functionality.
Apache Airflow is suited for a much wider set of use cases compared to Databricks. You can run it anywhere, and there is also no vendor lock-in. With Airflow, we can utilize almost any compute engine. Same thing we want to do with Databricks. There might be some level of difficulty based on the support.
I met him as a Control-M consultant, performing migrations from CA to BMC. I knew the product and interface were not as friendly or intuitive as Control-M's and could not integrate new technologies and external applications. One of CA's excellent capabilities was the ability to incorporate executions with Java Scripting to obtain variable values or to order processes natively in the application using its utilities.
awesome product.Control-M delivers advanced operational capabilities easily consumed by Dev, Ops, data teams, and lines of business.Control-M Workflow InsightsApplication and data workflow observability: Increased confidence that SLAs are being met for Control-M users and IT leadersComprehensive control and management capabilities: Enhanced dashboards and reporting with constant telemetry and intelligent analysis on executing workflowsSelf-service visibility: In-depth reporting to help teams work autonomously
Control-M has improved service delivery times, reliability and quality of batch processing.
It has simplified the management of the operation and the use of the alert system has made it possible to act in a coordinated and efficient manner to solve problems.
The implementation of policies has made it possible to make greater use of Control-M and thus reduce development costs that are generated unnecessarily when the potential of the system is not considered.
