Apache Airflow vs. KNIME Analytics Platform

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

KNIME Analytics Platform has vastly improved our effectiveness when working with large data sets. The self documenting GUI allows analysts to focus on what they are trying to accomplish, not complex code syntax. If we were to use traditional tools, like SQL, work would take much longer and it would be more difficult to collaborate both internally and with clients. Since KNIME Analytics Platform is database oriented, some spreadsheet functions are not supported, which is as it should be. For small data sets we often use Excel vlookup and pivot tables in place of KNIME Analytics Platform. If VBA code is requried, we go to KNIME Analytics Platform as we find VBA to be unstable in Excel.

• Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
• Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
• Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.

Incentivized

• Visual programming as oppose to scripting encourages data analysts to reap deeper insights from their data
• Large community contribution in extending the KNIME Analytics Platform into other areas of analytics, e.g. Text Analytics, Predictive Analytics, ML, etc.
• Open source with periodic updates ensures it is equipped to deal with the most sophisticated data analytics use case

• A local "dry run" or IDE plugin that can validate and simulate DAG execution without needing a full environment.
• Better feedback on DAG parse errors in the UI or CLI.
• Navigating large DAGs with hundreds of tasks can be slow and hard to understand visually.

Incentivized

• Automation - e.g. RapidMiner Studio provides a Turbo Prep function, where one can get to working on models more quickly (RapidMiner is not open source though)
• KNIME does not replace a regular reporting tool - it is not meant to. However, if I have already spent some time developing a data acquisition and analytical model, it would be nice to be able to deploy, for example, a monitoring or reporting module that would process data autonomously and react accordingly.

Incentivized

We are happy with Knime product and their support. Knime AP is versatile product and even can execute Python scripts if needed. It also supports R execution as well; however, it is not being used at our end

For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.

Incentivized

The training KNIME Analytics Platform provide helps you get to grips with a product that is already very intuitive. There is a KNIME Analytics Platform way of thinking about addressing problems, but once you understand a couple of patterns which you see again and again in your workflow it all makes sense.

Incentivized

KNIME's HQ is in Europe, which makes it hard for US companies to get customer service in time and on time. Their customer service also takes on average 1 to 2 weeks to follow up with your request. KNIME's documentation is also helpful but it does not provide you all the answers you need some of the time.

Incentivized

KNIME Analytics Platform is easy to install on any Windows, Mac or Linux machine. The KNIME Server product that is currently being replaced by the KNIME Business Hub comes as multiple layers of software and it took us some time to set up the system right for stability. This was made harder by KNIME staff's deeper expertise in setting up the Server in Linux rather than Windows environment. The KNIME Business Hub promises to have a simpler architecture, although currently there is no visibility of a Windows version of the product.

Incentivized

Apache Airflow is suited for a much wider set of use cases compared to Databricks. You can run it anywhere, and there is also no vendor lock-in. With Airflow, we can utilize almost any compute engine. Same thing we want to do with Databricks. There might be some level of difficulty based on the support.

Incentivized

There are two aspects which put KNIME Analytics Platform ahead of other products. Firstly the fact that KNIME Analytics Platform comes at no cost and no restrictions on its use is an instant winner for any organisation wanting to democratise their data. It means that a client is free to install it on as many machines as they wish without worrying about costs, the number of seats required or payment models or procurement negotiation. It also means that we are not building costs into our clients business. Secondly, KNIME Analytics Platform has a very comprehensive set of tools for importing/exporting data, data manipulation and data science. Some products offer analytics packages on top of their base offering at additional cost and they are still not as comprehensive as what you get with KNIME Analytics Platform for free. For some types of analysis you may require to download additional packages with KNIME Analytics Platform, but its invariably at no cost, those packages are kept out of the main download to keep the size down. Due to the easy integration with R and Python, I view KNIME Analytics Platform as also having the capabilities of those languages too. This has helped me in the past with seamlessly importing a rare filetype and using very specific models not directly available in KNIME Analytics Platform.

Incentivized

• Most of the ETL processes were automated, cutting down on human labor.
• Apache Airflow's user interface (UI) was very informative and straightforward.
• Since ETL processes were providing data via airflow, we were able to gain a deeper comprehension of the data at hand.

Incentivized

• It is suited for data mining or machine learning work but If we're looking for advanced stat methods such as mixed effects linear/logistics models, that needs to be run through an R node.
• Thinking of our peers with an advanced visualization techniques requirement, it is a lagging product.

Incentivized