Apache Flink vs. SQL Server Integration Services

In well-suited scenarios, I would recommend using Apache Flink when you need to perform real-time analytics on streaming data, such as monitoring user activities, analyzing IoT device data, or processing financial transactions in real-time. It is also a good choice in scenarios where fault tolerance and consistency are crucial. I would not recommend it for simple batch processing pipelines or for teams that aren't experienced, as it might be overkill, and the steep learning curve may not justify the investment.

Incentivized

Ideal for daily standard ETL use cases whether the data is sourced from / transferred to the native connectors (like SQL Server) or FTP. Best if the company uses MS suite of tools. There are better options in the market for chaining tasks where you want a custom flow of executions depending on the outcome of each process or if you want advanced functionality like API connections, etc.

Incentivized

• Low latency Stream Processing, enabling real-time analytics
• Scalability, due its great parallel capabilities
• Stateful Processing, providing several built-in fault tolerance systems
• Flexibility, supporting both batch and stream processing

Incentivized

• SSIS works very well pulling well-defined data into SQL Server from a wide variety of data sources.
• It comes free with the SQL Server so it is hard not to consider using it providing you have a team who is trained and experienced using SSIS.
• When SSIS doesn't have exactly what you need you can use C# or VBA to extend its functionality.

Incentivized

• Python/SQL API, since both are relatively new, still misses a few features in comparison with the Java/Scala option
• Steep Learning Curve, it's documentation could be improved to something more user-friendly, and it could also discuss more theoretical concepts than just coding
• Community smaller than other frameworks

Incentivized

• SSIS memory usage can be quite high particularly when SSI and SQL server are on the same machine
• SSIS is not available on any environment other than Microsoft Windows
• SSIS does not function with any database engine back-end other than Microsoft SQL Server

Incentivized

SSIS is responsible for running core business processed managing core business data. It can be managed, improved and expanded using minimal internal resources. It is also able to support all of our current data infrastructure. Replacing SSIS would be time consuming and costly with no apparent ROI.

Incentivized

SSIS has a drag and drop based developer interface, so it is relatively straight forward to get started. You can start to get into the weeds pretty quickly as your solution becomes more complex. However, most of the base functions are right in front of you for a developer. You can also set project and solution level parameters, so when you deploy to new environments, you don't have to jump into each package to change your variables and settings. (For example, default directory to ingest flat files).

Incentivized

Raw performance is great. At times, depending on the machine you are using for development, the IDE can have issues. Deploying projects is very easy and the tool set they give you to monitor jobs out of the box is decent. If you do very much with it you will have to write into your projects performance tracking though.

Incentivized

The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.

Incentivized

The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools

Incentivized

Apache Spark is more user-friendly and features higher-level APIs. However, it was initially built for batch processing and only more recently gained streaming capabilities. In contrast, Apache Flink processes streaming data natively. Therefore, in terms of low latency and fault tolerance, Apache Flink takes the lead. However, Spark has a larger community and a decidedly lower learning curve.

Incentivized

I think SQL Server Integration Services is better suited for on-premises data movement and ADF is more suited for the cloud. Though ADF has more connectors, SQL Server Integration Services is more robust and has better functionality just because it has been around much longer

Incentivized

• Allowed for real-time data recovery, adding significant value to the busines
• Enabled us to create new internal tools that we couldn't find in the market, becoming a strategic asset for the business
• Enhanced the overall technical capability of the team

Incentivized

• Without this, we would have to manually update a spreadsheet of our SQL Server inventory
• We would also have poor alerting; if an instance was down we wouldn't know until it was reported by a user
• We only have one other person who uses SQL Server Integration Services , he's the expert. It would fall to me without him and I would not enjoy being responsible for it.

Incentivized