Google Cloud Dataflow vs. IBM Streams (discontinued)

Based on my experience, streaming / real time / machine learning / AI type of processing and batch processing which needs less transformation are very well suited. Work load that needs complex transformation / multiple hops gets very complicated to implement. New feature like Dataflow SQL option will come in handy for sql heavy users.

Streams is a good fit for situations requiring low end-to-end latency, have complex real-time analytical processing needs on large fast data, or where the reduction of operational costs is important. However, it is very much a data-in-motion technology and not well suited for situations such as some forms of machine learning where the entire historical data set needs to be operated on. Note that it's fairly common to use Streams to perform online scoring using models that were trained offline using other technologies.

Incentivized

• Streaming, Real time work load
• Batch processing
• Auto scaling
• flexible pricing

• Query analysis of real time streaming data
• Filter out events based on time windows
• Scalability for large scale data, production tested

Incentivized

• inbuild template options can be expanded
• more data connector options
• easy of use

• Documentation could be more extensive, with more examples, although overall this is not too bad compared to some of the alternative solutions.
• Seems expensive to use in production.

Incentivized

Google Cloud Dataproc Cloud Datafusion

We are using Spark streaming as well as Storm for streaming options. Currently streams provides a better way of building applications easier faster and run efficiently. Also like the flexibility it provides with both us and SPL.

Incentivized

• cost saving from managing our own data center for ETL servers
• consumption based pricing
• with auto scaling feature, we were able to expand components to support work load

• Ability to do more with less
• Admins and data analyst can now focus on more thinking tasks
• No negative impacts yet

Incentivized