Apache Flume vs. Apache HBase

Apache Flume is well suited in small batch and near real time processing projects, taking data from one point to another with local processing (I mean not external enrichment).
Filtering, transforming and multiple push destinations are common grounds for Flume.
It is not so nice to use if your data needs external enrichment (taking data from external databases or web services), as transactions and (micro)batches may lead to reprocessing and it relies upon the application to avoid duplicates.

Incentivized

HBase is well suited for streaming ingest, fast lookups, massive datasets, data warehouse lookup tables, RDBMS replacement, MongoDB replacement, key-value store, data scans, logs, JSON storage and some binary storage. My preferred use case is for storing data points like time series or data produced by sensors. I often use HBase when I need data available immediately and I am not looking for transactions. This is a great store for really wide tables with tons of columns. It is also great if you are not sure what type of data you are going to have. It really excels at sparse data.

Incentivized

• Multiple sources of data (sources) and destinations (sinks) that allows you to move data form and to any relevant data storage
• It is very easy to setup and run
• Very open to personalization, you can create filters, enrichment, new sources and destinations

Incentivized

• Scalable and truly non-relational data
• HBase operations run in real-time on its database rather than MapReduce jobs
• Scales linearly to support billions of rows with millions of columns

Incentivized

• It is very specific for log data ingestion so it is pretty hard to use for anything else besides log data
• Data replication is not built in and needs to be added on top of Apache Flume (not a hard job to do though)

Incentivized

• Write performance
• Performance support for parquet file format. supports, but performance wise still not there
• API / library availability for spark, rather than creating a new library for it

Incentivized

There's really not anything else out there that I've seen comparable for my use cases. HBase has never proven me wrong. Some companies align their whole business on HBase and are moving all of their infrastructure from other database engines to HBase. It's also open source and has a very collaborative community.

Incentivized

Apache Flume is open-source so support is limited. Never the less, it has great documentation and best practices documents from their end-users so it is not hard to use, setup and configure.

Incentivized

Apache Flume is on par with Scribe with similar functions. Apache Kafka is a generation purpose while Apache Flume is specific to log aggregation. Google Pub/Sub and IBM MQ are costlier than Apache Flume ( open source ) and have a lot more cost associated with them. Apama Streaming Analytics and Tibco Steaming are more comprehensive streaming solutions than Apache Flume so for deeper performance guarantees, it is easier to use Apache Flume.

Incentivized

Compared NoSQL databases with traditional databases for faster retrieval and consistency. As MongoDB is a NoSQL supports dynamic fields, however, query performance is bad for aggregations and added maintenance. When compared with MySQL and Teradata, it could not scale up as fast as Hbase and added cost involved to it. HBase can be easily scalable to a huge volume of records, have a faster lookup and provides consistency

Incentivized

• Positive impact on ROI due to a reduction in manual labor to generate and maintain compliance reports based on logs.
• Positive impact on the business objective by reducing the need for provisioning compute for log aggregate IT stack in advance but adding on an as-needed basis.

Incentivized

• Positive: Open source, easy to use, good to store big data.
• Negative: SQL functionalities are not available.
• More memory utilization
• More troubleshooting

Incentivized