Apache Flume is a product enabling the flow of logs and other data into a Hadoop environment.
N/A
Hortonworks Data Platform
Score 5.0 out of 10
N/A
Hortonworks Data Platform (HDP) is an open source framework for distributed storage and processing of large, multi-source data sets. HDP modernizes IT infrastructure and keeps data secure—in the cloud or on-premises—while helping to drive new revenue streams, improve customer experience, and control costs.
Hortonworks merged with Cloudera in eary 2019.
Apache Flume is well suited in small batch and near real time processing projects, taking data from one point to another with local processing (I mean not external enrichment). Filtering, transforming and multiple push destinations are common grounds for Flume. It is not so nice to use if your data needs external enrichment (taking data from external databases or web services), as transactions and (micro)batches may lead to reprocessing and it relies upon the application to avoid duplicates.
I recommend [Hortonworks Data Platform] as Big Data platform in order to start your developments. It's free. It's easy to use. You can install in more server or use a sandbox with you favorite virtualization platform ( vmware or oracle virtualbox). There is also a containerized version.
Manage our data in hdfs is simple; you can interact with server with REST API.
As an open source project collection, it relies strongly on community activity. You still have the option to contract premium consulting or training services.
Altough it is quickly evolving into Data Science tools availability (eg. Tensorflow incorporate in HDP 3), it can be cumbersome from a developer transitioning from a traditional IDE, into the notebook vs. datalake metaphore.
As expected for a big data infranstructure, the resource requirements base line is rather high. This means that if used on premise, you need to think of about 10 machines for a minimal reasonable deploy.
Apache Flume is open-source so support is limited. Never the less, it has great documentation and best practices documents from their end-users so it is not hard to use, setup and configure.
Apache Flume is on par with Scribe with similar functions. Apache Kafka is a generation purpose while Apache Flume is specific to log aggregation. Google Pub/Sub and IBM MQ are costlier than Apache Flume ( open source ) and have a lot more cost associated with them. Apama Streaming Analytics and Tibco Steaming are more comprehensive streaming solutions than Apache Flume so for deeper performance guarantees, it is easier to use Apache Flume.
While Apache Hadoop is completely open sourced, Hortonworks Data Platform offers support as well as keeps pace with the open source versions. Also, the HDP open sources its own products, thus giving back to the community. I find using the Hortonworks Data Platform more intuitive than Cloudera or MapR versions.
Positive impact on ROI due to a reduction in manual labor to generate and maintain compliance reports based on logs.
Positive impact on the business objective by reducing the need for provisioning compute for log aggregate IT stack in advance but adding on an as-needed basis.
It provides a convenient way of quickly setting up a big data environment, easily setting up clusters with different configurations. It provides several security architectures that can be used as well. Since it provides a big list of components and packaged together, it is a great tool for companies to get set and utilize it for their use cases.
Since it uses Ambari extensively to install, upgrade and manage software, it is very convenient and easy to support and operationalize the components. Alerting and notifications, ability to create custom alerts give you the capability to add any number of alerts to meet your custom needs. It provides a great way to maintain other software by creating mpacks and the ability to add custom code, and you can add other software to be managed in a centralized tool.
The use and support of popular and useful open source software and the company's contribution to the community makes HDP a very useful tool that enables a quick, secure, easily maintainable suite of components that can help companies meet the needs of the business. What is great is that new components keep getting added based on any new useful tool that comes available, like Druid, and made available as part of the suite of components. That helps businesses keep up with new capabilities as they become available, and use them to solve their problems.
