Azure Data Lake Storage Gen2 is a highly scalable and cost-effective data lake solution for big data analytics. It combines the power of a high-performance file system with massive scale and economy to help you speed your time to insight. Data Lake Storage Gen2 extends Azure Blob Storage capabilities and is optimized for analytics workloads.
N/A
Hortonworks Data Platform
Score 5.0 out of 10
N/A
Hortonworks Data Platform (HDP) is an open source framework for distributed storage and processing of large, multi-source data sets. HDP modernizes IT infrastructure and keeps data secure—in the cloud or on-premises—while helping to drive new revenue streams, improve customer experience, and control costs.
Hortonworks merged with Cloudera in eary 2019.
Azure Data Lake storage is well suited for applications/use cases within organizations where capturing and storing large amounts of data in any format is required, primarily for storing and processing purposes. It's an easy and cost-effective cloud solution for your application data. The ability to integrate with other Azure Services like Azure Databricks and Azure Data Factory is superb.
I recommend [Hortonworks Data Platform] as Big Data platform in order to start your developments. It's free. It's easy to use. You can install in more server or use a sandbox with you favorite virtualization platform ( vmware or oracle virtualbox). There is also a containerized version.
Manage our data in hdfs is simple; you can interact with server with REST API.
Azure Data Lake Storage is extremely scalable. It allows us to scale up or down endlessly based on what we need including replication.
In terms of security, Azure Data Lake Storage fits our requirements really well as we can monitor and encrypt seamlessly. We can also assign permissions through roles and grant network-level access.
Due to the fact that it can scale, we are able to monitor the cost of storage and any given time and make financial decisions about our infrastructure based on how small or big we want to scale.
I'd like to see a better cross-platform native client. Azure Data Explorer is fine, but it's far from the "SSMS" kind of experience SQL Server users are used to.
Listing a large number of file is somewhat problematic and slow. Using the native C# library, running directly on an Azure VM, it can take several hours to list just a couple million files.
Switching from V1 to V2 requires the creation of a new Storage Account and that's pretty inconvenient.
As an open source project collection, it relies strongly on community activity. You still have the option to contract premium consulting or training services.
Altough it is quickly evolving into Data Science tools availability (eg. Tensorflow incorporate in HDP 3), it can be cumbersome from a developer transitioning from a traditional IDE, into the notebook vs. datalake metaphore.
As expected for a big data infranstructure, the resource requirements base line is rather high. This means that if used on premise, you need to think of about 10 machines for a minimal reasonable deploy.
The Azure Data Lake solution is designed for organizations that want to take advantage of big data. It provides a data platform that can help developers, data scientists, and analysts store data of any size and format and perform all types of processing and analytics across multiple platforms and programming languages. It can work with your existing solutions, such as identity management and security solutions. It also integrates with other data warehouses and cloud environments. It can be useful for organizations that need the above softwares.
While Apache Hadoop is completely open sourced, Hortonworks Data Platform offers support as well as keeps pace with the open source versions. Also, the HDP open sources its own products, thus giving back to the community. I find using the Hortonworks Data Platform more intuitive than Cloudera or MapR versions.
The cost can be high for more advanced work. In some cases, for instance, time limits and lab runtimes may be too short if you are too slow to learn what is explained as you go along.
promote flexible team communication. You can create different spaces for different teams, and share files and tasks.
It provides a convenient way of quickly setting up a big data environment, easily setting up clusters with different configurations. It provides several security architectures that can be used as well. Since it provides a big list of components and packaged together, it is a great tool for companies to get set and utilize it for their use cases.
Since it uses Ambari extensively to install, upgrade and manage software, it is very convenient and easy to support and operationalize the components. Alerting and notifications, ability to create custom alerts give you the capability to add any number of alerts to meet your custom needs. It provides a great way to maintain other software by creating mpacks and the ability to add custom code, and you can add other software to be managed in a centralized tool.
The use and support of popular and useful open source software and the company's contribution to the community makes HDP a very useful tool that enables a quick, secure, easily maintainable suite of components that can help companies meet the needs of the business. What is great is that new components keep getting added based on any new useful tool that comes available, like Druid, and made available as part of the suite of components. That helps businesses keep up with new capabilities as they become available, and use them to solve their problems.
