Amazon S3 (Simple Storage Service) vs. Databricks Data Intelligence Platform

For archiving old data that is infrequently accessed it is perfect. You can choose to let it go into cold/glacier storage which saves even further costs but at the expense of accessibility. I like that you can set access rules to automatically move it to the next storage tier after a certain amount of time that it has not been accessed. I also use it a lot with PHP via the API. We have some custom in-house applications that have a fair amount of data uploaded into them. S3 has been a perfect solution to store these files, taking the load off web servers and never having issues with running out of storage.

Incentivized

If you need a managed big data megastore, which has native integration with highly optimized Apache Spark Engine and native integration with MLflow, go for Databricks Lakehouse Platform. The Databricks Lakehouse Platform is a breeze to use and analytics capabilities are supported out of the box. You will find it a bit difficult to manage code in notebooks but you will get used to it soon.

Incentivized

• Reliable and secure way to store objects in cloud: Storing any type of file(text, pdf, doc, csv, etc) is very easy with S3. Fetching this stored content as and when you require is also pretty easy and can be done using both the console and AWS CLI. Appropriate permissions can be set up for buckets using IAM roles/policies.
• Versioning in buckets: S3 gives you a very handy feature to store multiple versions of objects stored in a bucket.
• Lifecycle policies: You can set up lifecycle policies in S3 that can move your older objects to IA or Glacier. This setup is very easy and can be done within minutes for a bucket.
• Replication: The cross-region replication that S3 provides is wonderful. Beware of the inter-regional data transfer costs though.

Incentivized

• There is databricks community, which is a free version. It is available for beginners to have an easy start with a big data platform. It does not have every feature of the full version but is still adequate for extremely new coders.
• There are many resourceful training elements that are available to developers, data scientists, data engineers and other IT professionals to learn Apache Spark.

Incentivized

• The biggest problem is to rename the bucket. There is no direct way to do it. One need to copy entire content to the different bucket with intended bucket name and then remove the old bucket. Sometimes it creates issues.
• There is no direct way to upload .zip file and extract it to inside the bucket.
• While uploading large files, sometimes you will find a drop of upload speed. I observe it so many times and while checking my internet speed, I find it absolutely perfect. So there must have something wrong on the AWS side.

Incentivized

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

Due to princing, availability and scalability.

Incentivized

The UI could have some improvements (better filters) and there is a lack of some useful functionality, such as renaming an existing bucket: the latter is much needed in the context of rapidly evolving companies. Overall though, Amazon S3 (Simple Storage Service) is easy to use and to onboard people and tools to, thanks to its various APIs and flexibility.

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

It depends on your tier within Amazon on how great of support you get. For us we have a dedicated Point of Contact that is great in taking in what we need and discussing it with the S3 team. The best thing is features we need or suggest have a good chance of landing on their roadmap.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

S3 is the most mature simple storage service on the web. It has direct competitors from Google and Azure, as well as a bunch of other competitors that focus on different aspects. For example, Backblaze specializes on file backups, and while s3 can also be used for that, Backblaze provides a better price point in exchange for more focused functionality. S3 really shines in that it performs simple things astonishingly well, while also being flexible enough to stretch itself to other situations (data lakes, file mounts, backup/restores systems, web hosting, etc.).

Incentivized

Databricks is a true all-in-one platform, and at the time of implementation, it had more features available to us, making it a clear choice over Snowflake. Moving our workloads from local computing to the servers in Databricks gave our start-up staff a great quality of life boost.

Incentivized

• Allows us to store large amounts of raw traffic from data providers to allow us to view data our systems received at particular times, in order to reconstruct inputs in case of errors
• Is capable of storing very large amounts of data cheaply without material impact to our business

Incentivized

• ROI for us has been tremendous. Time to market by processing raw data in our big data infrastructure has been pretty fast.
• Non engineers can easily use Databricks, hence helping business customers.
• Thousands of different data combinations can easily be joined and used by our data teams.

Incentivized