Reliable data storage and access is a big challenge in AI applications and IBM ESS is a service you need to solve the problem. If you are building an AI service and you already use the IBM ecosystem for AI compute requirements, ESS can serve as an excellent software based storage solution. Integration with other platforms such as EMR/Databricks is still a challenge and should be improved.
When you get a Spectrum Scale solution from IBM you are certain of two things. 1) You will need a specialized storage admin who will be able to take all of its advantages and make available for your organization. It is an appliance that not so many storage admins would be comfortable working on. Invest properly on both hardware and human resources. 2) You will scale forever. We started with a couple of hundred of Terabytes and grew to dozens of Petabytes.
IBM ESS is optimized for AI and Big Data usecases while S3 is a general purpose storage solutions. EMR and Databricks have lakehouse/data warehousing solutions for distributed computing but are more optimized for just the big data pipelining solutions and not essentially for AI usecases, especially for inference, when you need to load model artifacts really quickly.
