Mesos is really great when you have a big datacenter with many different applications and use cases. It will help you to optimize the resource usage, being a centralized API for your infrastructure. It will not suit well for small companies that just need to deploy a web app. In this case, I would recommend something smaller.
Along with all the best features and support by k8s, the automatic container scheduling to worker nodes and also self-healing containers which is what I like the most. On the other side, when I was installing the k8s cluster on CentOS 8, it was quite difficult for me, but never mind it is working as we expected and it is a one-time effort. Especially, in my case, there are more than 7 application containers required to run and communicate with each other, so for us, Kubernetes is an optimal solution.
Mesos may have many frameworks. If you have Mesos installed on your servers, you may use it for many kinds of tasks. Today we're running only web applications but the idea is to install a different framework for big data soon.
Unreliable deployments that would fail for no good reason. Sometimes our Docker container would be "restarting" forever because Mesos thought it didn't have enough resources to start the container.
Impossibly slow UI. Built in React under the hood with a lot of bloatware backed in, so loading the Mesos UI on a slow internet connection was painful.
No real logging solution - it would stream "console.log()" output to the UI, but searching for logs wasn't really possible without downloading a huge file.
No built-in support for redeploying containers from a CI. We had to create a service whose whole job was to expose an HTTP endpoint that restarted a container, and then made Circle CI ping the endpoint whenever we wanted to redeploy.
Local development, Kubernetes does tend to be a bit complicated and unnecessary in environments where all development is done locally.
The need for add-ons, Helm is almost required when running Kubernetes. This brings a whole new tool to manage and learn before a developer can really start to use Kubernetes effectively.
Finicy configmap schemes. Kubernetes configmaps often have environment breaking hangups. The fail safes surrounding configmaps are sadly lacking.
The Kubernetes is going to be highly likely renewed as the technologies that will be placed on top of it are long term as of planning. There shouldn't be any last minute changes in the adoption and I do not anticipate sudden change of the core underlying technology. It is just that the slow process of technology adoption that makes it hard to switch to something else.
It is an eminently usable platform. However, its popularity is overshadowed by its complexity. To properly leverage the capabilities and possibilities of Kubernetes as a platform, you need to have excellent understanding of your use case, even better understanding of whether you even need Kubernetes, and if yes - be ready to invest in good engineering support for the platform itself
Kubernetes is by far the best choice. More reliable and better developer experience. Mesos is prone to sporadic failures and not really designed to handle CI/CD-based deployments. Docker Cloud once shut down our entire cluster for "upgrades" without giving us any warning.
As I said earlier also - - K8s manage the workloads better as compared to OpenStack in terms of reliability, observability & reachability. - K8s is not limited to only a single networking or storage solution as compared to OpenStack. - Networking (which is a key concept) is much simpler in K8s as compared to OpenStack. - It is possible to upgrade your applications without downtime in K8s but in OpenStack, you either have to divert the traffic or face an outage because you have to delete the whole stack & then recreate it.
