Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research and by community contributors.
Caffe is only appropriate for some new beginners who don't want to write any lines of code, just want to use existing models for image recognition, or have some taste of the so-called Deep Learning.
I would recommend it for use when anyone wants to quickly develop a neural network. Or if a user is solving any machine learning problem that includes deep learning. And this kind of problem will be like image recognition, face recognition, doing some text analysis using deep learning which includes LSTM or some other algorithm.
Caffe's model definition - static configuration files are really painful. Maintaining big configuration files with so many parameters and details of many layers can be a really challenging task.
Besides imagine and vision (CNN), Caffe also gradually adds some other NN architecture support. It doesn't play well in a recurrent domain, so we have to say variety is a problem.
Caffe's deployment for production is not easy. The community support and project development all mean it is almost fading out of the market.
The learning curve is quite steep. Although TensorFlow's is not easy to master either, the reward for Caffe is much less than the TensorFlow can offer.
TensorFlow is kind of low-level API most suited for those developers who like to control the details, while Keras provides some kind of high-level API for those users who want to boost their project or experiment by reusing most of the existing architecture or models and the accumulated best practice. However, Caffe isn't like either of them so the position for the user is kind of embarrassing.
As Keras is the high level API, so using Keras, we don't have to be bothered by the low level TensorFlow complexity, and we can reduce a lot coding and testing efforts.