Kafka for tracking changes
Rating: 8 out of 10
IncentivizedUse Cases and Deployment Scope
We use Apache Kafka to stream order information across systems. An order may go through certain updates through its lifecycle. These updates need to be communicated to the systems in near real time and we rely on Kafka for this.Our business use case is to take these orders up with the insurance companies for approval and thus the order information need to be up to date. Kafka has been excellent at doing this so far.
Pros
- Receiving messages from publisher and sending to consumer in FIFO manner
- Handling of errors using Dead Letter Queue when message could not be consumed on the consumer end
- Fault tolerance
Cons
- Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
- Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
- Learning curve around creation of broker and topics could be simplified
Likelihood to Recommend
Kafka is well suited in scenarios where a message need to be sent to another system in fault tolerant manner. It is useful when the message size could be large and large number of messages could be floating around.
It would be less appropriate or rather an overkill to use Kafka in scenarios where we are sending short messages to offload certain tasks(like invoice generation and sending email) to a worker(like celery). For such use cases, simple queueing solutions like Amazon SQS should suffice.
It would be less appropriate or rather an overkill to use Kafka in scenarios where we are sending short messages to offload certain tasks(like invoice generation and sending email) to a worker(like celery). For such use cases, simple queueing solutions like Amazon SQS should suffice.