Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
Redis Software
Score 8.4 out of 10
N/A
Redis is an open source in-memory data structure server and NoSQL database.
N/A
Pricing
Apache Kafka
Redis Software
Editions & Modules
No answers on this topic
No answers on this topic
Offerings
Pricing Offerings
Apache Kafka
Redis Software
Free Trial
No
Yes
Free/Freemium Version
No
Yes
Premium Consulting/Integration Services
No
No
Entry-level Setup Fee
No setup fee
Optional
Additional Details
—
—
More Pricing Information
Community Pulse
Apache Kafka
Redis Software
Features
Apache Kafka
Redis Software
NoSQL Databases
Comparison of NoSQL Databases features of Product A and Product B
For brokering messages, Confluent Kafka is well suited since it offers a managed solution ready to use. Scenarios where the solution is not very well suited are for example, where pricing is an issue. The solution costs quite a lot for basic usage (for example: for 3 clusters, pricing is above 100k$ a year).
Perfect solution for caching needs. If you have a bottleneck due to frequent data access to your database, then Redis can really help you by diverting those traffic away from your database. Its key/value pair structure also makes data lookup very efficient, providing excellent performance.
Apache Kafka is able to handle a large number of I/Os (writes) using 3-4 cheap servers.
It scales very well over large workloads and can handle extreme-scale deployments (eg. Linkedin with 300 billion user events each day).
The same Kafka setup can be used as a messaging bus, storage system or a log aggregator making it easy to maintain as one system feeding multiple applications.
Easy for developers to understand. Unlike Riak, which I've used in the past, it's fast without having to worry about eventual consistency.
Reliable. With a proper multi-node configuration, it can handle failover instantly.
Configurable. We primarily still use Memcache for caching but one of the teams uses Redis for both long-term storage and temporary expiry keys without taking on another external dependency.
Fast. We process tens of thousands of RPS and it doesn't skip a beat.
The Kafka Tool is a community-made Java application that looks and feels from the past century.
Logging can be confusing. This certainly shows when we have to do troubleshooting.
Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.
Redis is super fast but it comes with a cost. Whole dataset resides in RAM. So it can be costly as primary memory is more costly, then secondary ones.
Persistence issues: To achieve it, Redis uses a memory dump to create a persistence snapshot, that's cool. But it requires some Linux Kernel tweaking to avoid performance degradation while the Redis server process is forking. This further causes latency.
Master-slave structure side effect: Master-slave architecture comes with its own side effects. Please note that there will be only one master with multiple slaves for replication. All writing goes to the master, which creates more load on the master node. So, when the master goes down, the whole architecture does.
Kafka has suited our use case very well so far. Going forward we are planning to expand our platform manifold so the load on Kafka and our reliance on Kafka is going to increase only.
We will definitely continue using Redis because: 1. It is free and open source. 2. We already use it in so many applications, it will be hard for us to let go. 3. There isn't another competitive product that we know of that gives a better performance. 4. We never had any major issues with Redis, so no point turning our backs.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
It is quite simple to set up for the purpose of managing user sessions in the backend. It can be easily integrated with other products or technologies, such as Spring in Java. If you need to actually display the data stored in Redis in your application this is a bit difficult to understand initially but is possible.
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
The support team has always been excellent in handling our mostly questions, rarely problems. They are responsive, find the solution and get us moving forward again. I have never had to escalate a case with them. They have always solved our problems in a very timely manner. I highly commend the support team.
Apache Kafka is built for scale. From high throughput and real-time data streaming, it has a strong advantage over RabbitMQ with its low latency. This put Apache Kafka at the forefront as the platform of choice for large datasets messaging and ensuring scalability when data scale up tremendously. RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale. In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.
UI isn't that great compared to the other competitors. The management of our memcached cluster was becoming pretty complicated as the application grew in size. Redis is a much better option compared to memcached. Redis is bit unreliable compared to the alternative RabbitMQ especially when it needs to be integrated with Celery.
Positive: bursts of traffic on special holidays are easy to handle because Kafka can absorb and buffer all the messages we need to process long enough to let an understaffed set of back-end services catch up on processing. Hard to put a number to it but we probably save $5k a month having fewer machines running.
Positive: makes decoupling the web and API services from the deeper back-end services easier by providing topics as an interface. This allowed us to split up our teams and have them develop independently of each other, speeding up software development.
Negative: our engineers have made mistakes such as accidentally dropping a few thousand messages due to the CLI being confusing to use, and as a result a customer lost some of their precious data. I'd say that was more our fault than Kafka's though.
Existing tools like Redisson that were built over Redis reduced dev time in solving challenging problems, which had a positive impact on ROI.
We initially misused Redis for persistent storage which had a negative impact on ROI because we were paying a lot for inactive users.
The increased performance we achieved using Redis in areas like locking helped us improve the performance of our system reducing the likelihood of system timeouts.
