When building Node.js applications that rely on message brokers for asynchronous communication, developers often face the challenge of choosing between Apache Kafka and Apache Pulsar. While both are excellent distributed messaging systems, Pulsar has distinct advantages when it comes to scaling Node.js applications.

This blog post explains why Pulsar’s architecture and features make it a better fit for the Node.js ecosystem compared to Kafka, especially for dynamic, horizontally scaled systems.

The Scaling Challenge in Node.js Applications

Before we begin, it’s key to understand the architecture and scaling strategies for Node.js applications. Node.js is single-threaded, which means it can only use up to a single CPU core. This constraint means that in order for our Node.js applications to scale beyond a single CPU core so we can handle greater workloads and request volumes, we must rely on horizontal scaling.

Horizontal scaling is the act of running multiple instances of our application. With multiple instances of our Node.js app, we can utilize additional CPU cores and handle increased workloads. This is commonly achieved in environments like Kubernetes by scaling up a deployment so we increase the number of pods (instances) of our app. In fact, the Kubernetes Horizontal Pod Autoscaler does this for us automatically, based on scaling metrics we provide like CPU & memory utilization

If you’d like to learn more about the Horizontal Pod Autoscaler and actually implement it yourself, I have a dedicated video here. Now that we have the architecture and scaling strategy of Node.js out of the way, let’s dig deeper into how Kafka and Pulsar both contribute to it.

Kafka’s Partition-Based Consumer Model

Kafka’s consumer model relies on partitions to distribute messages across consumers. Each partition can have only one active consumer per consumer group. This introduces constraints:

• The number of consumers is tied to the number of partitions.
• Scaling up Node.js instances beyond the partition count results in idle instances, wasting resources.
• Rebalancing during scaling events causes consumption delays, disrupting real-time processing.

Kafka partitions are static, and scaling is inherently limited by their number:

• Example: If you have a topic with 3 partitions and 5 Node.js instances, only 3 instances will consume messages; the other 2 remain idle.
• Scaling beyond the partition count requires increasing the number of partitions, which introduces operational complexity.
• When a consumer is added or removed from a partitioned topic, a rebalance must occur. This rebalance mechanism disrupts processing of all connected consumers until it is complete.

Pulsar’s Dynamic Consumer Model

Pulsar, on the other hand, introduces a more flexible consumer model. With shared subscriptions, multiple consumers can process messages from the same partition or topic. This makes it ideal for dynamically scaling Node.js applications:

• Each Node.js instance can independently consume messages, maximizing utilization.
• Adding or removing instances doesn’t require rebalancing, ensuring uninterrupted message processing.
• Topics can be created without partitions which means we don’t have worry about the operational overhead of modifying it when our deployment scale.

The shared subscription type in Pulsar allows multiple consumers to attach to the same subscription. Messages are delivered in a round-robin distribution across consumers, and any given message is delivered to only one consumer. When a consumer disconnects, all the messages that were sent to it and not acknowledged will be rescheduled for sending to the remaining consumers.

What this means is that we can dynamically scale our Node.js application at any time without having to worry about partition counts. New consumers will simply join the shared subscription pool and begin processing messages in a round-robin right away. Any consumers that disconnect will not affect the processing of others.

Other Benefits

There are some other great benefits to using Pulsar that I have noticed.

Out-of-the-Box Features

Pulsar includes built-in features like:

• Delayed delivery for retry mechanisms.
• Message deduplication to ensure idempotency.
• Dead-letter queues for handling failures.

These features simplify application logic and reduce the need for custom implementations.

Community Support

The most active & feature-rich Node.js Kafka client availbile today is kafkajs. This project is no longer actively maintained and hasn’t seen any new feature PRs in almost a year.

Compare that to pulsar-client-node which is actively maintained with new releases being pushed frequently. In addition, I think the documentation for Pulsar is very rich and makes learning about it a breeze.

Conclusion

While both Apache Kafka and Apache Pulsar are powerful tools, Pulsar’s flexibility and dynamic scaling capabilities make it a better choice for Node.js applications. Its shared subscription model, seamless scaling, and rich feature set align well with the needs of horizontally scaled, event-driven Node.js systems.

For developers looking to maximize efficiency, reduce complexity, and support dynamic scaling, Pulsar is the clear winner.

What are your thoughts?

Do you agree that Pulsar is better for Node.js, or do you have experiences where Kafka outshined? Let us know in the comments!

Take Your Skills to the Next Level with My Latest Course! 🚀

Ready to master the art of building scalable, distributed Node,js microservices? In my latest course, I dive deep into Node.js horizontal scaling and Apache Pulsar’s shared subscription model. You’ll learn:

• How to design scalable Node.js systems with dynamic message consumption.
• Best practices for leveraging Pulsar in real-world applications.
• Hands-on projects showcasing the power of distributed asyc messaging.

Whether you’re a backend developer or an architect looking to enhance your skill set, this course will help you scale your expertise to new heights.