Queues have a tendency to get unruly. Whether it is a human queue, or a job queue in code, it does not matter. Without some planing and thought into the queue, you can end up with chaos. This goes doubly for non-linear code, where there can be multiple workers taking jobs off the queue.

A not-unruly queue A not-unruly queue

Recently, I have been doing some work on a pet project which is related to IoT. Part of the project requires a device, with a potentially unreliable connection to the server. The device sends messages regarding the state of certain variables, which are then processed by the server.

One of the first things I realized when I was testing, is that things went south fast if the connection broke between device and server. So I started on a quest to make it more robust. Mind you, this should not have been a quest, but I could not find the tools I needed. The most important tool for handling failed connections is a queue, which can act as a buffer for when the thing emptying the queue (the connection) stops.

There are MANY queues available for NodeJS, but after trying 4-5 of them, I could not find one which met all of my requirements. The requirements are:

  • Fast Performance (non-blocking)
  • Easy to use
  • Retries tasks on failure
  • Emits Events
  • Configurable Worker and Concurrency
  • Allows workers to be paused and resumed.

After a few days screwing around with it, I finally have landed what I feel is a pretty robust queue for NodeJS. Since I am creative with my names, I opted to call it “robust-queue”. It is a package on NPM and open sourced on GitHub. Check it out here.

The queue makes use of the EventEmitters3 package, which improves the performance over the standard EventEmitter package. It also allows for pausing and resuming of the processing, which is important when it is being used a buffer for unreliable internet connections.

I have some additional enhancements such as modular persistence that I want to do to the queue over time, but for now it seems pretty robust ?