ZeroMQ and scalability

The elegance of ZeroMQ messaging is that it provides easy scalability. The API for 0MQ sockets is the same whether you are doing in-process (inter thread) communication, inter-process communication, peer-to-peer network communication or multicast communication.

As long as you are putting data and not pointers in your messages (*cough* Async::Worker *cough*), you convert your code from in-process communication to cross-network with a one line change:

// Create the socket: same code in either case.
zmq::socket_t outSocket(zmqContext, ZMQ_REP) ;

...

outSocket.connect("in-proc://my-connection") ;
// becomes
outSocket.connect("tcp://192.168.0.65:2342") ;

and voila – your application is networking instead of talking to itself.

They also provide three external utilities that make it a doddle to scale your application across multiple machines.

zmq_queue muxes requests between multiple clients/servers all connecting to a single address – load balancing, if you will.

zmq_ forwarder allows for re-broadcasting and filtering of broadcast/multicast messages; e.g. if you have a bunch of clients subscribed to a very-active publisher, the forwarder will save them having to receive messages they aren’t interested in.

Lastly, zmq_streamer will handle the kind of parallel-pipeline that my Async::Worker does if you wanted the ability to send the work to a single socket and have it able to distribute the work across multiple machines simply by having them connect to the streamer in addition to your local workers…

By using the message-passing paradigm to ensure parallel-friendly encapsulation of your data, and the ZeroMQ messaging library for delivery, you get the benefits of scalable parallelism available in Erlang from the language of your choice.