ZeroMQ and scalability

The elegance of ZeroMQ messaging is that it provides easy scalability. The API for 0MQ sockets is the same whether you are doing in-process (inter thread) communication, inter-process communication, peer-to-peer network communication or multicast communication.

As long as you are putting data and not pointers in your messages (*cough* Async::Worker *cough*), you convert your code from in-process communication to cross-network with a one line change:

// Create the socket: same code in either case.
zmq::socket_t outSocket(zmqContext, ZMQ_REP) ;


outSocket.connect("in-proc://my-connection") ;
// becomes
outSocket.connect("tcp://") ;

and voila – your application is networking instead of talking to itself.

They also provide three external utilities that make it a doddle to scale your application across multiple machines.

zmq_queue muxes requests between multiple clients/servers all connecting to a single address – load balancing, if you will.

zmq_ forwarder allows for re-broadcasting and filtering of broadcast/multicast messages; e.g. if you have a bunch of clients subscribed to a very-active publisher, the forwarder will save them having to receive messages they aren’t interested in.

Lastly, zmq_streamer will handle the kind of parallel-pipeline that my Async::Worker does if you wanted the ability to send the work to a single socket and have it able to distribute the work across multiple machines simply by having them connect to the streamer in addition to your local workers…

By using the message-passing paradigm to ensure parallel-friendly encapsulation of your data, and the ZeroMQ messaging library for delivery, you get the benefits of scalable parallelism available in Erlang from the language of your choice.


so what are some real world examples of how this helps? cell to cell client tracking? running physics on a second core :D?

Well, in the very here and now, I am using it to shunt massive amounts of debugging info to a second thread that converts it into logging information to help us find the cause of the concussion bug and a few of those weird state things that are going on.

They’re just not happening when we run it in debug mode, and the amount of processing the logging stuff takes would drop the frame rate to 1 fps. But using zmq to ship it to a second thread, you perhaps only lose 1 fps.

It’s unlikely that I’m going to refit our current networking stack with ZeroMQ (I think our “teulKit” over ZeroMQ would be freakin’ awesome and absolutely what Marty was aiming for with teulKit). But I might use it to start offloading various pieces of work.

For instance, I could very easily split some of the “low priority” work like area chat and aws (priority in computational terms here, not importance to the game) out of their various busy processes and use ZeroMQ to offload it to whatever machine isn’t busy through the zmq_queue utility.

I’m also thinking that I could take the current squad management code, which is a little spread out (some of it is on chat, some of it is on the map host, some of it is in the database proxy server), build it as one stand-alone host process and just make it sit the freak there with the entire squad database loaded in memory at all times so it can be truly and wholly authoritative for squad management.

What’s kick ass about this is that using the Java, PHP or Perl ZeroMQ APIs we could have the website interface to the same process so that we could get squad management working on the web again — we can’t do that at the moment because of the argy-bargy between server processes over who is authoritative already.


disclaimer … lol.

[random thought]

when the time comes to think about squad management et al, whenever that may be, might i suggest some sort of calendaring thing? nothing fancy just something really simple although if you want to get fancy allow sister squads (or read lists between HCs and squad COs or etc etc) to share events. would certainly make scheduling events, attacks, RDP raids etc a bit easier. could also see ahead of time who is going to be there for squad night or … etc etc.

just an idea.
[/random thought]

If CRS had more $ think of all the cool things we could do :-)

oo, I forgot about the squad management on the web. Sounds like some good opportunities for the app.

I did my part. Upgraded from gold to plat and an alt to gold.
anyone else throwing into the pot?

Trackbacks and Pingbacks

Instead… « kfsone's pittanceJuly 26, 2010 at 7:37 am

[…] keep harping on about ZeroMQ but I don’t think I’ve yet managed to really convey why it’s so worth […]

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: