ZeroMQ and scalability

The elegance of ZeroMQ messaging is that it provides easy scalability. The API for 0MQ sockets is the same whether you are doing in-process (inter thread) communication, inter-process communication, peer-to-peer network communication or multicast communication.

As long as you are putting data and not pointers in your messages (*cough* Async::Worker *cough*), you convert your code from in-process communication to cross-network with a one line change:

// Create the socket: same code in either case.
zmq::socket_t outSocket(zmqContext, ZMQ_REP) ;

...

outSocket.connect("in-proc://my-connection") ;
// becomes
outSocket.connect("tcp://192.168.0.65:2342") ;

and voila – your application is networking instead of talking to itself.

They also provide three external utilities that make it a doddle to scale your application across multiple machines.

In Defense of Nettersheim, Finale

The finale to the Nettersheim series, hope you enjoy :)

Query and Chart Builder?

Some time ago, I was looking at tools for building graphs and charts from database tables. Someone sent me a link to a commercial product that was aimed at delivery to non-programmers, allowing you to drill down to pre-existing graphs/charts and even customize queries to build your own.

The ideal thing for handing off charts to management and producers.

We didn’t have a perceived need for it at the time and I can’t find the freakin’ link for the life of me (sorry, Bloo). No, it wasn’t jpgraph (which is a programmer tool).

Any ideas?

I might be misremembering that it was web-based, it might have been an actual application. And no, I’m not thinking of Excel ;)

Async::Worker: Parallelism with ZeroMQ

I’ve put the source to my Async::Worker system, documentation and examples here.

From the examples, how to offload batches of work for processing in parallel:

    int main(int argc, const char* const argv[])
    {
        static const size_t NumberOfElements = 20000000 ;
        static const size_t GroupSize = 8192 ;
        Numbers numbers ;
        numbers.resize(NumberOfElements) ;
        for ( size_t i = 0 ; i < NumberOfElements ; ++i )
        {
            numbers[i] = (rand() & 65535) + 1 ;
        }

        uint64_t parallelResult = 0 ;

        // Dispatch groups of numbers to workers.
        Numbers::iterator it = numbers.begin() ;
        do
        {
            Numbers::iterator end = std::min(it + GroupSize, numbers.end()) ;
            Async::Queue(new CrunchNumbersRange(it, end, &parallelResult)) ;
            it = end ;
        }
        while ( it != numbers.end() ) ;

        // Wait for all the results, calling Result() on each
        // returned object to produce a total.
        Async::GetResults() ;

        printf("Done. Calculated sum as %lu.\n", (unsigned long int)parallelResult) ;

        return 0 ;
    }

More on ZeroMQ

ZeroMQ is the messaging infrastructure I mentioned a little while back.

I’ve had a little opportunity to dabble with it now and, I have to say, I’ve taken to it. The interface is really nice and lean. It’s “core standard” too – it looks like sockets, it plays like sockets. It plays nicely with real sockets. The O/S can schedule around it like sockets – which is a huge boon on just about every OS running today.

And it’s incredible frugality and minimalism helps achieve  impressive performance: one of my (-O0) unit tests manages to pump an incredible 65,000 messages from one thread and back to the original thread in under 1 millisecond, running on a virtual Ubuntu 10.04 on a physical core-2-duo.