kfsone
·
July 21, 2010
·
Coding
·
parallelization, zeromq
The elegance of ZeroMQ messaging is that it provides easy scalability. The API for 0MQ sockets is the same whether you are doing in-process (inter thread) communication, inter-process communication, peer-to-peer network communication or multicast communication.
As long as you are putting data and not pointers in your messages (*cough* Async::Worker *cough*), you convert your code from in-process communication to cross-network with a one line change:
// Create the socket: same code in either case.
zmq::socket_t outSocket(zmqContext, ZMQ_REP) ;
...
outSocket.connect("in-proc://my-connection") ;
// becomes
outSocket.connect("tcp://192.168.0.65:2342") ;
and voila – your application is networking instead of talking to itself.
They also provide three external utilities that make it a doddle to scale your application across multiple machines.
kfsone
·
July 21, 2010
·
Coding, General
Some time ago, I was looking at tools for building graphs and charts from database tables. Someone sent me a link to a commercial product that was aimed at delivery to non-programmers, allowing you to drill down to pre-existing graphs/charts and even customize queries to build your own.
The ideal thing for handing off charts to management and producers.
We didn’t have a perceived need for it at the time and I can’t find the freakin’ link for the life of me (sorry, Bloo). No, it wasn’t jpgraph (which is a programmer tool).
Any ideas?
I might be misremembering that it was web-based, it might have been an actual application. And no, I’m not thinking of Excel ;)
kfsone
·
July 21, 2010
·
Coding
·
async, parallelization, zeromq
I’ve put the source to my Async::Worker system, documentation and examples here.
From the examples, how to offload batches of work for processing in parallel:
int main(int argc, const char* const argv[])
{
static const size_t NumberOfElements = 20000000 ;
static const size_t GroupSize = 8192 ;
Numbers numbers ;
numbers.resize(NumberOfElements) ;
for ( size_t i = 0 ; i < NumberOfElements ; ++i )
{
numbers[i] = (rand() & 65535) + 1 ;
}
uint64_t parallelResult = 0 ;
// Dispatch groups of numbers to workers.
Numbers::iterator it = numbers.begin() ;
do
{
Numbers::iterator end = std::min(it + GroupSize, numbers.end()) ;
Async::Queue(new CrunchNumbersRange(it, end, ¶llelResult)) ;
it = end ;
}
while ( it != numbers.end() ) ;
// Wait for all the results, calling Result() on each
// returned object to produce a total.
Async::GetResults() ;
printf("Done. Calculated sum as %lu.\n", (unsigned long int)parallelResult) ;
return 0 ;
}
kfsone
·
July 21, 2010
·
Coding
·
parallelization, zeromq
ZeroMQ is the messaging infrastructure I mentioned a little while back.
I’ve had a little opportunity to dabble with it now and, I have to say, I’ve taken to it. The interface is really nice and lean. It’s “core standard” too – it looks like sockets, it plays like sockets. It plays nicely with real sockets. The O/S can schedule around it like sockets – which is a huge boon on just about every OS running today.
And it’s incredible frugality and minimalism helps achieve impressive performance: one of my (-O0) unit tests manages to pump an incredible 65,000 messages from one thread and back to the original thread in under 1 millisecond, running on a virtual Ubuntu 10.04 on a physical core-2-duo.
Recent Comments