Automating unit test generation?

Currently I’m using a homespun Unit Test system that I developed before I looked at any others so I would have a clear idea of my own pre-conceptions when reviewing The Real Thing.

Some time ago, I found this excellent article on C++ Unit Testing. Given that I created my system for a similar benchmarking purpose, I still find it clunky.

Despite Noel’s indepth review, I still find all of those he linked, and the one he himself has since created, just as clunky. I feel we both set the bar too low for an “ideal” testing suite.

Maybe this is the maintenance programmer in me, but I think Unit Testing and documentation should go hand-in-hand.


Maybe I notice it more because I’m just wetting my feet in Doxygen, but it’s always bothered me that I have to “write” tests for all the minutae. Consider the following:

#include "tests/testSuite.h"

#include "include/async-worker.h"

//! Worker that changes the value of a variable.
class TestWorker1 : public Async::FireAndForget
{
private:
  TestWorker1() { throw std::logic_error("Default constructor called") ; }
public:
  TestWorker1(volatile unsigned int* dest, unsigned int value, volatile bool* destructorTracker)
    : Async::FireAndForget()
    , m_dest(dest)
    , m_value(value)
    , m_destructorTracker(destructorTracker)
  {
    ASSERT( ExpectFalse( *m_destructorTracker ) ) ;
  }
  ~TestWorker1()
  {
    ASSERT( ExpectNotEqual( m_destructorTracker, NULL ) ) ;
    ASSERT( ExpectFalse( *m_destructorTracker ) ) ;
    *m_destructorTracker = true ;
  }
  virtual bool Work() const { *m_dest = m_value ; }
  volatile unsigned int* m_dest ; //!< Address to write to.
  unsigned int m_value ;  //!< *Value* to apply to m_dest.
  virtual bool* m_destructorTracker ; //!< Set to true when destructor called.
} ;

class AsyncWorkerTest : public TestUnit
{
public:
  AsyncWorkerTest() : TestUnit("AsyncWorker") {}
  virtual bool Run()
  {
    START_TEST_STAGE( "Work Execution" )
      static const unsigned int testPattern = 0xa37e8800ff ;
      volatile unsigned int testDest = 0 ;
      volatile bool destructorCalled = false ;
      TestWorker1* worker = new TestWorker1(&testDest, testPattern, &destructorTracker) ;

      START_SUB_STAGE( "Initialization" )
        // Sanity checks.
        ExpectEqual( testDest, 0 ) ;
        ExpectEqual( worker->m_dest, &testDest ) ;
        ExpectEqual( worker->m_value, testPattern ) ;
        ExpectFalse( destructorCalled ) ;
      END_SUB_STAGE( "Initialization" )

      START_SUB_STAGE( "Dispatch" )
        // Dispatch the workload to a thread.
        worker->Queue() ;
      END_SUB_STAGE( "Dispatch" )

      START_SUB_STAGE( "Execution" )
        // Should take less than a microsecond to run, but
        // we'll wait up to half a second to give it chance.
        // (fac(1000) is close to 500,000)
        size_t waited = 0 ;
        for ( size_t i = 0 ; *testDest == 0 && waited < 500000 ; waited += i, ++i )
        {
          OpenThreads::YieldCurrentThread() ;
          // Incremental wait.
          OpenThreads::CurrentThread->microSleep(i) ;
        }

        // Test for completion.
        ExpectEqual( *testDest, testPattern ) ;

        // Test we didn't wait too long.
        static const size_t MaximumMicroSecondWaitExpected = 5 ; // 5 microseconds is way too long.
        Expect( waited, <, MaximumMicroSecondWaitExpected ) ;

        // Produce a warning if we actually had to wait more than once, if at all.
        IDEALLY( ExpectLt( waited, 2 ) ) ;

        // Destructor should have been called.
        ExpectTrue( destructorCalled ) ;
      END_SUB_STAGE( "Execution" )
    END_TEST_STAGE( "Work Execution" )
 }
} ;

(My system has the concept of stages and sub-stages, where sub-stages can be nested; that allows you to avoid the overhead of fixtures, the downside of which is it leads to “Run” functions which are arguably too long)

The need to write tests for things ought not to be so entirely decoupled so from the original source or the advertised API. And it should be somewhat language agnostic.

My thinking is that a large degree of the basic testing ought to be able to be written as part of the documentation/prototyping process itself.

namespace Async  //!< Houses classes that encapsulate work-offloading technique with ZeroMQ.
{
  /*!
   * Workload that can safely be executed at some
   * future point in the background and delete itself.
   *
   * Inherit to your own class, marshall with all data
   * that will be used and implement the virtual bool Work
   * function ensuring all work is thread safe.
   *
   * Call the worker->Queue() function on a pointer to
   * an instance to hand off to the worker pool.
   *
   * @seealso RunAndReturn
   * @seealso PooledProcessing
   */
  class Payload
  {
  public:
    //! Default constructor.
    //? [constructor] m_hasRun equals false
    Payload() : m_hasRun(false) {}

  public:
    // (Note: This code wouldn't work ;-P)
    //! Determine if the object should auto-destruct.
    //? [member] DiscardOnCompletion() equals true or false
    virtual bool DiscardOnCompletion() const = 0 ;

  public:
    //! Pass workload to the worker pool.
    //! If DiscardOnCompletion is true, the
    //! object will self-delete after usage.
    /*?
     *? [member] Queue()
     *? [wait upto 500 miliseconds, < 1 microsecond ideal, 5 microseconds too long] m_hasRun equals true
     */
    void Queue() const ;

  public:
    //! Encapsulates the work load to be executed
    //! asynchronously.
    //! @return true if work completed successfully, otherwise false.
    virtual bool Work() = 0 ;

  private:
    bool m_hasRun ; //!< Purely to demonstrate constructor test.
  } ;
}

/*?
 *  "optional" so it doesn't fail when being tested by a 3rd-party.
 *? [optional] @include tests/longWindedAsyncTest.tests
 *
 *  A language-specific test example:
 *[ Lua:
 *  require 'async-worker'
 *  myWorker = Async.Work:new()
 *  myWorker.worked = false
 *  function myWorker:Work()
 *      self.worked = true
 *      return true
 *  end
 *  function myWorker.DiscardOnComplete()
 *    return false
 *  end
 *  myWorker.Queue()
 *?  [wait upto 500 miliseconds, < 1 microsecond ideal, 5 microseconds too long] myWorker.worked equals true
 *]
 */

It needs to be a pseudo-natural language so that it “reads” as the documentation does, describing constraints and API expectations. For example, documenting a function that returns 3 values:

// f(N) returns 0 if N equals 0,
// returns 1 if N equals 1,
// else returns 4
// int f(int N) { if ( n == 0 ) return 0 ; if ( n == 1 ) return 1 ; return 3 ; }
[/sourecode]

(good, you were paying attention)

Just writing this made me aware of some of the reasons why we perhaps don’t already see something like this: the case of the constructor forced me to stop and think.

But still, sharper minds than mine must already have toiled at this and there has to be something like this out there already, skulking in the dark recesses of the web that Google just doesn’t frequent?

Am I about to rush off and try and create this myself? Not this time; for a start, I don’t much feel like running off and creating my own C/C++ parser; something like this would probably make a good Doxygen extension, though.

One Comment

Not exactly what you are talking about, but “natural language” specifications of tests is what Cucumber does in the RUby world:

I haven’t used it so I cannot comment much, but I thought of the approach so I thought you might find it interesting.
S!

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: