Waiting for the right moment – in integration testing

When you have to test multi-threaded programs, there is always the need to wait until the system arrives at a particular state, at which point the test can verify that the proper state has been reached.

The usual way to do it is to insert a “probe” in the system which will signal a synchronization primitive (like a Semaphore) and the test waits until the semaphore gets signaled or a timeout passes. (Two things which you should never do – but are a frequent mistake – is to insert sleeps into your code – because they slow you down and are fragile – or to use the Object.wait method without looping around it – because you might get spurious wakeups which will result in spurious, hard to diagnose and very frustrating test failures).

This is all nice and good (although a little verbose – at least until the Java 8 lambdas arrive), but what if the second thread calls a third thread and doesn’t wait for it to finish, but in the test we want to wait for it? A concrete example would be: an integration test which verifies that a system composed out of a client which communicates trough a messaging middleware with a datagrid properly writes the data to the datagrid? Of course we will use a mock middleware and a mock datagrid, thus the startup/shutdown and processing will be very fast, but they would be still asynchronous (suppose that we can’t make it synchronous because the production one isn’t and the code is written such that it relies on this fact).

The situation is described visually in the sequence graph below: we have the test running on T0 and we would like for it to wait until the task on T3 has finished before it checks the state the system arrived to.

We can achieve this using a small modification to our execution framework (which probably is some kind of Executor). Given the following interface:

public interface ActivityCollector {
        void before();        
        void after();
}

We would call before() at the moment a task is enqueued for execution and after() after it has executed (these will usually occur on different threads). If we now consider that before increments a counter and after decrements it, we can just wait for the counter to become zero (with proper synchronization) at which point we know that all the tasks were processed by our system. You can find an Executor which implements this here. In production you can of course use an implementation of the interface which does nothing, thus removing any performance overhead.

Now lets look at the interface which defines how we wait for the “processed” condition:

interface ActivityWatcher {
        void await(long time, TimeUnit timeUnit);
}

Two personal design choices used here were: only provide a way to wait for a specific time and no longer (if the test takes too long that’s probably a performance regression one needs to take a look at) and to use unchecked exceptions to make testing code shorter.

A final feature would be to collect exceptions during the execution of the tasks and abort immediately if there is an exception somewhere rather than timing out. This means that we modify our interface as follows:

public interface ActivityCollector {
        void before();        
        void after();
        void collectException(Throwable t);
}

And the code wrapping the execution would be something like the following:

try {
        command.run();
} catch (Throwable t) {
        activityCollector.collectException(t);
        throw t;
} finally {
        activityCollector.after();
}

You can find an implementation of ActivityWatcher/ActivityCollector here (they are quite linked, thus the one class implementing them both). Happy testing!

A couple of caveats:

• This requires some modification to your production code, so it might not be the best solution (for example you can try creating synchronous mocks of your subsystems and do testing that way).
• This solution is not well suited for cases where Timers are involved because there will be times when “no tasks are waiting”, but in fact a task is waiting in a timer. You can work around this by using a custom timer which calls “before” when scheduling and “after” at the finish of the task.
• The same issue can come up if you are using network communication for more authenticity (even if it is inside of the same process): there will be a moment when no tasks are scheduled because they are serialized in the OSs network buffer.
• The ActivityCollector is a single point of synchronization. As such it might decrease performance and it might hide concurrency bugs. There are more complicated ways to implement it which avoids some of the synchronization overhead (like using a ConcurrentLinkedQueue), but you can’t eliminate it completely.

PS. This example is based on an IBM article I can’t seem to find (dear lazyweb: if somebody finds it, please leave a comment – before/after were called tick/tock in it) as well as work by my colleagues. My only role was to write it up and synthesize it.

Author: gpanther