There was a race condition where the connector would try to connect

before the listener was ready (on gentoo x86 buildslave).  This
caused the listener to not exit normally since nobody connected to it
(waited in accept()).  The exception was raised in the other thread
and the test failed.

This fix doesn't completely eliminate the race, but should make it
near impossible to trigger.  Hopefully it's good enough.