Added tests for tokenize, blocked corresponding checkin from trunk.