Restore back-off on listener setup failure

Supports exponential back-off as well now.

Closes gh-36809