Revert "Fix quic-go leaking connections"

This reverts commit fcad2b6ed66e9a17e0eeba06f43a54b265b981a5.