gh-111246: Remove listening Unix socket on close (#111483)

Try to clean up the socket file we create so we don't add unused noise to the file system.