Revert "server: ensure frontend has correct port for PTY websocket connections (#5898)"

This reverts commit a05915ddc8f34d592c05ac6a4f4e8d932a9a0964.