Actually remove directories from sys.path if they do not exist; the intent

is to avoid as many stat() calls as we can.