stream_file: don't use poll() on directories

POSIX leaves poll() behavior on directories unspecified. While on
Linux, it seems to behave the same way as regular files (always
return immediately), this is not guaranteed. At least with OSX
10.12, it seems to wait, which essentially means that opening
directories will "hang".

Fixes #3530 and #3649.