Set TESTFN_UNENCODEABLE on non-Windows OSes

* Use 0xff byte on non-Windows OSes
 * mbcs is now really strict by default: i closed the issue #850997, so use the
   filesystem encoding and not Latin-1
 * Rename TESTFN_UNICODE_UNENCODEABLE to TESTFN_UNENCODEABLE