recursive_isinstance(), recursive_issubclass(): New code here returned

NULL in case of error, but the functions are declared to return int.
MSVC 6 properly complains about that.  Return -1 on error instead.