mac-videotoolbox: Fix handling of unsuccessful encoder creation

When an encoder was not created in create_encoder, the appropriate
OSStatus value is returned but the calling code expects a boolean
return value.

The negative OSStatus code sent on error is thus interpreted as a
truthy value and the error is not detected. Changing the call signature
to correctly return an OSStatus (and change the caller to detect
error situations) fixes this.