rtmp-services: Enforce encoder settings per protocol

For mpegts output, used in rtmp-custom service, encoder settings
require:
- ADTS with fdk-aac.
For all non-rtmp protocols, one also requires:
- video encoder header repetition;
This future proofs the code against protocol addition.

Additionally, I've added a check against NULL audio settings, which was
in my fork but that I forgot to add in the PR [1].

[1] https://github.com/obsproject/obs-studio/pull/6163

Signed-off-by: pkv <pkv@obsproject.com>