correct `single_match` lint suggestion (#13824)

fix #12758

The `single_match` lint makes incorrect suggestions when comparing with
byte string literals. There seems to be an issue with the logic that
calculates the number of refs when creating suggestions.

changelog: [`single_match`]: No longer make incorrect suggestions when
comparing with byte string literals.