Fix incorrect locking and attempts check in ShadowTree experiment (#52681)

Summary:
Pull Request resolved: https://github.com/facebook/react-native/pull/52681

Changelog: [internal]

In the original change I made in D78418504 / https://github.com/facebook/react-native/pull/52645 I made 2 mistakes:
1. Used a lock that would try to re-lock on itself without it being recursive (which would cause a deadlock). I didn't see that because when testing I didn't hit the case where we'd exhaust the options.
2. The `attemps` variable wasn't incremented, so we never left the loop in case of exhaustion.

This propagates a flag to `tryCommit` to indicate we've already locked on the commitMutex_ so we don't need to lock again in that case and increases the counter, fixing the issue.

Reviewed By: cortinico

Differential Revision: D78497509

fbshipit-source-id: 546ccd0c84aed5416ce1aef47d79419b4fe06f66