Merge pull request #138559 from briantwatson/refactor-annotate-remove-factory

Refactor annotate to remove factory dependency

Kubernetes-commit: e55a0716270455f04d023e35387db898c623d429