feat: add wxO deployment adapter (#12079)

* checkout wxo deployment adapter implementation

* checkout wxo deps and flow reqs impl

* clean up / minor refactor with updated tests

* major refactor: split up the implementation into folders and files

* clean up logic

* refactor and clean up of execution logic, improve type safety of "retry_create" helper, and remove dead code

* remove unused wxo service file

* remove "provider_name" arg and references

* fix: harden watsonx orchestrate deployment adapter for PR readiness

- Fix rollback AttributeError when agent_create_response is None
- Fix NoneType access on params in list() when called without params
- Fix inconsistent error types: use DeploymentNotFoundError in get/update
- Fix typo "occured" -> "occurred" in all error prefix messages
- Fix variable shadowing of fastapi.status in get_status()
- Fix pre-existing test bugs (wrong exception types, stale method refs)
- Fix e2e monkey-patching of non-existent service methods
- Add structured logging to create, delete, retry, and rollback flows
- Add jitter to retry backoff to avoid thundering herd
- Add __repr__ to WxOCredentials that fully masks the API key
- Extract hard-coded LLM model string to DEFAULT_WXO_AGENT_LLM constant
- Remove commented-out snapshot update code
- Expand test coverage from 24 to 75 tests covering retry logic,
  service methods, client auth, utilities, execution/status helpers,
  and artifact roundtrip

* fix(deployment): fix bugs and harden watsonx orchestrate adapter

- Fix update method silently discarding changes instead of sending them
  to the WXO API
- Wrap all synchronous SDK calls in asyncio.to_thread to avoid blocking
  the event loop
- Add error handling to get_status (was propagating raw exceptions)
- Use DeploymentType enum values instead of raw strings for
  SUPPORTED_ADAPTER_DEPLOYMENT_TYPES
- Fix type annotations in list method (list -> set)
- Fix typo in comment (reccomend -> recommend)
- Remove dead code: extract_agent_connection_ids,
  require_exclusive_resource, require_non_empty_string,
  resolve_health_environment_id, fetch_agent_release_status,
  normalize_release_status, resolve_lfx_runner_requirement,
  _pin_requirement_name, sync_langflow_tool_connections,
  create_langflow_flow_tool, resolve_snapshot_connections, and unused constants
- Make assert_create_resources_available and validate_connection async
- Make create_agent_run and get_agent_run async
- Add tests for list_types, get_status error handling, and update
  side-effects
- Update existing tests for async function signatures

* actually remove the dead code from tools.py

* properly await "validate_connection"

* update e2e test file

* add more scenarios to e2e test runner

* refactor(watsonx-orchestrate): improve WxOClient encapsulation and error messages
Encapsulate SDK private method access and endpoint paths inside
WxOClient wrappers so callers never handle raw paths or touch internal
_get/_post methods. Route all private HTTP calls through a single
_base client auto-created in __post_init__, removing the externally
constructed `base` field.
Additionally fix double-period and redundant text in error messages
produced by the ErrorPrefix + handler pattern, and update E2E/unit
assertions to match the sanitised error output.
Changes:
- types.py: bake endpoint paths into wrapper signatures (post_run,
  get_run, upload_tool_artifact, get_agents_raw); auto-create _base
  via __post_init__; remove base from constructor
- client.py: drop BaseWXOClient import and base= constructor arg
- core/execution.py: pass run_id / query_suffix instead of raw paths
- core/tools.py: pass tool_id instead of raw path
- service.py: fix double-period in ErrorPrefix interpolation; remove
  redundant restatements in generic except handlers
- tests: update _with_wxo_wrappers helper and test doubles to use
  _base; update FakeBaseClient._get to accept params
- e2e: update rollback scenario detail_contains to match sanitised msg

* skip import and tests of wxo adapter if current env is running python 3.10

* clarify random prefix / retry  behavior

* implement comments

* fix: align watsonx adapter with updated deployment schema
- Add `deployment_type` parameter to `get`, `update`, `redeploy`,
  `duplicate`, `delete`, `get_status`, and `undeploy_deployment` in
  WatsonxOrchestrateDeploymentService to match the updated
  BaseDeploymentService ABC
- Update e2e script to use renamed `add_ids` field on
  SnapshotDeploymentBindingUpdate

* remove non-interface method undeploy_deployment

* implement listing configs and snapshots

* add update implementation and improve http->deployment error translation and add tests

* improve exception handling

* checkout payload slot work

* custom payload schema for update

* new update implementation

* stop passing client cache to helpers

* add docs for ordereduniquests

* improve import patterns and document future risks for wxo dependencies

* remove global-variable prefixing of flows

* ref: harden typing, DRY helpers, and correctness fixes
- Replace `db: Any` with `db: AsyncSession` across client, config, and
  update_helpers modules
- Extract `_ensure_dict` / `ensure_langflow_connections_binding` helpers
  to eliminate repeated nested-dict safety logic in tools.py and
  update_helpers.py, with documentation explaining why malformed API
  payloads are silently corrected rather than rejected
- Use `PayloadSlot.parse()` for update payload validation instead of
  standalone helper; remove `parse_provider_update_payload`
- Fix lambda late-binding with default-argument captures in service.py
- Use `zip(strict=True)` in update_helpers for defensive mismatch
  detection
- Replace O(n²) duplicate detection with `collections.Counter` in
  payloads.py
- Simplify `dedupe_list` to `list(dict.fromkeys(...))`
- Move type-annotation-only imports into TYPE_CHECKING blocks in
  types.py and config.py
- Introduce `UPDATE_MAX_RETRIES` as a distinct constant from
  `CREATE_MAX_RETRIES`
- Fix "watsonX" → "watsonx" casing in error messages
- Clarify `get_status` docstring and `validate_connection` error message
- Add TODO(deployments-cache) for client cache invalidation on
  credential updates

* ref: drop client cache and adopt typed deployment context across the wxo adapter path, add request-context memoization with strict mixed-context guards, and lazily initialize wxo SDK clients. Add safer credential handling by storing only authenticators in WxOCredentials. Also improve provider error-detail extraction/messages and make the direct wxo E2E conflict scenario diagnostics and expectations more resilient.

* use explicit naming for context class; providerId

* fix error handling, retry safety, and exception diagnostics in wxo adapter

- Raise DeploymentError on empty API responses instead of fabricating
  fake success in create_agent_run_result and get_agent_run
- Elevate rollback failure logging from WARNING to ERROR for alerting
- Apply retryable filter to retry_rollback to skip 401/403/409/422
- Preserve exception chains (from exc) instead of suppressing (from None)
  across service.py, utils.py, execution.py, and update_helpers.py
- Broaden credential resolution catch to handle arbitrary DB exceptions
- Separate status code dispatch from string heuristics in
  raise_for_status_and_detail to prevent misclassification
- Add tests for all behavioral changes

* [autofix.ci] apply automated fixes

* fix: harden wxO adapter safety, immutability, and error diagnostics

- Make WxOClient and WxOCredentials frozen dataclasses with eager SDK
  client initialization to eliminate thread-safety races from
  asyncio.to_thread workers and prevent post-construction mutation
- Move instance_url validation and normalization into type __post_init__
- Raise DeploymentError on missing run_id instead of returning partial
  success with execution_id=None
- Preserve exception chains (from exc) instead of suppressing (from None)
  across create, update, and delete service methods
- Add warning logs when _ensure_dict replaces non-dict binding values
  and when _resolve_lfx_requirement falls back to minimum version
- Initialize derived_spec before try block to prevent potential NameError
- Log all ToolUploadBatchError errors before re-raising the first
- Change SUPPORTED_ADAPTER_DEPLOYMENT_TYPES from mutable set to frozenset
- Add tests for 409/422 error mapping in create, unsupported deployment
  type rejection, empty update rejection, zip artifact extraction paths,
  validate_connection negative paths, missing run_id, multiple deployment
  ID rejection, exception chain preservation, and _ensure_dict warning

* [autofix.ci] apply automated fixes

* fix ruff errors and and todo for status method

* fix mypy errors

* [autofix.ci] apply automated fixes

---------

Co-authored-by: Jordan Frazier <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>