feat: add core deployment implementation (#12108)

* checkout api handlers

* add missing table

* update to use "version" terminology for flows instead of outdated "history" verbage

* Fix provider account id mapping

* recover a little todo comment

* Add flow version migration and minor exception handling, etc

  1. deployments.py — Added AuthenticationError import + handling (401) in all 9 error handler blocks. Added update_deployment_db import + call to persist name changes to local DB after adapter update succeeds.
  2. flow_version/exceptions.py — Added FlowVersionDeployedError for blocking deletion of deployed versions.
  3. flow_version/crud.py —
    - Added has_deployment_attachments() helper that checks if a flow version has any deployment attachments
    - delete_flow_version_entry() now raises FlowVersionDeployedError if the version is attached to deployments
    - Pruning now excludes deployed versions via a NOT IN subquery on FlowVersionDeplottachment, preventing provider-side snapshot orphaning
  4. flow_version/__init__.py — Exports FlowVersionDeployedError
  5. flow_version.py (API route) — Imported FlowVersionDeployedError, added it to _translate_version_error as 409 Conflict
  6. models/__init__.py — Registered FlowVersionDeploymentAttachment for alembic/SQLModel metadata detection
  7. Migration c0d2ce43b315 — Creates flow_version_deployment_attachment table with all columns, FKs (CASCADE), and indexes. Single head, chains off fc7f696a57bf.

* [autofix.ci] apply automated fixes

* address bugs and inconsistencies

* rename column to "provider_snapshot_id"

* harden deployment API: fail loudly on invalid state instead of silently passing

Replace silent fallbacks, warning logs, and dropped values with hard
failures (422/500/502) across the deployment orchestration layer. Adds
logging to bare exception handlers and wraps materialize_snapshots in
the adapter error handler.

* [autofix.ci] apply automated fixes

* add materialize_snapshots to deployment service protocol

Promotes materialize_snapshots from duck-typed getattr usage to a
first-class method on DeploymentServiceProtocol, BaseDeploymentService,
and the no-op DeploymentService stub. Introduces MaterializeSnapshotsResult
schema for typed return values.

Updates deployments.py to call the method through the protocol instead of
getattr, giving static analysis coverage over the contract.

Documents the snapshot abstraction across BaseFlowArtifact, SnapshotItem,
MaterializeSnapshotsResult, and FlowVersionDeploymentAttachment.provider_snapshot_id
— explaining that snapshots are immutable, provider-owned copies of flow data
with opaque provider-assigned identifiers (e.g. wxO tool ID, K8s ConfigMap
name, S3 key).

* deployment sync: extract helpers, server-side type filter, orphan detection

- Extract _fetch_provider_resource_keys helper for provider validation
- Rename _sync_page_with_provider → _list_deployments_synced
- Match resource keys by provider ID only (not name)
- Restore server-side deployment_type filter with guard against
  false deletions (skip rows whose type doesn't match the filter
  instead of deleting them)
- Add orphan/divergence logging for post-create, post-update,
  post-duplicate DB write failures
- Return 422 on invalid UUID in update remove list (was silently ignored)
- Handle NotImplementedError → 501 from provider adapters
- Convert attachment IntegrityError to ValueError with descriptive message
- Add tests for sync helpers (15 cases)

* rebase on release-1.9.0 and align with lfx/services

* refactor(deployments): align provider mapper routing and WXO update payload mapping

Align deployment mapper resolution with adapter-type/provider-key routing and
refactor PATCH update handling to use mapper resolve/shape contracts end-to-end.
Map Langflow flow_version_id references at the API boundary into provider update
operations for Watsonx bind/unbind/remove paths, with expanded mapper tests.

* patch down-revision

* first pass with formalized boundary rules

* fix(deployments): harden watsonx payload boundary contracts

Enforce fail-fast payload slot parsing for required adapter results, split execution create/status slot contracts, and route execution-create mapping through deployment mappers.
Require watsonx flow artifact source_ref and move update reconciliation output to provider_result to keep mapper/adapter boundaries explicit and typed.

* refactor(deployments): modularize watsonx orchestrate create/update flow

Extract create/update logic into dedicated core modules with shared helpers to tighten deployment boundary contracts.
Align backend/lfx payload schema mapping and expand e2e/unit coverage for response mapping and update schema behavior.

* api impl for wxo-specific create payload

* further refactoring. add rollback of existing tools (undo new app bindings) in the create path

* add todo in execution.py

* refactor(deployments): replace snapshot_id/reference_id with source_ref-correlated tool refs
Introduce WatsonxToolRefBinding to correlate source_ref (flow version id)
with provider tool_id across all operation types. This replaces the prior
reference_id and snapshot_id fields with a unified structure that carries
provenance through create, bind, unbind, and remove_tool operations.
Key changes:
- Flatten API operation payloads: hoist flow_version_id onto operations,
  remove nested WatsonxApiUpdateToolReference wrapper
- Replace tools.existing_ids with inline tool_id_with_ref on bind operations
- Rename WatsonxCreateSnapshotBinding to WatsonxToolRefBinding (input) and
  WatsonxResultToolRefBinding (output, with created flag)
- Add created_app_ids to update results for connection tracking
- Raise HTTPException on contract violations in _to_api_tool_app_bindings
  instead of silently dropping unmappable bindings
- Add schema-level validation for conflicting source_ref on same tool_id
- E2E: cache tool_id→source_ref from create results, use helpers to build
  refs with distinct source_ref vs tool_id values

* Enforce DeploymentType enum and add description column

Make deployment_type a required column backed by a SQLAlchemy
TypeDecorator that validates on write (rejects None and invalid strings)
and coerces to DeploymentType on read. Add nullable description column
to the deployment model and surface it through the API.

Key changes:
- Add _DeploymentTypeColumn TypeDecorator for enum round-trip fidelity
- Make deployment_type non-optional in Deployment model, DeploymentRead,
  CRUD functions, and API layer
- Add description (Text, nullable) to Deployment model and fold its
  migration into the existing c0d2ce43b315 revision
- Remove _resolve_deployment_type helper — TypeDecorator handles coercion
- Remove DeploymentType fallbacks and backward-compat shims from API
  endpoints, base mapper, and watsonx orchestrate mapper
- Document cross-package coupling: DeploymentType is owned by lfx but
  persisted by langflow; member values must never be removed
- Fix pre-existing bug: provider_account_id → deployment_provider_account_id
  in get_deployment_status endpoint

Note: deployment_type is nullable=True at the DB level to satisfy the
EXPAND-phase migration validator; NOT NULL is enforced at the application
layer by the _DeploymentTypeColumn TypeDecorator.

* refactor(deployments): extract route helpers, harden sync and error handling
Move bulk of deployment route logic into mappers/helpers layer to slim
down deployments.py and enforce clearer boundary between routes, mappers,
and adapters (documented in DEPLOYMENT_BOUNDARY_RULES.md).
Key changes:
- Extract ~700 lines from deployments.py into helpers.py (pagination,
  adapter/mapper resolution, attachment management, snapshot sync,
  rollback, response shaping)
- Add read-path snapshot-level sync: get_deployment and
  list_deployments_synced verify provider_snapshot_ids against the
  provider and prune stale attachments, with graceful fallback on error
- Add compensating rollback for create (rollback_provider_create) and
  update (rollback_provider_update) when DB commit fails after provider
  mutation, using mapper-driven payload reconstruction
- Introduce handle_adapter_errors() context manager centralising
  DeploymentServiceError → HTTP status mapping via
  http_status_for_deployment_error; sanitise 500 detail to avoid
  leaking internals
- Add DeploymentNotConfiguredError → 503 mapping
- Add util_snapshot_ids_to_verify and resolve_rollback_update to base
  mapper with WxO overrides for provider-specific snapshot ID extraction
  and put_tools-based rollback payloads
- Add put_tools field to WatsonxDeploymentUpdatePayload for full tool
  list replacement; early-return in build_provider_update_plan and
  validate_operation_references when put_tools is set
- Extract verify_tools_by_ids into core/tools.py helper
- Harden resource_name_prefix with strip_whitespace + min_length=1
- Deduplicate snapshot_ids before provider calls
- Add deterministic order_by(created_at) to attachment CRUD queries
- Add exc_info=True to all best-effort rollback/compensate error logs
- Add session.rollback() in get_deployment snapshot sync error path
- Warn when list_snapshots receives both deployment_ids and snapshot_ids
- Add E2E scenarios for empty snapshot list, mixed snapshot IDs, tools
  endpoint, and deployment re-list after update
Tests:
- Add test_deployment_route_handlers.py covering stale-row delete +
  commit, non-404 adapter errors, handle_adapter_errors wiring,
  snapshot sync (happy path, skip, error fallback), project-scoped
  flow version validation for create and update
- Expand test_deployment_sync.py with snapshot-phase tests, rollback
  tests, pagination guard, and project-scoped validation
- Add deployment_type assertion to response mapping test
- Add DeploymentNotConfiguredError and bare DeploymentServiceError cases
  to exception mapping tests
- Add put_tools schema and update plan tests

* remove pompous performance commentary

* fix(deployments): remove upsert behavior and fail fast on duplicate name

The create_deployment endpoint previously performed a
get_deployment_by_resource_key lookup before inserting the DB row,
silently tolerating duplicates. Since the provider adapter returns a
fresh resource ID on every create, this lookup could never legitimately
match — and if it did, it would mask a data inconsistency bug.

Changes:
- Remove the get-or-create (upsert) pattern; go straight to
  create_deployment_db and let the unique constraint surface conflicts.
- Add deployment_name_exists CRUD function and an early 409 guard so
  duplicate names are rejected before any provider call, avoiding
  costly provider-side rollback for a locally-checkable condition.
- Update existing route-handler tests to reflect the removed lookup.
- Add tests for deployment_name_exists and the 409 duplicate-name path.

* feat: convert provider_key and deployment_type columns to DB-level enums
Replace plain string columns with SQLAlchemy Enum types backed by
Postgres/SQLite enum constraints, enforcing valid values at the DB
layer rather than only in application code.
Migration follows expand-contract pattern (add enum column, backfill,
drop old string column, rename) with index ops outside batch context
to avoid SQLite column-lookup issues. Upgrade and downgrade are fully
atomic.
- Add DeploymentProviderKey enum as single source of truth for
  provider identifiers; remove magic strings and _DeploymentTypeColumn
  TypeDecorator
- Make provider_key immutable after creation (remove from update
  request schema and API handler)
- Fix pre-existing test gap: add missing deployment_type argument to
  all TestDeploymentCRUD create_deployment() calls

* feat(flow-versions): add deployment awareness and sync-on-read

Add is_deployed field to flow version responses and provider-verified
sync to the list/get read paths, matching the existing deployment
endpoint sync pattern.

API changes:
- list_flow_versions returns is_deployed per entry and accepts optional
  deployment_ids filter to scope by specific deployments
- get_single_flow_version returns is_deployed on the full response
- Both endpoints now use write sessions and run best-effort
  snapshot-level sync before returning results

Sync-on-read (helpers.sync_flow_version_attachments):
- Queries all attachments for a flow's versions joined with deployment
  and provider account info in a single query
- Groups by provider, resolves adapter/mapper per group, verifies
  provider_snapshot_ids via list_snapshots, and prunes stale attachment
  rows through the existing sync_attachment_snapshot_ids helper
- Per-provider error handling so one provider outage doesn't block the
  response

New CRUD: list_attachments_for_flow_with_provider_info joins
FlowVersionDeploymentAttachment -> Deployment -> DeploymentProviderAccount
to avoid N+1 queries during sync grouping.

Tests: 50 passing (11 new) covering is_deployed indicator, deployment_ids
filtering, stale attachment pruning on list/get, and sync failure
resilience.

* Add user id authentication to a few missing endpoints

* [autofix.ci] apply automated fixes

* Update tests

Removes some mock tests that did nothing
Adds sqlite for true testing of db accessors
Reduces Mock objects usage

* refactor(deployments): enforce ownership boundaries on execution responses
Move provider-owned execution identifiers (execution_id, agent_id,
status, timestamps, errors) out of the top-level API response and into
provider_data, keeping only Langflow-owned fields (deployment_id) at the
top level.  This prevents future collisions if Langflow introduces its
own execution tracking.
Key changes:
- Remove execution_id from _ExecutionResponseBase; provider's opaque
  run identifier now lives exclusively inside provider_data
- Rename WatsonxExecutionResultData → WatsonxAgentExecutionResultData
  (adapter layer) and split the API-layer class into a private base
  (_WatsonxApiAgentExecutionResultBase) with dedicated
  WatsonxApiAgentExecutionCreateResultData and
  WatsonxApiAgentExecutionStatusResultData subclasses
- Translate WXO run_id → execution_id at the adapter boundary
  (create_agent_run_result / get_agent_run).
- Collapse util_execution_id + util_execution_deployment_resource_key
  into a single util_resource_key_from_execution that trusts the
  adapter-provided result.deployment_id directly
- Remove build_orchestrate_runs_query and extra payload fields
  (thread_id, llm_params, guardrails, etc.) unused in MVP
- Simplify WxOClient.post_run signature (drop query_suffix)
- Exclude provider_data from flow tool artifact to avoid unexpected
  top-level keys in the WxO tool runtime
- Document ownership boundary rules in DEPLOYMENT_BOUNDARY_RULES.md §14
- Add E2E polling for terminal execution status, input format variants,
  and missing-deployment negative test
- Expand unit tests for renamed schemas, field mapping, passthrough
  validation, and simplified payload builder

* feat: add name column to deployment_provider_account
Add a required, user-chosen display name to provider accounts
(e.g. "staging", "prod") that is unique within a given provider_key.
Includes model, CRUD, API schema/route, mapper, migration with
backfill, and tests.

* fix: replace hand-written migration with Alembic-generated revision
Replace the manually authored migration (b4e6f8a2c1d3) with an
Alembic-generated one (8255e9fc18d9) for the deployment_provider_account
name column.
Fix SQLite compatibility: sa.func.concat() generates a concat() function
call which does not exist in SQLite. Use sa.literal().concat() instead,
which produces the || operator and works on both PostgreSQL and SQLite.

* remove bogus unverified math from migration file

* feat: verify provider credentials before account creation
Add a verify_credentials step to the provider account creation flow
that validates API keys against the provider before persisting them.
This prevents storing invalid or revoked credentials and gives users
immediate feedback.
Key changes:
- Add verify_credentials to the deployment adapter interface (base,
  service, protocol) with WXO implementation that obtains a token
  via the IBM authenticator
- Add SSRF-hardened URL validation for provider_url (HTTPS-only,
  private IP blocklist, localhost rejection, normalization)
- Introduce ValidatedUrl/ValidatedUrlOptional annotated types in
  the API schema layer
- Refactor raise_for_status_and_detail to accept an optional cause
  parameter for explicit exception chain control
- Use ResourceNotFoundError (parent) instead of DeploymentNotFoundError
  in raise_for_status_and_detail for provider-agnostic 404 mapping
- Narrow get_authenticator return type to concrete union

* use whitelist only for valid urls

* feat: BREAKING: move credentials into provider_data and centralize update logic in mapper
- Replace top-level `api_key: SecretStr` with opaque `provider_data: dict` in API schemas;
  mapper extracts credentials via `resolve_credential_fields` for DB storage
- Add `resolve_provider_account_update` to base mapper so routes delegate
  full update-kwargs assembly (including cross-field logic) to the mapper
- WXO mapper override re-derives `provider_tenant_id` when `provider_url` changes
- Add tenant extraction utilities and `validate_tenant_url_consistency` in
  `deployment_provider_account/utils.py` as single source of truth
- Add `model_validator` on `DeploymentProviderAccount` for defense-in-depth
  tenant/URL consistency checks
- Rename `DEPLOYMENT_BOUNDARY_RULES.md` → `RULES.md`; document DB-direction
  mapper contract, credential flow, and update assembly
- Update all tests for new `provider_data` shape, mapper update methods,
  tenant extraction, and model-level consistency validation

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* fix(deployments): Harden provider account validation and WXO rollback

Use provider_url when resolving WXO credentials and scope provider account names per user within each provider. Re-verify provider account updates before persisting them and return 4xx responses for account conflicts instead of surfacing 500s.

Block deleting provider accounts that still own deployments and extend create rollback so failed WXO writes clean up provider-side resources. Add route, schema, mapper, sync, CRUD, and WXO tests to cover the new behavior.

* [autofix.ci] apply automated fixes

* update url validator docs; remove outdated reference to private url blacklist logic

* fix(deployments): Isolate snapshot sync writes

Use nested transactions around best-effort snapshot cleanup so provider sync failures cannot leak partial attachment deletes into the outer request transaction.

Persist explicit description clears during deployment PATCH requests, and add regression tests for both deployment sync paths and the route handler update flow.

* fix(deployments): Harden delete cleanup and wxO create tests

Treat missing provider agents as stale local cleanup so delete can
finish instead of leaving orphaned deployment rows behind.

Commit local delete operations eagerly, retry once on commit failure,
and move wxO create-path tests toward fake client objects so the real
service logic is exercised without external calls.

* new head

* fix(deployments): stop prefixing wxo raw connection app_ids
Preserve caller app_ids for newly created wxo connections while keeping lf_ prefixing for tool/deployment naming, centralize resource_name_prefix validation, and update mapper/service schema tests and docs to reflect the new behavior.

* fix(deployments): Harden provider account cleanup

Reconcile stale deployment rows before provider-account deletion so
out-of-band provider removals do not leave account cleanup blocked.

* fix: restore py310 compatibility and align payload slot tests
Import Self from typing_extensions in the deployment provider account model to keep backend imports working on Python 3.10, and update payload formalization tests to assert strict rejection of cross-model BaseModel inputs.

* fix(deployments): resolve mypy typing regressions across deployment flows
Normalize SQLModel query expressions for typed SQL operators, make deployment-related model IDs non-nullable where appropriate, and tighten provider mapper/update typing guards. Update deployment tests to align with stricter type contracts.

* fix(deployments): restore py310 test compatibility and migration idempotency
Replace Python 3.11-only UTC imports with timezone.utc in deployment-related tests and the watsonx e2e helper, and make the provider-account name migration safe to re-run when the column and unique constraint are created in separate steps.

* fix(deployments): restore provider account route ownership helpers
Reuse the shared provider-account lookup in update/delete routes so route tests keep the expected ownership path, and use the explicit-field helper when deciding whether to reverify credentials.

* pass name to create_provider_account helper in tests

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>