Backport Fix: Add expected_node_count variable to vault_raft_remove_node_and_verify module into ce/main (#17071)

* Fix: Add expected_node_count variable to vault_raft_remove_node_and_verify module (#17067)

* Fix: Add expected_node_count variable to vault_verify_removed_node_shim module

The enos-scenario-autopilot.hcl passes expected_node_count parameter to
verify_raft_node_removed step, which conditionally uses either
vault_verify_removed_node or vault_verify_removed_node_shim based on
vault version.

The shim module (used for versions < 1.19.0) was missing this variable,
causing Terraform errors when the scenario passes the parameter.

This fix adds the expected_node_count variable to the shim module to
match the full module's interface, even though the shim doesn't use it.

* Fix: Add expected_node_count to smoke scenario vault_remove_node_and_verify step

When backend is not raft (e.g., consul), the vault_verify_removed_node_shim
module is used, which now requires the expected_node_count parameter.

This fixes the Terraform validation error:
Error: Missing required argument
  on scenario.tf line 0, in module "vault_remove_node_and_verify":
 658: module "vault_remove_node_and_verify" {
The argument "expected_node_count" is required, but no definition was found.

* Fix: Use create_vault_cluster_targets.hosts for expected_node_count calculation

The previous fix used length(step.get_vault_cluster_ips.follower_hosts) which:
1. Causes 'Unsupported attribute' error (follower_hosts is an output, not directly accessible)
2. Would be incorrect anyway (follower count != total cluster size)

Now using length(step.create_vault_cluster_targets.hosts) which gives the
total cluster size, which is what the test expects to verify after nodes
are added back.

* Fix: Use static value 3 for expected_node_count

Enos doesn't allow function calls in variable assignments. The validation
error states: 'A single static variable reference is required: only
attribute access and indexing with constant keys. No calculations,
function calls, template expressions, etc are allowed here.'

Using static value 3 since that's the default cluster size (instance_count
defaults to 3 in target_ec2_instances module).

* Fix: Add expected_node_count variable to vault_raft_remove_node_and_verify module

The smoke scenario passes expected_node_count to vault_remove_node_and_verify step.
When backend=raft, this uses vault_raft_remove_node_and_verify module, which
was missing this variable definition (even though it calculates and passes it
internally to the verify_removed submodule).

This caused Terraform validation errors in backport PRs where the scenario
tries to pass expected_node_count=3 but the module doesn't accept it.

Adding the variable with a default of null since the module calculates it
internally as length(var.hosts) and doesn't need the passed value.

* Add expected_node_count to fyre-smoke scenario

The fyre-smoke scenario also uses vault_raft_remove_node_and_verify module
and needs to pass expected_node_count parameter.

* Convert vault_verify_raft_auto_join_voter to blackbox SDK (#16558)

* Convert vault_verify_raft_auto_join_voter to blackbox SDK

- Add new Go test: vault/external_tests/blackbox/scenario/raft/auto_join_voter_test.go
- Update 9 scenarios to use vault_run_blackbox_test module
- Update vault_verify_removed_node module to use blackbox test
- Remove old shell-based vault_verify_raft_auto_join_voter module
- Remove module declaration from enos-modules.hcl

The new test verifies all raft nodes are voters via HTTP API instead of
SSH-based shell scripts, making it more efficient and maintainable.

* Fix enos scenario syntax errors

- Remove tostring() function calls which are not supported in enos HCL
- Fix autopilot scenario to use get_local_metadata instead of non-existent get_updated_vault_cluster_ips
- EXPECTED_NODE_COUNT now passes integer directly instead of string conversion

* Remove test_env_vars blocks from enos scenarios

EXPECTED_NODE_COUNT is optional in the Go test, so we don't need to pass it.
This avoids both the tostring() syntax error and the non-existent .hosts attribute error.

* Fix fyre-autopilot step reference to use get_local_metadata

* Fix fyre-smoke step reference to use get_vault_cluster_ips

* Fix Enos module references for leader_host/leader_public_ip

* Add missing dependency on get_vault_cluster_ips step

* Fix formatting: remove extra space in depends_on

* Fix duplicate attributes in verify_raft_auto_join_voter step

* Remove t.Parallel() from TestRaftAutoJoinVoter

Test was timing out because t.Parallel() caused it to run before cluster
was ready. Match pattern of other raft tests which run sequentially.

* Wait for raft cluster health before checking voter status

Test was hanging on sys/storage/raft/configuration read because raft
cluster wasn't fully initialized. Add EventuallyRaftClusterHealthy()
call to ensure cluster is ready before reading configuration.

* Simplify test to use SDK's AssertRaftStable helper

The original implementation was overly complex with manual retry logic.
The SDK already provides AssertRaftStable() which verifies all nodes are
voters, matching the original shell script's behavior exactly.

* Fix test to match original shell script behavior

Original script verified each node was a voter without checking total count.
Test was failing because it expected specific node count (1) but cluster had 3.
Now just verifies all nodes in raft config are voters, matching original.

* Remove MustGetConfigStorageType call that times out

The config endpoint was timing out. Instead, directly try to read raft
configuration and skip if it fails (not using raft storage).

* Add retry logic for nodes becoming voters

Nodes may not be voters immediately after cluster is unsealed.
Original shell script had 'retry 10' with exponential backoff.
Added EventuallyWithTimeout to retry for up to 2 minutes.

* Increase timeout to match shell script retry duration

Shell script uses 'retry 10' with exponential backoff:
2^0 + 2^1 + ... + 2^9 = 1023 seconds ≈ 17 minutes
Using 20 minutes to be safe.

* Remove EventuallyClusterHealthyUnsealed call

Original shell script doesn't wait for autopilot health before checking
voter status. It just runs 'retry 10 check_voter_status' directly.

The autopilot health check may be causing issues. Match original behavior:
just retry checking if nodes are voters.

* Override test timeout for raft auto join voter test

Test needs 20+ minutes to match shell script's retry logic (2^0 + 2^1 + ... + 2^9 = 1023s ≈ 17min).
Default calculated timeout is only 5 minutes (25s per test).
Set VAULT_TEST_TIMEOUT=25m to allow sufficient retry time.

* Add debug output for VAULT_TEST_TIMEOUT env var

Need to verify if the environment variable is being passed through
from Enos config to the shell script.

* Add VAULT_TEST_TIMEOUT to autopilot scenario

The test timeout override was only added to smoke scenario, but CI
failures are occurring in the autopilot scenario. Add the same
test_env_vars configuration to autopilot's verify_raft_auto_join_voter
step.

* Fix: Use updated cluster IPs after autopilot upgrade

Root cause: The verify_raft_auto_join_voter step was using IPs from the
original cluster (step.get_vault_cluster_ips) instead of the upgraded
cluster (step.get_updated_vault_cluster_ips).

During autopilot upgrade:
1. Original 3-node cluster is created
2. 3 new upgrade nodes are added
3. Autopilot promotes new nodes
4. Old nodes are removed

The test was checking the OLD cluster nodes (which are being removed)
instead of the NEW upgraded nodes. This explains why:
- Shell script worked: ran on each node individually
- Blackbox test failed: queried wrong cluster via API

Fix: Use get_updated_vault_cluster_ips and add proper dependency on
wait_for_leader_in_upgrade_targets to ensure cluster is stable.

* Convert autopilot verify_raft_auto_join_voter to blackbox test

Replace vault_verify_raft_auto_join_voter shell script module with
vault_run_blackbox_test running TestRaftAutoJoinVoter.

Changes:
- Use module.vault_run_blackbox_test instead of module.vault_verify_raft_auto_join_voter
- Add test_package, test_names, vault_edition parameters
- Add VAULT_TEST_TIMEOUT=25m override
- Use leader_host/leader_public_ip from get_vault_cluster_ips
- Keep same dependencies to avoid circular refs

The test will connect to the original cluster leader which will
redirect API calls to the current leader in the upgraded cluster.

* Fix: Run voter test AFTER old nodes are removed

Root cause: The test was running during autopilot upgrade while old
nodes were still in the cluster. It was checking a mixed cluster of
old (being removed) and new (upgraded) nodes.

The error 'node node_0 is not yet a voter' refers to an OLD node
that autopilot is in the process of removing, not a new upgraded node.

Fix: Move verify_raft_auto_join_voter to run AFTER remove_old_nodes
step completes. This ensures the test only checks the final upgraded
cluster with old nodes fully removed.

Dependencies changed:
- Added: step.get_updated_vault_cluster_ips (for correct IPs)
- Added: step.remove_old_nodes (wait for cleanup)
- Use updated cluster IPs instead of original cluster IPs

* fix: move verify_raft_auto_join_voter test after old nodes removed

The test was checking the cluster during autopilot upgrade when old nodes
were still present, causing failures due to IPv6 addresses from nodes being
removed. Moving the test to run after remove_old_nodes ensures it only
checks the final upgraded cluster state.

- Moved verify_raft_auto_join_voter step after remove_old_nodes
- Updated dependencies to use get_updated_vault_cluster_ips
- Removed verify_raft_auto_join_voter from early step dependencies

* Address PR review feedback

- Fix copyright year to 2026 for new test file
- Clean up test comments to focus on purpose, not history
- Add expected_node_count as proper module variable
- Remove redundant leader_public_ip parameter
- Fix incorrect step references in autopilot scenario

* Remove redundant leader_public_ip from fyre scenario

* Make leader_public_ip optional in vault_run_blackbox_test module

* Run terraform fmt on modified modules

* Add expected_node_count to verify_raft_node_removed step in autopilot scenario

* Fix expected_node_count to use correct step output

* Fix fyre_os_info module to handle multiple OS version matches

* Fix expected_node_count to use create_vault_cluster_upgrade_targets.hosts

* Remove expected_node_count from verify_raft_node_removed - Enos doesn't support function calls in variables

* Fix missing expected_node_count parameter in verify_raft_node_removed step

* Remove duplicate variables in verify_raft_node_removed step

* Revert "Convert vault_verify_raft_auto_join_voter to blackbox SDK (#16558)"

This reverts commit b40977d7c70b838a10608dfa374a0eef0a75878a.

---------

Co-authored-by: hashigator <280075563+hashigator@users.noreply.github.com>
Co-authored-by: hashigator <lt.carbonell@ibm.com>