Update e2e backup tests for backup-rework CLI changes - #1255
Open
RaunakJalan wants to merge 26 commits into
Open
Update e2e backup tests for backup-rework CLI changes#1255RaunakJalan wants to merge 26 commits into
RaunakJalan wants to merge 26 commits into
Conversation
RaunakJalan
commented
Aug 19, 2026
RaunakJalan
commented
Collaborator
- backup import: positional arg → --from-file flag (3 locations)
- backup restore: remove --cluster-id flag (no longer exists)
- backup import: positional arg → --from-file flag (3 locations) - backup restore: remove --cluster-id flag (no longer exists)
The LVol model uses `crypto_bdev` (a device name string like
"crypto_LVOL_60"), not `crypto` (boolean). The assertion was always
failing because `d.get("crypto")` returned None.
max-subsys moved to cluster-level; sn configure no longer accepts it.
RaunakJalan
force-pushed
the
fix/backup-rework-e2e
branch
from
August 21, 2026 09:19
2c7c2b4 to
8ae760a
Compare
…EM to 50 Adjust for 12k max-subsys limit: 60 subsystems x 50 ns/subsystem = 3000 lvols + 9000 snaps = 12000 entities. Both Docker and K8s variants.
RaunakJalan
force-pushed
the
fix/backup-rework-e2e
branch
from
August 21, 2026 10:00
a04c217 to
415e9e1
Compare
Docker workflows: fix --max-lvol → --max-subsys in bootstrap calls. K8s native workflows: move maxSubsystemCount and vcpuCount to StorageCluster spec, replace partitions with enableJournalDevice in StorageNodeSet spec, remove corePercentage (replaced by vcpuCount calculated from CPU count * percentage), rename max_lvol input to max_subsys across all workflows.
…eCluster CRDs These fields were removed from the operator CRD in PR #440. Setting them causes strict decoding errors on latest operator.
hugepages_mem is now a required field in cluster create API. Default to 15G for all K8s native pipelines.
RaunakJalan
force-pushed
the
fix/backup-rework-e2e
branch
from
August 21, 2026 13:10
b3f2356 to
635c358
Compare
…cleanup timeouts Three changes to prevent the test from running 16+ hours when the cluster is broken (see run 32497309385): 1. _phase_rapid_restart_cycles: require >70% of iterations to detect node going offline, otherwise raise RuntimeError immediately. Previously all 30 iterations could silently time out as no-ops. 2. _wait_lvols_deleted: reduce stall_timeout from 1800 to 600s. The old value equalled the overall timeout so the stall check never fired independently. 3. _phase_cleanup: reduce CLEANUP_TIMEOUT from 1800 to 600s and pass stall_timeout=120s to delete_all_clones/delete_all_lvols. When the cluster is SUSPENDED/in_activation, cleanup now gives up after 2 min of no progress per step instead of 30 min. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The NFS base directory is created with sudo (root-owned), so the subsequent mkdir for the timestamped subdirectory also needs sudo. Add chown after mkdir so the cp that follows works without sudo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split time window into 1-hour chunks processed newest-first so progress is visible and high-volume containers (SPDK) don't lose data from scroll timeout expiry - Each chunk writes to a separate subfolder under OUTPUT_DIR (e.g. chunk_01_of_10_20260822_0900_to_1000/) - Increase OpenSearch scroll keepalive from 2m to 5m - Add RUNNER workflow input (vm22 / vm-runner-43) so the job always lands on a runner with NFS mounted - Fix job-summary total_lines to count logs in subdirectories Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…t breaker OOM
The parallel 8-thread fetching was creating 8 simultaneous OpenSearch
scroll contexts, each consuming ~125MB heap. With 500+ containers,
this triggered circuit_breaking_exception ("Data too large") at the
972MB heap limit.
Changes (matching patterns from collect_logs.py in stress pipelines):
- Sequential container fetching: one scroll context at a time instead
of 8 parallel threads — prevents heap pressure
- 429 retry with backoff: wait 10s/30s/60s on circuit_breaking_exception
instead of silently dropping data
- Graylog fallback: if OpenSearch returns 0 lines for a container,
automatically retry via Graylog REST API
- Scroll keepalive back to 2m: with sequential fetching, scroll contexts
complete quickly and don't need 5m keepalive
- Progress counter: [1/503] prefix shows sequential progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…pload - Replace per-file listing in job summary with per-chunk summary (chunk name, file count, size) to stay under GitHub's 1024KB limit - Remove total_lines count from summary (was cat-ing all logs just for a count) - Remove GitHub artifact upload step — logs go to NFS only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- collect-logs.yml: rm -rf OUTPUT_DIR before mkdir to prevent stale files from previous runs mixing with new chunk directories - k8s-native-upgrade.yaml: sed GF_INSTALL_PLUGINS to empty in R25 chart template before helm install (Talos DNS can't reach grafana.com) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Talos enforces PodSecurity baseline/restricted by default, blocking CSI hostpath plugin, fluent-bit, and node-exporter DaemonSets that require privileged containers and hostPath volumes. Create and label the namespace before helm install to allow these workloads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OpenShift uses its own SCC system and k3s/GCP don't enforce PodSecurity admission, so only apply the privileged namespace label when cluster_environment == 'local' (Talos). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- StorageCluster: remove isSingleNode and strictNodeAntiAffinity (deleted from operator CRD), add maxSubsystemCount (moved from StorageNodeSet) - StorageNodeSet: remove maxSubsystemCount (moved to StorageCluster) - Fail fast on CR apply errors (BadRequest, NotFound, strict decoding) instead of logging a warning and continuing - Fix CR verification: remove 2>&1 that caused false positives via substring match in error messages, check stderr separately - Verify all three CRs (storagecluster, storagepool, storagenodeset) Both _apply_custom_resources() methods updated (base + nodesPerSocket). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _get_vcpu_count() that reads VCPU_COUNT env or computes from worker CPU count (50% for OpenShift, 30% otherwise, fallback 4) - Add vcpuCount field to StorageCluster CR YAML in both blocks - Extend fail-fast to catch 'is invalid' and 'Required value' errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the R25->R26 maintenance window upgrade, the CSI external-provisioner sidecar's Kubernetes PVC watches go stale (~25 min gap between pod creation at Step 6 and Phase 4.3). The provisioner never calls CreateVolume for clone-from-snapshot PVCs, causing them to stay Pending until timeout. Bounce simplyblock-csi-controller-0 before Phase 4.3 to give the provisioner fresh watches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
During the maintenance window upgrade, `helm uninstall spdk-csi` deletes the StorageClass `simplyblock-csi-sc` owned by that chart. The new `simplyblock-operator` chart does not recreate it with the same name, so post-upgrade clone/restore operations fail with: storageclass.storage.k8s.io "simplyblock-csi-sc" not found Fix: patch the spdk-csi Helm release manifest AND annotate the live StorageClass with `helm.sh/resource-policy: keep` before uninstall, same pattern used for FDB resources. Also generalize _patch_helm_release_keep_annotations to accept custom resource names. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…replicas in 12k test - SbcliUtils: add _wait_for_api_recovery() that pings GET /storagenode every 15s (up to api_recovery_timeout) when quick retries exhaust on 502/503/504 or ConnectionError. Disabled by default (timeout=0). - Stress tests: enable 30-min API recovery timeout so transient infra outages (e.g. Docker Swarm Raft stalls) don't waste 10+ hour runs. - MassCreateRapidRestart_6k_3Snap_Docker: remove 3 redundant WebAppAPI replicas before test to free memory on mgmt node. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The wait-for-registered and wait-for-online loops in the R25 cluster setup silently fell through on timeout, causing cluster activate to run with 0 online nodes and fail. Now both loops exit 1 on timeout with diagnostic output from `sbcli-dev sn list`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- detect_openshift(): Check for openshift-apiserver namespace instead of oc CLI presence. Machines with oc installed but targeting Talos clusters were incorrectly detected as OpenShift, causing oc debug commands to fail. - _get_vcpu_count(): Use kubectl get node for CPU detection on all platforms (works on both OpenShift and Talos). Raise default and floor from 4/1 to 6/6 to match API minimum. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s do not exist The enable-cpu-topology init job fails on Talos because it looks for /var/lib/kubelet/config.yaml which Talos does not use. This caused sn restart to hang for 300s waiting for the job, breaking upgrades. Set enableCpuTopology to false when cluster_environment is 'local' (Talos) in all k8s-native workflows, and add detect_talos() to k8s_utils.py for the Python upgrade test code. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Talos does not use file-based kubelet configuration, so both skipKubeletConfiguration and enableCpuTopology must be disabled. The other workflows already had this correct; only the upgrade workflow was missing the Talos case. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The StorageNodeSet CRs in the Python upgrade test were missing skipKubeletConfiguration. Set it to true on Talos (alongside enableCpuTopology: false) to match the workflow CRs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all k8s-native workflows from deploying an in-cluster MinIO to using the shared external MinIO at 192.168.10.164:9000 (same as Docker e2e pipeline). This fixes the DNS resolution failure with the release-26.3.0 operator which validates backup endpoint DNS during reconcileCreate. - Remove in-cluster MinIO deployment (namespace, deployment, service) - Update localEndpoint to http://192.168.10.164:9000 - Update credentials to minioadmin/minioadmin (matching Docker pipeline) - Connect mc trace directly to external MinIO instead of port-forwarding
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.