feat: add Slurm image lifecycle jobs by nabinchha · Pull Request #896 · NVIDIA-NeMo/DataDesigner

nabinchha · 2026-08-27T22:18:24Z

📋 Summary

Add the structured CPU Slurm lifecycle needed to import digest-qualified OCI images and inspect existing SQSH artifacts. The lifecycle stages checksum-bound package resources beneath the selected workspace and submits a thin, isolated batch job through the shared Slurm launcher.

🔗 Related Issue

Part of #867

🔄 Changes

Add a versioned image lifecycle plan that binds the authored request, selected profile, attempt-local paths, source OCI digest, and package-owned runtime resources.
Render CPU-only OCI import and existing-SQSH inspection jobs with safe Enroot URI normalization, exact checksum verification, restrictive workspace staging, and isolated sbatch submission.
Ship a standard-library-only inspector that records factual client Python/package/installer metadata or serving vLLM version/executable metadata inside the target image.
Extract the existing batch directive and shell quoting logic into shared launcher primitives without changing generation script output.
Add behavioral coverage for both lifecycle modes, contract mutations, source credential rejection, symlink/path safety, exact-byte tamper rejection, and client/serving inspection records.

🔍 Attention Areas

⚠️ Reviewers: Please pay special attention to the following:

images/lifecycle.py — CPU Slurm and Enroot command semantics, checksum boundaries, and attempt-local staging.
images/records.py — persisted lifecycle invariants and separation of source OCI identity from produced SQSH identity.
Atomic publication, per-target lifecycle locking, collision handling, and failure cleanup remain intentionally scoped to 867#3.

🧪 Testing

make check-slurm passes
make test-slurm passes — 600 tests
make test-slurm-wheel-install passes
Unit tests added for OCI import and existing-SQSH inspection
E2E tests: N/A for this slice; sealed real-Slurm acceptance remains in Harden the Slurm runtime and validate sealed release artifacts #870

✅ Checklist

Follows commit message conventions
Commits are signed off (DCO)
Architecture docs: N/A — this internal lifecycle slice does not change a documented public surface

Stage digest-bound OCI import and existing-SQSH inspection plans beneath the selected workspace. Reuse safe batch rendering and isolated Slurm submission while shipping a standalone factual inspector for target images. Part of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

greptile-apps · 2026-08-27T22:21:57Z

Greptile Summary

The PR adds structured Slurm jobs for importing digest-qualified OCI images and inspecting existing SQSH images.

Adds checksum-bound lifecycle plans, staged runtime resources, and CPU-only batch scripts.
Adds standard-library image inspection for client and vLLM serving environments.
Extracts shared batch rendering and supports submitting verified scripts over standard input.

Confidence Score: 5/5

The PR appears safe to merge based on the eligible follow-up findings available for this review.

No blocking failure remains in the eligible review set.

Important Files Changed

Filename	Overview
packages/data-designer-slurm/src/data_designer/slurm/images/lifecycle.py	Adds lifecycle preparation, deterministic script rendering, staged-artifact verification, and Slurm submission for OCI and SQSH operations.
packages/data-designer-slurm/src/data_designer/slurm/images/records.py	Adds validated lifecycle records that bind requests, profiles, operation modes, paths, resources, and source OCI identity.
packages/data-designer-slurm/src/data_designer/slurm/images/resources/inspect_image.py	Adds a standalone inspector that records package, Python, installer, and vLLM executable metadata.
packages/data-designer-slurm/src/data_designer/slurm/launcher/batch.py	Centralizes validated Slurm directive rendering and literal shell-value quoting.
packages/data-designer-slurm/src/data_designer/slurm/launcher/client.py	Adds verified batch-script submission through sbatch standard input.
packages/data-designer-slurm/src/data_designer/slurm/launcher/runner.py	Extends the command runner boundary to accept optional text input while preserving argument-vector execution.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Lifecycle
    participant Workspace
    participant Slurm
    participant Enroot
    participant Inspector
    Caller->>Lifecycle: Prepare image lifecycle request
    Lifecycle->>Workspace: Stage plan, script, inspector, and Enroot config
    Caller->>Lifecycle: Submit prepared job
    Lifecycle->>Workspace: Verify exact staged bytes
    Lifecycle->>Slurm: Submit script over stdin
    Slurm->>Enroot: Import OCI or open existing SQSH
    Enroot->>Inspector: Run inspection inside image
    Inspector->>Workspace: Atomically write digest-bound inspection.json

_{Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'origin/fea..." | Re-trigger Greptile}

Restrict the container to a dedicated output mount, isolate Enroot user configuration, and bind vLLM metadata to its owning distribution.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

andreatnvidia · 2026-08-31T14:45:44Z

+            "OCI image source must be a credential-free registry reference without a scheme"
+        ) from error
+    workspace_root = Path(selected_profile.profile.workspace_root)
+    if any(delimiter in workspace_root.as_posix() for delimiter in (":", ",")):


The workspace can contain spaces here, but the path later becomes an Enroot --mount field. Enroot treats that value as colon-separated fstab input, so an unescaped space splits the source field and enroot start rejects a profile that the current fake accepts. Rejecting whitespace and backslashes during preparation and plan validation, or escaping them for Enroot with a rendered-mount regression, would keep accepted profiles runnable.

Fixed in 2b2f9ad4. Preparation and plan validation now share validate_enroot_mount_path, which rejects whitespace, backslashes, colons, and commas before any lifecycle directories are created. The regression matrix covers space, backslash, colon, and comma paths, while the renderer test verifies the accepted mount form.

andreatnvidia · 2026-08-31T14:45:49Z

+    missing = tuple(name for name in _REQUIRED_CLIENT_DISTRIBUTIONS if name not in versions)
+    if missing:
+        raise RuntimeError(f"required client distributions are not installed: {', '.join(missing)}")
+    installer_path = shutil.which("pip")


installer_version comes from the active Python environment, while installer_path is the first pip on PATH. In an image with multiple Python environments, that can persist one pip version beside another environment's executable, which later dependency installation will trust. Resolving pip from its owning distribution like the serving path now does for vLLM, with a shadow-PATH fixture, would keep those facts tied to the same installation.

andreatnvidia · 2026-08-31T14:45:55Z

+    source_block = ""
+    if plan.operation is ImageLifecycleOperation.IMPORT_OCI:
+        source_block = f"""readonly DD_OCI_SOURCE={quote_shell_value(_format_enroot_oci_uri(plan.request.source))}
+if [[ -e "${{DD_IMAGE_SQSH}}" ]]; then


-e returns false for a dangling symlink, so a planted candidate.sqsh symlink passes this precheck. Enroot canonicalizes the output path before writing, which lets the import escape the job directory before the later -L check notices it. Please reject -L at this precheck as well. The broader replacement race can stay in the existing follow-up.

andreatnvidia · 2026-08-31T14:46:00Z

+            or _compute_regular_file_sha256(expected_path) != expected_sha256
+        ):
+            raise ImageLifecycleError(f"prepared image lifecycle {label} no longer matches its digest")
+    return client.submit(prepared.script_file.path)


Here the script is hashed by pathname, then sbatch opens that pathname again. A replacement between those operations can submit bytes that never passed verification, and _compute_regular_file_sha256 also separates lstat from read_bytes, leaving an earlier window. Can we reuse the descriptor-bound verification pattern from compute_sqsh_file_sha256 and submit the verified script bytes through stdin instead of reopening the path?

Fixed in 2b2f9ad4. Lifecycle artifacts are now opened once with O_NOFOLLOW, checked with fstat, and read through that descriptor. The verified script contents are submitted with sbatch --parsable --export=NIL through stdin, so sbatch never reopens the path. A mutation regression replaces the script path during submission and verifies that the previously verified contents—not the replacement—are sent.

andreatnvidia · 2026-08-31T14:46:05Z

+            ("partition", profile.image_build.partition),
+            ("nodes", "1"),
+            ("ntasks", "1"),
+            ("cpus-per-task", "1"),


This requests one CPU, but Enroot defaults ENROOT_MAX_PROCESSORS to nproc and uses it for parallel import and extraction. Depending on site affinity, the job can either consume more CPUs than requested or process a large serving image on one core under the fixed time limit. Would it make sense to bind ENROOT_MAX_PROCESSORS to SLURM_CPUS_PER_TASK and move CPU, memory, and time into ImageBuildProfile so sites can size these jobs explicitly?

Fixed in 2b2f9ad4. ImageBuildProfile now requires cpus_per_task, memory, and time_limit; the lifecycle directives use those values. The job validates SLURM_CPUS_PER_TASK and exports it as ENROOT_MAX_PROCESSORS, keeping Enroot parallelism within the Slurm allocation. Profile boundary and renderer regressions cover the new contract.

One compatibility gap remains: older Enroot create paths do not pass ENROOT_MAX_PROCESSORS to unsquashfs, so existing-SQSH inspection can use all visible CPUs despite the Slurm allocation. A minimal smoke test on internal infrastructure confirmed this behavior. The compatibility check should cover processor-bounded extraction rather than relying on the environment variable alone.

Addressed in 76620f0. Existing-SQSH inspection now runs the same actual compute-node Enroot 4+ compatibility preflight before enroot create. Enroot 4 propagates ENROOT_MAX_PROCESSORS to the unsquashfs -processors extraction path, while older incompatible implementations are rejected instead of trusting the environment variable alone. The execution test verifies that Enroot 3 reaches only the version probe and never create/import.

andreatnvidia · 2026-08-31T14:46:11Z

+        distribution = importlib.metadata.distribution("vllm")
+    except importlib.metadata.PackageNotFoundError as error:
+        raise RuntimeError("required distribution 'vllm' is not installed") from error
+    executable_path = _find_distribution_console_script(distribution, "vllm")


These two inspector-1 implementations now have different serving semantics: this staged inspector binds vLLM to its distribution, while images/inspection.py still combines metadata with PATH. Registry and serving tests continue to build records through the older in-process inspectors, so the fakes no longer model the shipped runtime and the version gate cannot distinguish them. Either consolidating on the staged implementation or versioning the two semantics separately with parity coverage would prevent those paths from drifting further.

Fixed in 2b2f9ad4. The in-process inspector now imports the staged inspector's distribution-owned console-script resolver, and both client and serving inspectors receive version/path as one paired fact from that resolver. The fake environment models the same paired API. A parity regression with a shadowed PATH asserts that the package and standalone vLLM inspectors emit identical records.

Reject Enroot-unsafe mount paths and dangling import targets. Submit descriptor-verified scripts through standard input and bind lifecycle resource limits to the selected profile. Resolve pip and vLLM console scripts from their owning distributions with shared inspection semantics. Part of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Reconcile serving-resolution contract fixtures with the image lifecycle resource profile and preserve the review-fix commits. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

andreatnvidia · 2026-08-31T17:59:32Z

+    )
+    source_block = ""
+    if plan.operation is ImageLifecycleOperation.IMPORT_OCI:
+        source_block = f"""readonly DD_OCI_SOURCE={quote_shell_value(_format_enroot_oci_uri(plan.request.source))}


Every accepted container image source keeps its @sha256 suffix here, but Enroot releases before 4 reject that URI form. A minimal smoke test on internal infrastructure confirmed the failure. Because the code accepts profiles without checking this capability, please add a digest-import probe or enforce the required Enroot version before submission.

Addressed in 76620f0. The staged lifecycle now checks the actual compute-node enroot version, rejects malformed output and Enroot versions below 4 with exit 78, and does so before the digest-qualified import. The check intentionally runs inside the allocation because the submit host binary need not match the compute node. Execution coverage now proves Enroot 4 succeeds and Enroot 3 is rejected before any import or create operation.

andreatnvidia · 2026-08-31T17:59:46Z

+    missing = tuple(name for name in _REQUIRED_CLIENT_DISTRIBUTIONS if name not in versions)
+    if missing:
+        raise RuntimeError(f"required client distributions are not installed: {', '.join(missing)}")
+    installer_distributions = tuple(


The two client inspectors still disagree when the environment contains duplicate same-version pip distributions. This standalone path requires exactly one, while SystemInspectionEnvironment collapses them and selects one, so tests and fakes can accept an image that the shipped lifecycle rejects. Would it make sense to share the selection rule or add client parity coverage for this duplicate case?

Addressed in 76620f0. Both the in-process and standalone inspectors now use the same canonical exact-one distribution selector. Duplicate same-version pip installs are rejected by both paths, with explicit parity coverage for that case; the serving path shares the selector as well.

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

…/867-image-lifecycle # Conflicts: # packages/data-designer-slurm/tests/integration/golden/finalization_chain.json # packages/data-designer-slurm/tests/slurm_test_fakes/golden/rendered/multi_node.sbatch # packages/data-designer-slurm/tests/slurm_test_fakes/golden/rendered/single_node.sbatch # packages/data-designer-slurm/tests/slurm_test_fakes/test_rendered_scripts.py

nabinchha requested a review from a team as a code owner August 27, 2026 22:18

andreatnvidia reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/data-designer-slurm/src/data_designer/slurm/images/lifecycle.py Outdated

andreatnvidia reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/data-designer-slurm/src/data_designer/slurm/images/lifecycle.py

andreatnvidia reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/data-designer-slurm/src/data_designer/slurm/images/lifecycle.py

andreatnvidia reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/data-designer-slurm/src/data_designer/slurm/images/resources/inspect_image.py Outdated

fix: harden Slurm image inspection

0c8a6cb

Restrict the container to a dedicated output mount, isolate Enroot user configuration, and bind vLLM metadata to its owning distribution.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

github-actions Bot mentioned this pull request Aug 31, 2026

Agentic CI: Issue & PR Triage Tracker #562

Open

andreatnvidia reviewed Aug 31, 2026

View reviewed changes

nabinchha added 2 commits August 31, 2026 10:36

Merge updated Slurm integration branch

441ff74

Reconcile serving-resolution contract fixtures with the image lifecycle resource profile and preserve the review-fix commits. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

nabinchha requested a review from andreatnvidia August 31, 2026 17:51

andreatnvidia reviewed Aug 31, 2026

View reviewed changes

nabinchha added 2 commits August 31, 2026 16:35

fix: validate image lifecycle compatibility

76620f0

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

nabinchha requested a review from andreatnvidia September 1, 2026 00:01

Uh oh!

Conversation

nabinchha commented Aug 27, 2026

📋 Summary

🔗 Related Issue

🔄 Changes

🔍 Attention Areas

🧪 Testing

✅ Checklist

Uh oh!

greptile-apps Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Aug 27, 2026 •

edited

Loading