Skip to content

Add one-box Linux AMD64 and ARM64 development setup support - #5568

Draft
ahunnargikar-nvidia wants to merge 39 commits into
NVIDIA:mainfrom
ahunnargikar-nvidia:dgx-spark-arm64
Draft

Add one-box Linux AMD64 and ARM64 development setup support#5568
ahunnargikar-nvidia wants to merge 39 commits into
NVIDIA:mainfrom
ahunnargikar-nvidia:dgx-spark-arm64

Conversation

@ahunnargikar-nvidia

@ahunnargikar-nvidia ahunnargikar-nvidia commented Aug 29, 2026

Copy link
Copy Markdown

This PR adds a one-box development setup path for NICo on Linux hosts running
either AMD64 or ARM64. The goal is to let a developer build the required
artifacts and bring up the local NICo stack on a single machine without assuming
an x86-only build host or a multi-node production topology.

The branch combines architecture-aware container and PXE builds with the
single-node deployment fixes needed to make the prerequisite, database, Vault,
Temporal, Core, and site-agent setup repeatable on a development box.

Related issues

None linked.

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Change breakdown

Multi-architecture build support

  • Makes the Machine-a-Tron Dockerfile target-aware so its native builder can
    compile and publish both linux/amd64 and linux/arm64 runtime images.
  • Selects the Rust target, cross-linker, compiler, development libraries, and
    runtime platform from BuildKit's build and target architecture values.
  • Makes the PXE builder run natively on either AMD64 or ARM64 while continuing
    to produce the x86_64 Scout, qcow, and iPXE artifacts required by the local
    managed-host workflow.
  • Adds the AMD64 multiarch packages and Rust/C/C++ cross-toolchain needed to
    build x86_64 artifacts on an ARM64 host without running the compiler under
    QEMU.
  • Installs a pinned AMD64 binfmt handler on non-x86 hosts for package scripts
    that mkosi must execute while assembling x86_64 images.
  • Marks the affected mkosi profiles explicitly as x86-64 and configures a
    matching Ubuntu Noble tools tree.
  • Updates the cargo-make entry points and build documentation for the same
    AMD64/ARM64 workflow.

One-box prerequisite installation

  • Installs helmfile under the user's local bin directory instead of requiring
    root access to /usr/local/bin.
  • Uses versioned direct or OCI chart sources for cert-manager, Vault,
    External Secrets, PostgreSQL, and MetalLB prerequisites.
  • Extends MetalLB chart parsing so direct HTTP, OCI, repository-qualified, and
    local chart coordinates work consistently.
  • Recovers interrupted Helm installs, upgrades, and rollbacks before retrying
    the prerequisite workflow.
  • Keeps the setup phases rerunnable after an interrupted SSH or local automation
    session.

Single-node PostgreSQL reliability

  • Derives synchronous replication from the configured replica count: disabled
    for a one-instance development database and enabled when a standby exists.
  • Detects and repairs stale single-node synchronous-replication state and
    terminates stranded SyncRep sessions before reconciliation.
  • Discovers the current PostgreSQL primary rather than relying on a fixed pod
    name.
  • Creates and reconciles the NICo databases and required REST extensions
    through psql, with bounded retries and advisory locking for concurrent
    setup attempts.
  • Makes prerequisite-mode REST setup skip waits that belong to the later
    application deployment phase.

Vault bootstrap and certificate reconciliation

  • Discovers the configured Vault replica count during initialization and
    unseal rather than assuming a fixed three-pod topology.
  • Adds the service DNS names required for one-box TLS bootstrap and reconciles
    managed Vault certificates before initialization.
  • Stabilizes first-time Vault startup, initialization, unseal rounds, and
    readiness checks so a single-node deployment can converge reliably.
  • Ignores empty optional certificate DNS names instead of rendering invalid
    entries.

Temporal, Core, and site-agent connectivity

  • Uses Temporal's stable headless frontend service for local namespace setup
    and the site-agent runtime.
  • Corrects Temporal namespace CLI arguments, configures retention, and exposes
    an explicit public client endpoint.
  • Uses absolute cluster-local DNS names for Core, Flow, and Temporal to avoid
    resolver search-path delays on a one-box cluster.
  • Adds a config checksum to roll the site-agent when its generated connection
    settings change.
  • Creates the legacy Core namespace alias needed by fresh one-box setups and
    exposes the configurable Kea DHCP hook-library path used by architecture-
    specific one-box configuration.

Documentation

  • Documents topology-derived PostgreSQL synchronous mode in the quick start.
  • Documents the native AMD64/ARM64 PXE builder and x86_64 cross-build flow.
  • Updates prerequisite notes for the user-local Helmfile installation path.

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Passed while preparing this draft:

  • git diff --check origin/main...dgx-spark-arm64
  • bash -n for helm-prereqs/setup.sh, health-check.sh, preflight.sh,
    unseal_vault.sh, and rest-api/scripts/setup-local.sh
  • helm lint helm/charts/nico-api
  • helm lint helm/rest/nico-rest-site-agent
  • helm lint rest-api/temporal-helm/temporal
  • Buildx Dockerfile checks for linux/amd64,linux/arm64 on
    crates/machine-a-tron/Dockerfile
  • Buildx Dockerfile checks for linux/amd64,linux/arm64 on
    dev/docker/Dockerfile.pxe-build-container

Not run during this draft-preparation pass:

  • Full cargo make pre-commit-verify-workspace
  • Complete AMD64 and ARM64 image builds
  • A fresh end-to-end one-box deployment and teardown on both architectures
  • Fern and rumdl documentation validation

Additional Notes

  • This is intentionally a draft. Upstream main advanced by one commit after
    this branch's last rebase, so the branch must be refreshed before it is marked
    ready for review.
  • All 37 branch commits contain a DCO Signed-off-by trailer. Local Git reports
    that they are not cryptographically signed; the repository's commit-signature
    requirement must be resolved before merge readiness.
  • The one-box settings are development defaults and recovery behavior. They do
    not remove support for multi-replica prerequisite deployments.

Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
ahunnargikar-nvidia and others added 7 commits August 29, 2026 08:33
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Andy Wrenn <awrenn@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Ashish Hunnargikar <ahunnargikar@nvidia.com>
Signed-off-by: Andy Wrenn <awrenn@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@thossain-nv thossain-nv added the rest-api Add this label when an issue or PR concerns NICo REST API label Aug 31, 2026 — with ChatGPT Codex Connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

helm charts rest-api Add this label when an issue or PR concerns NICo REST API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants