diff --git a/content/en/docs/next/install/cozystack/platform.md b/content/en/docs/next/install/cozystack/platform.md index 981033f8..7f03288f 100644 --- a/content/en/docs/next/install/cozystack/platform.md +++ b/content/en/docs/next/install/cozystack/platform.md @@ -109,7 +109,46 @@ enable a provider-specific load balancer, or use a different network setup. Check out the [provider-specific installation]({{% ref "/docs/next/install/providers" %}}) section. It may include a complete guide for your provider that you can use to deploy a production-ready cluster. -### 2.3. Define Network Configuration +### 2.3. Choose a Storage Backend + +Cozystack runs one of two storage control planes. Both are driven by the same CSI driver and expose the same StorageClasses, so the choice does not change how workloads request volumes — only what manages them underneath. + +{{< tabs name="storage_backend" sync="storage-backend" >}} +{{% tab name="LINSTOR" %}} +The default, and the backend every existing cluster runs. Nothing needs to be set: omitting `storage` selects it. + +```yaml +spec: + components: + platform: + values: + storage: + backend: linstor +``` +{{% /tab %}} +{{% tab name="Blockstor" %}} +Experimental. A LINSTOR-API-compatible control plane that stores its state in Kubernetes custom resources instead of an in-cluster database. + +```yaml +spec: + components: + platform: + values: + storage: + backend: blockstor +``` +{{% /tab %}} +{{< /tabs >}} + +{{% alert color="warning" %}} +Blockstor is **experimental**. Use it on clusters where you can tolerate storage-layer problems, and prefer LINSTOR for production until this notice is removed. +{{% /alert %}} + +An unrecognised value fails the render rather than falling back to a default, so a typo cannot quietly deploy the storage control plane you did not ask for. + +To move an existing cluster from LINSTOR to Blockstor, see [Migrating from LINSTOR to Blockstor]({{% ref "/docs/next/storage/blockstor-migration" %}}). Do not simply flip this value on a cluster that already holds data — the switch installs a new control plane that knows nothing about the existing volumes until they are adopted. + +### 2.4. Define Network Configuration Replace `example.org` in `publishing.host` and `publishing.apiServerEndpoint` with a routable fully-qualified domain name (FQDN) that you control. If you only have a public IP, but no routable FQDN, use [nip.io](https://nip.io/) with dash notation. @@ -130,7 +169,7 @@ networking: Cozystack gathers anonymous usage statistics by default. Learn more about what data is collected and how to opt out in the [Telemetry Documentation]({{% ref "/docs/next/operations/configuration/telemetry" %}}). {{% /alert %}} -### 2.4. Apply Platform Package +### 2.5. Apply Platform Package Once the configuration file is ready, apply it: @@ -198,65 +237,95 @@ kubectl taint nodes --all node-role.kubernetes.io/control-plane- ## 3. Configure Storage Kubernetes needs a storage subsystem to provide persistent volumes to applications, but it doesn't include one of its own. -Cozystack provides [LINSTOR](https://github.com/LINBIT/linstor-server) as a storage subsystem. +Cozystack provides [LINSTOR](https://github.com/LINBIT/linstor-server) as a storage subsystem, and [Blockstor](https://github.com/cozystack/blockstor) as an experimental alternative. -In the following steps, we'll access LINSTOR interface, create storage pools, and define storage classes. +In the following steps, we'll access the storage interface, create storage pools, and define storage classes. + +Both backends speak the same command grammar, so the steps below differ only in which client you run. The backend tabs on this page are linked: pick one in [step 2.3](#23-choose-a-storage-backend) or in any step below, and the rest switch with it. ### 3.1. Check Storage Devices -1. Set up an alias to access LINSTOR: +{{< tabs name="devices_client" sync="storage-backend" >}} +{{% tab name="LINSTOR" %}} +The client lives inside the controller pod, so reach it with an alias: - ```bash - alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor' - ``` +```bash +alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor' +``` -1. List your nodes and check their readiness: +List your nodes and check their readiness: - ```bash - linstor node list - ``` +```bash +linstor node list +``` - Example output shows node names and state: +```console ++-------------------------------------------------------+ +| Node | NodeType | Addresses | State | +|=======================================================| +| srv1 | SATELLITE | 192.168.100.11:3367 (SSL) | Online | +| srv2 | SATELLITE | 192.168.100.12:3367 (SSL) | Online | +| srv3 | SATELLITE | 192.168.100.13:3367 (SSL) | Online | ++-------------------------------------------------------+ +``` - ```console - +-------------------------------------------------------+ - | Node | NodeType | Addresses | State | - |=======================================================| - | srv1 | SATELLITE | 192.168.100.11:3367 (SSL) | Online | - | srv2 | SATELLITE | 192.168.100.12:3367 (SSL) | Online | - | srv3 | SATELLITE | 192.168.100.13:3367 (SSL) | Online | - +-------------------------------------------------------+ - ``` +List available empty devices: -1. List available empty devices: +```bash +linstor physical-storage list +``` - ```bash - linstor physical-storage list - ``` +```console ++--------------------------------------------+ +| Size | Rotational | Nodes | +|============================================| +| 107374182400 | True | srv3[/dev/sdb] | +| | | srv1[/dev/sdb] | +| | | srv2[/dev/sdb] | ++--------------------------------------------+ +``` +{{% /tab %}} +{{% tab name="Blockstor" %}} +`blockstor` is a standalone client that talks to the Kubernetes API directly, so there is no controller pod to exec into and no alias to set — it reads your kubeconfig the way `kubectl` does. Build it from the [Blockstor repository](https://github.com/cozystack/blockstor) with `make build`, or take it from a release. The command grammar is LINSTOR's, short forms included. - Example output shows the same node names: +Check the client reaches the cluster: - ```console - +--------------------------------------------+ - | Size | Rotational | Nodes | - |============================================| - | 107374182400 | True | srv3[/dev/sdb] | - | | | srv1[/dev/sdb] | - | | | srv2[/dev/sdb] | - +--------------------------------------------+ - ``` +```bash +blockstor controller version +``` +List your nodes and check their readiness: +```bash +blockstor node list +``` -### 3.2. Create Storage Pools +```console ++-------+-----------+--------------+--------+ +| Node | NodeType | Addresses | State | ++=======+===========+==============+========+ +| srv1 | SATELLITE | 10.20.0.11 | ONLINE | +| srv2 | SATELLITE | 10.20.0.12 | ONLINE | +| srv3 | SATELLITE | 10.20.0.13 | ONLINE | ++-------+-----------+--------------+--------+ +``` -1. Create storage pools using ZFS or LVM. +List available empty devices: - You can also restore previously created storage pools after a node reset. +```bash +blockstor physical-storage list +``` +{{% /tab %}} +{{< /tabs >}} + +### 3.2. Create Storage Pools - {{< tabs name="create_storage_pools" >}} - {{% tab name="ZFS" %}} +Create storage pools using ZFS or LVM. The same commands restore a pool after a node reset. + +{{< tabs name="pools_client" sync="storage-backend" >}} +{{% tab name="LINSTOR" %}} +**ZFS:** ```bash linstor ps cdp zfs srv1 /dev/sdb --pool-name data --storage-pool data @@ -264,8 +333,7 @@ linstor ps cdp zfs srv2 /dev/sdb --pool-name data --storage-pool data linstor ps cdp zfs srv3 /dev/sdb --pool-name data --storage-pool data ``` -It is [recommended](https://github.com/LINBIT/linstor-server/issues/463#issuecomment-3401472020) -to set `failmode=continue` on ZFS storage pools to allow DRBD to handle disk failures instead of ZFS. +It is [recommended](https://github.com/LINBIT/linstor-server/issues/463#issuecomment-3401472020) to set `failmode=continue` on ZFS storage pools, so DRBD handles a disk failure rather than ZFS: ```bash kubectl exec -ti -n cozy-linstor pod/linstor-satellite.srv1 -- zpool set failmode=continue data @@ -273,8 +341,7 @@ kubectl exec -ti -n cozy-linstor pod/linstor-satellite.srv2 -- zpool set failmod kubectl exec -ti -n cozy-linstor pod/linstor-satellite.srv3 -- zpool set failmode=continue data ``` - {{% /tab %}} - {{% tab name="LVM" %}} +**LVM:** ```bash linstor ps cdp lvm srv1 /dev/sdb --pool-name data --storage-pool data @@ -282,40 +349,76 @@ linstor ps cdp lvm srv2 /dev/sdb --pool-name data --storage-pool data linstor ps cdp lvm srv3 /dev/sdb --pool-name data --storage-pool data ``` - {{% /tab %}} - {{% tab name="Restore ZFS/LVM storage-pool on nodes after reset" %}} +**Restore after a node reset:** ```bash for node in $(kubectl get nodes --no-headers -o custom-columns=":metadata.name"); do echo "linstor storage-pool create zfs $node data data" done -# linstor storage-pool create zfs data data ``` - {{% /tab %}} - {{< /tabs >}} +Check the result: -1. Check the results by listing the storage pools: +```bash +linstor sp l +``` - ```bash - linstor sp l - ``` +```console ++-------------------------------------------------------------------------------------------------------+ +| StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | +|=======================================================================================================| +| DfltDisklessStorPool | srv1 | DISKLESS | | | | False | Ok | +| data | srv1 | ZFS | data | 96.41 GiB | 99.50 GiB | True | Ok | ++-------------------------------------------------------------------------------------------------------+ +``` +{{% /tab %}} +{{% tab name="Blockstor" %}} +**ZFS:** - Example output: +```bash +blockstor ps cdp zfs srv1 /dev/sdb --pool-name data --storage-pool data +blockstor ps cdp zfs srv2 /dev/sdb --pool-name data --storage-pool data +blockstor ps cdp zfs srv3 /dev/sdb --pool-name data --storage-pool data +``` - ```console - +-------------------------------------------------------------------------------------------------------------------------------------+ - | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName | - |=====================================================================================================================================| - | DfltDisklessStorPool | srv1 | DISKLESS | | | | False | Ok | srv1;DfltDisklessStorPool | - | DfltDisklessStorPool | srv2 | DISKLESS | | | | False | Ok | srv2;DfltDisklessStorPool | - | DfltDisklessStorPool | srv3 | DISKLESS | | | | False | Ok | srv3;DfltDisklessStorPool | - | data | srv1 | ZFS | data | 96.41 GiB | 99.50 GiB | True | Ok | srv1;data | - | data | srv2 | ZFS | data | 96.41 GiB | 99.50 GiB | True | Ok | srv2;data | - | data | srv3 | ZFS | data | 96.41 GiB | 99.50 GiB | True | Ok | srv3;data | - +-------------------------------------------------------------------------------------------------------------------------------------+ - ``` +Set `failmode=continue` for the same reason as on LINSTOR. The satellite carries the ZFS tooling: + +```bash +kubectl exec -ti -n cozy-linstor -- zpool set failmode=continue data +``` + +**LVM:** +```bash +blockstor ps cdp lvm srv1 /dev/sdb --pool-name data --storage-pool data +blockstor ps cdp lvm srv2 /dev/sdb --pool-name data --storage-pool data +blockstor ps cdp lvm srv3 /dev/sdb --pool-name data --storage-pool data +``` + +**Restore after a node reset:** + +```bash +for node in $(kubectl get nodes --no-headers -o custom-columns=":metadata.name"); do + echo "blockstor storage-pool create zfs $node data data" +done +``` + +Check the result: + +```bash +blockstor sp l +``` + +```console ++----------------------+-------+----------+----------+--------------+---------------+--------------+-------+ +| StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | ++======================+=======+==========+==========+==============+===============+==============+=======+ +| DfltDisklessStorPool | srv1 | DISKLESS | | | | False | Ok | +| data | srv1 | ZFS_THIN | data | 237.80 GiB | 254 GiB | True | Ok | ++----------------------+-------+----------+----------+--------------+---------------+--------------+-------+ +``` +{{% /tab %}} +{{< /tabs >}} ### 3.3. Create Storage Classes diff --git a/content/en/docs/next/storage/_index.md b/content/en/docs/next/storage/_index.md index ec5849d2..5f0bf9da 100644 --- a/content/en/docs/next/storage/_index.md +++ b/content/en/docs/next/storage/_index.md @@ -7,4 +7,6 @@ aliases: - /docs/next/operations/storage --- -These guides will show you how to perform typical tasks related to the LINSTOR storage system in Cozystack. +These guides will show you how to perform typical tasks related to the storage subsystem in Cozystack. + +Cozystack runs one of two storage control planes, selected at install time: LINSTOR, the default, or Blockstor, which is experimental. See [Choose a Storage Backend]({{% ref "/docs/next/install/cozystack/platform#23-choose-a-storage-backend" %}}) to pick one, and [Migrating from LINSTOR to Blockstor]({{% ref "/docs/next/storage/blockstor-migration" %}}) to move an existing cluster across. Unless a guide says otherwise, the pages in this section describe LINSTOR. diff --git a/content/en/docs/next/storage/blockstor-migration.md b/content/en/docs/next/storage/blockstor-migration.md new file mode 100644 index 00000000..35059a98 --- /dev/null +++ b/content/en/docs/next/storage/blockstor-migration.md @@ -0,0 +1,174 @@ +--- +title: "Migrating from LINSTOR to Blockstor" +linkTitle: "Migrating to Blockstor" +description: "How to move an existing cluster's storage control plane from LINSTOR to Blockstor without recreating volumes" +weight: 7 +--- + +Blockstor is a LINSTOR-API-compatible storage control plane that keeps its state in Kubernetes custom resources. Because it serves the same CSI wire shape through the same CSI driver, an existing cluster can move across without recreating volumes or re-provisioning workloads: the StorageClasses, the PersistentVolumes bound to them, and the DRBD devices on the nodes all stay exactly as they are. What changes is which control plane manages them. + +{{% alert color="warning" %}} +Blockstor is **experimental**. This migration rewrites which component owns your storage metadata. Read the whole page before starting, take the backup in step 1, and rehearse on a cluster you can afford to lose. +{{% /alert %}} + +## How the migration works + +Your data is not copied or moved. DRBD runs in the kernel and is host-scoped: it keeps serving volumes while the control plane is replaced above it. The migration converts LINSTOR's metadata — nodes, storage pools, resource definitions, replicas — into Blockstor custom resources, and Blockstor then *adopts* the running devices rather than creating new ones. + +The order matters, and it is the opposite of what feels natural. You switch the backend **first**, which installs Blockstor's CRDs and starts its control plane against an empty database, and you adopt the existing volumes **second**. Adoption cannot come first because the resources it creates have nowhere to live until the CRDs exist. + +## Before you start + +You need: + +- `linstor-migrate`, the converter shipped with Blockstor. Build it from the Blockstor repository with `make build` or take it from a release. +- A Cozystack release whose pinned Blockstor can register your storage pools. This is not optional and it is the first thing to check — see the box below. +- Enough of a maintenance window that CSI cannot attach or detach volumes for its duration. Running workloads keep their volumes; new pods that need an attach will wait. +- Room in the storage pool for the replicas adoption will add. Blockstor brings replica counts up to each resource group's `placeCount`, and a volume LINSTOR left under-replicated is topped up as soon as the controller starts — a new replica and a full sync each. Check that against your free space before you switch, not after. + +{{% alert color="warning" %}} +**Check your Cozystack version before anything else.** You do not pick a Blockstor version: it arrives pinned inside the Cozystack release, so what decides whether the migration can work is which Cozystack you are on. Registering a storage pool that LINSTOR created requires reading the pool name from `StorDriver/StorPoolName`, which is where LINSTOR stores it; a pinned Blockstor without that support logs `unknown storage pool ""` on every reconcile and adopts nothing. The failure is safe — Blockstor refuses before touching the data plane — but the migration cannot proceed, and the fix is to upgrade Cozystack rather than to swap an image. +{{% /alert %}} + +## 1. Stop everything that writes + +Freeze the writers first, and only then take the backup. LINSTOR's controller keeps writing to its database for as long as it runs, so a backup taken underneath it does not match the dump you take a moment later, and the converted manifest would describe a cluster that no longer exists. + +Order matters. piraeus-operator owns the controller Deployment and the satellite DaemonSets, so it goes first: scale the controller down while the operator is still running and the operator simply puts it back. + +```bash +kubectl -n cozy-system scale deploy/piraeus-operator-controller-manager --replicas=0 +kubectl -n cozy-linstor scale deploy/linstor-controller --replicas=0 +kubectl -n cozy-linstor scale deploy/linstor-csi-controller --replicas=0 +``` + +The CSI provisioner is on that list for a reason of its own, covered in step 6: a volume it cannot find is a volume it re-provisions. + +Confirm nothing is left writing before you continue: + +```bash +kubectl -n cozy-linstor get deploy +``` + +## 2. Back up LINSTOR's metadata + +Everything the migration reads lives in LINSTOR's custom resources. Save them, and the CRD definitions themselves, before you touch the cluster: + +```bash +kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \ + xargs kubectl get crds -ojson > crds.json + +kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \ + xargs -I{} sh -xc "kubectl get {} -ojson > {}.json" + +tar czvf backup-$(date +%d.%m.%Y).tgz *.json +``` + +Switching the backend does not delete these resources, so this backup is what lets you go back. + +## 3. Dump the LINSTOR tables + +The converter reads a directory of per-table JSON dumps: + +```bash +mkdir -p linstor-dump +for crd in $(kubectl get crd -o name | grep 'internal.linstor.linbit.com' \ + | sed 's|customresourcedefinition.apiextensions.k8s.io/||'); do + kubectl get "$crd" -o json > "linstor-dump/${crd}.json" +done +``` + +This works even when the LINSTOR controller itself is unhealthy: the converter reads the custom resources directly and never talks to the LINSTOR API. + +## 4. Capture the live DRBD ports + +Adopted replicas must keep the TCP port their running DRBD connection already uses. If the port is not supplied, Blockstor allocates a fresh one and the mesh reconnects — a brief interruption on every replicated volume. + +The port is not stored in LINSTOR's custom resources, so read it from the running kernel. It appears inside each connection's `path` block: + +```bash +for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do + pod=$(kubectl get pods -n cozy-linstor -o name | grep "linstor-satellite.${node}-" | head -1) + kubectl exec -n cozy-linstor "$pod" -c linstor-satellite -- drbdsetup show 2>/dev/null \ + | awk '/^resource / { res = $2; gsub(/"/, "", res); next } + /_this_host[ \t]+ipv4/ { addr = $3; gsub(/;/, "", addr) + n = split(addr, p, ":") + if (res != "" && p[n] ~ /^[0-9]+$/) print res, p[n] }' +done | sort -u > drbd-ports.txt +``` + +A single-replica volume has no peer and therefore no port to preserve; it will not appear in this file, and that is correct. + +## 5. Convert + +```bash +linstor-migrate -in linstor-dump -drbd-ports drbd-ports.txt -out blockstor-resources.yaml +``` + +Read the warnings. Resources that LINSTOR has marked for deletion are skipped and named. Flags the converter does not recognise are reported and dropped rather than guessed at. + +## 6. Switch the backend + +The CSI provisioner was stopped in step 1, and this is the window it was stopped for. It keeps reconciling while the control plane is being replaced, and a volume it cannot find is a volume it re-provisions: it creates a fresh resource definition through the LINSTOR-compatible API — with newly allocated DRBD minors and ports and a different node ID — for a volume that already exists and still holds data. Those definitions also lack `spec.initialized`, so the satellite treats them as new and queues them for `create-md`. Leaving the provisioner running is how a migration quietly acquires duplicate definitions that disagree with the live mesh. + +Set the storage backend on the platform Package, as described in [Choose a Storage Backend]({{% ref "/docs/next/install/cozystack/platform#23-choose-a-storage-backend" %}}): + +```yaml +spec: + components: + platform: + values: + storage: + backend: blockstor +``` + +Apply it and wait for the Blockstor control plane to come up. The LINSTOR controller and the piraeus-managed satellites go away; piraeus-operator stays on in external mode to keep driving the CSI driver. + +{{% alert color="info" %}} +On a cluster that already ran LINSTOR, the compatibility Service that Blockstor provides for the scheduler and GUI collides with the one piraeus-operator owns. Helm refuses to import it. Take ownership of the existing Service so the release can proceed; a fresh install never hits this. +{{% /alert %}} + +Your volumes keep serving throughout this step. Blockstor does not yet know about them, so CSI cannot attach or detach until adoption finishes. + +## 7. Stop the controller, then adopt + +Scale the Blockstor controller to zero before applying the converted resources: + +```bash +kubectl -n cozy-linstor scale deploy/blockstor-controller --replicas=0 +kubectl apply -f blockstor-resources.yaml +kubectl -n cozy-linstor scale deploy/blockstor-controller --replicas=1 +``` + +The scale-down is not a nicety. Applying the file in one pass makes the resource definitions visible before the replicas further down the file exist, and a running controller reacts by auto-placing replicas from the resource group's placement policy — giving them freshly allocated ports. The real replicas are then rejected, because a replica's DRBD port can be set once and not changed. You are left with replicas whose ports disagree with the live mesh, which is worse than it sounds: reconciling that state reconfigures a running mesh onto ports its peers do not share. + +With the controller stopped, the whole file lands before anything reacts to it. + +## 8. Verify + +Confirm every replica was adopted rather than recreated, and that the ports match what the kernel is using: + +```bash +kubectl get resources.blockstor.cozystack.io -o custom-columns=\ +'RD:.spec.resourceDefinitionName,NODE:.spec.nodeName,PORT:.spec.drbdPort' +``` + +Compare against `drbd-ports.txt`. Then check the data plane is untouched — every peer should still be `UpToDate` and no resource should be syncing: + +```bash +kubectl exec -n cozy-linstor ds/blockstor-satellite -- drbdsetup status +``` + +A resource that starts a full resync after adoption means it was treated as new rather than adopted. Stop and investigate before letting it run. + +Expect the controller to create replicas for any volume LINSTOR left under-replicated. Blockstor reconciles replica count against the resource group's `placeCount` continuously, where LINSTOR only places on request, so a volume sitting at one replica under a three-replica storage class gets two more — each a full sync. This is correct behaviour, not a migration fault, but it is worth knowing before it happens on a pool with no room for it. + +Once the replicas check out, bring the CSI provisioner back: + +```bash +kubectl -n cozy-linstor scale deploy/linstor-csi-controller --replicas=1 +``` + +## Rolling back + +Switching back is the same operation in reverse: set `storage.backend` to `linstor` and apply. LINSTOR's custom resources are not deleted by the migration, so its controller finds its state where it left it. The Blockstor custom resources can be removed afterwards. diff --git a/layouts/partials/hooks/body-end.html b/layouts/partials/hooks/body-end.html index 3cf8ee44..d995b891 100644 --- a/layouts/partials/hooks/body-end.html +++ b/layouts/partials/hooks/body-end.html @@ -19,3 +19,54 @@ } {{ end }} + +{{- /* Page-wide tab switching for tab sets that opt in with `sync`. + Clicking "Blockstor" in one set selects it in every other set on + the page, and the choice survives navigation — so an operator + picks their storage backend once and the whole runbook follows. */ -}} + diff --git a/layouts/shortcodes/tabs.html b/layouts/shortcodes/tabs.html index 0108a4aa..279c663e 100644 --- a/layouts/shortcodes/tabs.html +++ b/layouts/shortcodes/tabs.html @@ -2,7 +2,11 @@ {{- $tab_set_id := .Get "name" | default (printf "tabset-%s-%d" (.Page.RelPermalink) (.Page.Scratch.Get "tabset-counter") ) | anchorize -}} {{- $tabs := .Scratch.Get "tabs" -}} {{- if .Inner -}}{{- /* We don't use the inner content, but Hugo will complain if we don't reference it. */ -}}{{- end -}} -