kernel: distribute darwin/arm64 lib as nested per-platform module (go get, no build step)

.gitignore

@@ -67,7 +67,15 @@ _tmp*
 __debug_bin
 .DS_Store
 
-# Kernel (SEA) backend build artifacts — produced by `make kernel-lib`, never committed.
+# Kernel (SEA) backend build artifacts.
+#
+# The committed distribution model (see README) commits the small
+# platform-independent C header (include/) and each platform's prebuilt archive
+# under kernellib/<platform>/ (nested modules), so `go get` needs no build step.
+# Those committed paths are intentionally NOT ignored.
+#
+# The paths below remain ignored: they are scratch dirs still used by
+# `make kernel-lib` for platforms not yet committed (linux/windows) and for
+# source builds — never committed.
 /build/kernel-src/
 /internal/backend/kernel/lib/
-/internal/backend/kernel/include/

README.md

@@ -9,6 +9,7 @@ standard `database/sql` interface.
 ## Contents
 
 - [Quick start](#quick-start)
+- [Cloning the repository](#cloning-the-repository)
 - [Choosing a backend (Thrift vs SEA/kernel)](#choosing-a-backend-thrift-vs-seakernel)
 - [Building](#building)
 - [Connecting](#connecting)
@@ -46,6 +47,48 @@ defer rows.Close()
 See [`doc.go`](./doc.go) for full package documentation or the Databricks documentation
 for the [SQL Driver for Go](https://docs.databricks.com/dev-tools/go-sql-driver.html).
 
+> **Using the driver in your own project?** You never clone this repository — you
+> add it with `go get github.com/databricks/databricks-sql-go` and `go build`.
+> `go get` fetches per-version module archives, not git history, and for a
+> default Thrift build it pulls **no** kernel binaries at all. The guidance below
+> is only for people who `git clone` this repo directly (contributors / CI).
+
+## Cloning the repository
+
+This repo commits a small number of **prebuilt kernel binaries** (per-platform
+`libdatabricks_sql_kernel.a`, ~62 MB each, each in its own nested module under
+`internal/backend/kernel/kernellib/<platform>`) so that the SEA/kernel backend
+works straight from `go get` with **no build step** (see
+[SEA/kernel](#seakernel--cgo--a-linked-rust-static-library); for how these are
+versioned and published, see [docs/RELEASING.md](./docs/RELEASING.md)). A
+consumer's `go get` pulls only the target platform's archive at the
+driver-pinned version — never all platforms. Committed binaries cannot be
+delta-compressed by git, so a *full* clone accumulates their whole history over
+releases.
+
+**If you clone this repo directly, use a partial clone** so you download only the
+binary versions you actually check out, not the entire history:
+
+```bash
+git clone --filter=blob:none https://github.com/databricks/databricks-sql-go
+```
+
+`--filter=blob:none` fetches commits and trees immediately and pulls file blobs
+lazily, only when a checkout needs them. This keeps `.git` small and — unlike a
+naive `git clone` — it does **not** grow with the number of releases (only your
+current checkout's blobs are fetched). GitHub serves this by default. To also
+avoid materializing other platforms' archives in your working tree, add
+`--sparse` and select the paths you need:
+
+```bash
+git clone --filter=blob:none --sparse https://github.com/databricks/databricks-sql-go
+cd databricks-sql-go
+git sparse-checkout set --no-cone '/*' '!/internal/backend/kernel/kernellib' \
+    'internal/backend/kernel/kernellib/darwin_arm64'   # keep only your platform
+```
+
+CI checkouts in this repo use `--filter=blob:none` for the same reason.
+
 ## Choosing a backend (Thrift vs SEA/kernel)
 
 The driver has **two execution backends**, selected once per connection:

docs/RELEASING.md

@@ -0,0 +1,114 @@
+# Releasing the driver (with the kernel/SEA backend)
+
+This repo ships the kernel/SEA backend as **committed, per-platform, prebuilt
+static archives** carried by `go get` — no `make kernel-lib` step for consumers,
+no Rust toolchain. This document explains how those archives are versioned and
+published so that a `go get github.com/databricks/databricks-sql-go@vX.Y.Z`
+resolves the matching kernel archive automatically.
+
+## The module layout
+
+The driver is a multi-module repository:
+
+```
+github.com/databricks/databricks-sql-go                       (the driver module)
+└── internal/backend/kernel/kernellib/<platform>/             (one NESTED module per platform)
+    ├── go.mod            → github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/<platform>
+    ├── link.go           → //go:build cgo && databricks_kernel && <os> && <arch>  (+ #cgo LDFLAGS)
+    └── libdatabricks_sql_kernel.a   (the committed prebuilt archive for this platform)
+```
+
+Each `kernellib/<platform>` directory is **its own Go module** (it has a
+`go.mod`). This is deliberate: Go downloads a module's zip only when a build
+compiles a file from it, and each `link.go` is build-tag-gated to one platform +
+`databricks_kernel`. So:
+
+- a **Thrift build** (`CGO_ENABLED=0`, no tag) downloads **none** of them;
+- a **kernel build** for, say, darwin/arm64 downloads **only** the
+  `darwin_arm64` module — never the other platforms' archives.
+
+## How versioning works
+
+The driver's `go.mod` `require`s each platform module at a **real version**, and
+also carries a `replace` pointing at the in-tree source:
+
+```
+require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v1.2.3
+
+replace github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 => ./internal/backend/kernel/kernellib/darwin_arm64
+```
+
+- The **`require` version pins the kernel**. `go get .../databricks-sql-go@v1.2.3`
+  reads that tag's `go.mod`, sees `require .../darwin_arm64 v1.2.3`, and resolves
+  that exact archive version from the module proxy. **Upgrading the driver is what
+  moves the kernel version** — deterministic, per-driver-version pinning.
+- The **`replace` is local-only and consumer-invisible.** Per the
+  [Go module spec](https://go.dev/ref/mod#go-mod-file-replace), a `replace` in a
+  *dependency's* `go.mod` is ignored — it applies only when this repo is the main
+  module. So it lets THIS repo build against the committed archive during
+  development, while a downstream `go get` always resolves the published version
+  from the proxy. (Verified: a consumer building against a published module sees
+  the proxy copy, not the replace target.)
+
+## Publishing: path-prefixed tags
+
+Go publishes a nested module using a **tag whose name is the module's
+subdirectory path plus the version**. To release the darwin/arm64 kernel module
+at `v1.2.3`:
+
+```
+git tag internal/backend/kernel/kernellib/darwin_arm64/v1.2.3
+git tag internal/backend/kernel/kernellib/linux_amd64/v1.2.3
+# ... one tag per platform module ...
+git tag v1.2.3                      # the driver module itself
+git push origin --tags
+```
+
+The module proxy serves each nested module's zip **excluding** any nested-module
+subtree, and serves it at the version from its path-prefixed tag. The driver
+module's own zip (tag `v1.2.3`) excludes the `kernellib/*` subtrees — consumers
+pull those separately at the versions the driver `require`s.
+
+## Release steps
+
+1. **Build the archives.** The kernel repo's `build-c-abi-libs` workflow builds
+   `libdatabricks_sql_kernel.a` per platform on native runners and pushes them
+   into the `kernellib/<platform>/` directories here (see that repo's workflow;
+   it opens a sync PR against this repo). Alternatively, drop a locally built
+   archive in for a single platform during development.
+2. **Bump the `require` versions** in the driver `go.mod` to the new release
+   version (keep the matching `replace` lines).
+3. **Tag every module** at the new version using the path-prefixed tags above,
+   plus the plain `vX.Y.Z` for the driver.
+4. **Push tags.** The proxy indexes each module at its tag; `go get @vX.Y.Z`
+   now resolves the driver and, transitively, the matching per-platform kernel
+   archive.
+5. **Refresh `go.sum`.** While developing, the `replace` points at the in-tree
+   source so no `go.sum` hash is needed for the nested modules. Once they are
+   published and the driver `require`s the real versions *without* relying on the
+   replace for resolution (i.e. for the tagged release consumers fetch), run
+   `GOFLAGS=-mod=mod GOWORK=off go mod tidy` against the published versions so the
+   nested-module checksums land in `go.sum`. Consumers verify against these.
+   (The committed `replace` still shadows the download in THIS repo's own builds;
+   the `go.sum` entries are what a downstream `go get` verifies.)
+
+## Adding a new platform
+
+1. Create `internal/backend/kernel/kernellib/<platform>/` with its own `go.mod`,
+   a build-tag-gated `link.go` (matching `//go:build` + `#cgo LDFLAGS`), and the
+   committed archive.
+2. Add a `require` + `replace` pair for it in the driver `go.mod`.
+3. Add a build-tagged shim (`cgo_<os>_<arch>.go`) in the `kernel` package that
+   blank-imports the new module (so its `#cgo LDFLAGS` are collected at link).
+4. Tag it alongside the others at release.
+
+## Consumer experience (for reference)
+
+- **Thrift (default):** `go get ...` + `go build` — pure Go, no cgo, no archive
+  downloaded.
+- **Kernel/SEA:** `go get ...` + `CGO_ENABLED=1 go build -tags databricks_kernel`
+  — pulls only the target platform's archive at the driver-pinned version; no
+  `make kernel-lib`, no Rust.
+- **Cloning this repo directly** (contributors/CI): use
+  `git clone --filter=blob:none` to skip the committed-archive history. See the
+  README "Cloning the repository" section.

go.mod

@@ -52,3 +52,24 @@ require (
 	github.com/rs/zerolog v1.28.0
 	golang.org/x/sys v0.45.0 // indirect
 )
+
+// Nested per-platform kernel library modules. Each carries one platform's
+// prebuilt kernel static archive + its cgo link directive; a build downloads
+// only the archive for the platform it targets (build-tag gated), and a pure-Go
+// Thrift build downloads none of them.
+//
+// The `require` versions are REAL published versions, released via path-prefixed
+// tags (e.g. `internal/backend/kernel/kernellib/darwin_arm64/v1.2.3`) and bumped
+// in lockstep with each driver release — see docs/RELEASING.md. A consumer's
+// `go get github.com/databricks/databricks-sql-go@v1.2.3` therefore transitively
+// pins the matching per-platform kernel archive, and upgrading the driver is what
+// moves the kernel version.
+//
+// The `replace` directives point each module at its in-tree source so THIS repo
+// builds against the committed archive during development. Per the Go module
+// spec, a `replace` in a dependency's go.mod is IGNORED by consumers — it only
+// applies when this repo is the main module — so it is safe to ship: it never
+// affects a downstream `go get`, which always resolves the published version.
+require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.1
+
+replace github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 => ./internal/backend/kernel/kernellib/darwin_arm64

internal/backend/kernel/cgo_darwin.go

@@ -2,19 +2,15 @@
 
 package kernel
 
-// Link flags for darwin/arm64. NOTE: this platform is not yet exercised in CI
-// (M0 is linux/amd64); the flags below are the intended shape but must be
-// validated on a mac before darwin is enabled.
+// darwin/arm64 link wiring. The kernel static archive and its `#cgo LDFLAGS`
+// live in a NESTED per-platform module
+// (internal/backend/kernel/kernellib/darwin_arm64) so that a `go get`/`go build`
+// only downloads the darwin archive when actually building for darwin/arm64 with
+// the databricks_kernel tag — see that module's link.go and the repo README.
 //
-// Two darwin-specific differences from linux:
-//   - Apple's ld64 does NOT accept the GNU `-l:<file>.a` extension, so the
-//     archive is passed as a positional input by absolute ${SRCDIR} path
-//     instead. Since only the .a is placed under lib/darwin_arm64 (see
-//     kernel-lib.sh), there is no .so to accidentally prefer.
-//   - -lc++ (not -lstdc++) is the macOS C++ runtime; @loader_path keeps any
-//     dynamic reference resolvable relative to the built binary.
-
-/*
-#cgo LDFLAGS: ${SRCDIR}/lib/darwin_arm64/libdatabricks_sql_kernel.a -lc++ -lm -Wl,-rpath,@loader_path
-*/
-import "C"
+// This file's sole job is to import that module for its link side-effect: cgo
+// collects `#cgo LDFLAGS` from every imported cgo package at final link time, so
+// the blank import below is what pulls libdatabricks_sql_kernel.a into the
+// binary. It carries the same build constraint as the nested link.go so the two
+// are always selected (or excluded) together.
+import _ "github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64"