Python environment

To get started hacking, install Python 3.10+ and uv. Then run:

$ uv sync --all-packages

Master configuration

The master is deployed via Docker Compose, which manages three services: a PostgreSQL database, the Buildbot master, and a Caddy reverse proxy with automatic HTTPS.

halide-llvm (and other) wheels used to be hosted here via a bundled pypiserver instance at pypi.halide-lang.org, fronted by this same Caddy. That's gone: the MIT-hosted VM this appliance runs on turned out to be subject to intermittent border-security IP quarantines outside anyone's control (shared cloud egress IPs occasionally get flagged for unrelated scanning activity from other tenants, with no exception mechanism available). Package hosting has moved to halide/pypi (GitHub Releases + Pages), which routes CI traffic off that network path entirely. pypi.halide-lang.org now points there via DNS; this repo no longer serves it.

Secrets

Five secrets control authentication with external users and servers. These will need to be determined before starting up a new master.

1. Obtain a GitHub personal access token with at least the repo scope enabled (other scopes that are not currently used but might be later are write:packages and delete:packages). Call this GITHUB_TOKEN.
2. Generate a secret for the workers to authenticate with the master. Call this WORKER_SECRET.
3. Generate a secret to authenticate GitHub's webhook updates with the master. Call this WEBHOOK_SECRET.
4. Choose a password for the halidenightly user to authenticate with the web interface. Call this WWW_PASSWORD.
5. Generate a password for the PostgreSQL database. Call this DB_PASSWORD. This is only needed when using PostgreSQL (i.e., when HALIDE_BB_MASTER_DB_URL contains {DB_PASSWORD}). The default SQLite backend does not require it.

A convenient command for generating a secure secret is openssl rand -hex 20.

Write all the secrets to the corresponding files:

$ echo "$GITHUB_TOKEN" > secrets/github_token.txt
$ echo "$WORKER_SECRET" > secrets/halide_bb_pass.txt
$ echo "$WEBHOOK_SECRET" > secrets/webhook_token.txt
$ echo "$WWW_PASSWORD" > secrets/buildbot_www_pass.txt
$ echo "$DB_PASSWORD" > secrets/db_password.txt

GitHub configuration

Make your way to the Webhooks section of your repository settings. The url is https://github.com/{owner}/{repo}/settings/hooks. The following settings are the correct ones:

1. Payload URL: https://buildbot.halide-lang.org/master/change_hook/github
2. Content type: application/json
3. Secret: $WEBHOOK_SECRET
4. SSL verification: Select Enable SSL verification
5. Which events would you like to trigger this webhook? a. Let me select individual events. Check "Pull requests" and "Pushes".

Starting the master

$ docker compose up -d --build

The database is automatically initialized on first start. Caddy handles TLS certificates, so there is no manual web server configuration needed. The Buildbot web interface is available at https://buildbot.halide-lang.org/master/.

Port 9990 must be reachable by workers. Port 8012 is internal only; Caddy proxies it on ports 80/443.

Running without Docker

By default, the master uses a SQLite database (sqlite:///state.sqlite), so no external database is required. To use PostgreSQL instead, set HALIDE_BB_MASTER_DB_URL:

$ export HALIDE_BB_MASTER_DB_URL="postgresql://buildbot:{DB_PASSWORD}@db/buildbot"

Then start the master:

$ ./master.sh start

The script automatically runs upgrade-master before starting. If the master is already running, invoking ./master.sh with no arguments will reconfig it instead.

Worker configuration

The master recognizes workers by their reported names, e.g. linux-worker-4 or win-worker-1.

Linux

Write the worker secret and set the worker name, then use the wrapper script:

$ echo "$WORKER_SECRET" > worker/halide_bb_pass.txt
$ export HALIDE_BB_WORKER_NAME=$WORKER_NAME
$ ./worker.sh

Windows

> Set-Content worker/halide_bb_pass.txt "$WORKER_SECRET"
> $env:HALIDE_BB_WORKER_NAME = "$WORKER_NAME"
> .\worker.ps1

Optional environment variables

Variable	Default	Description
HALIDE_BB_MASTER_ADDR	buildbot.halide-lang.org	Master hostname
HALIDE_BB_MASTER_PORT	9990	Master PB port

Platform-specific installation

Automated installation scripts that set up system dependencies and configure the worker to start automatically are provided under worker/:

• Windows: worker/windows/install.ps1 — installs dependencies via winget, sets up Visual Studio 2022, installs uv, and bootstraps vcpkg with the required libraries.