DatasetGateway

Unified authorization service for neuroscience datasets.

DatasetGateway is a single Django service that centralizes dataset access control across multiple platforms:

• Neuroglancer — implements the ngauth protocol for GCS token-based access
• Clio and neuprint — provides authorization APIs these services call to check user permissions
• CAVE — preliminary middle_auth-compatible endpoints are implemented; full support is planned pending CAVE deployment testing, token migration validation, and review
• WebKnossos — planned; will require coordination with ScalableMinds

Quick start

Prerequisites

• pixi
• Docker (for production deployment only)
• A Google OAuth 2.0 client (for login — the setup wizard walks you through it)

One-time setup

cd dsg
pixi install
pixi run setup              # interactive wizard — generates .env, runs migrations

Option A: Local development

pixi run serve

Starts the Django dev server. If .env doesn't exist yet, the setup wizard runs automatically.

To run detached (survives logout, logs to dsg/serve.log, PID in dsg/serve.pid):

pixi run serve-bg
pixi run stop-serve   # to stop

Option B: Docker production

pixi run deploy

Builds the Docker image, starts the container, runs migrations and seed commands. Put a reverse proxy (nginx/caddy) in front for TLS.

Option C: gunicorn behind an existing nginx (no Docker)

For a host that already has nginx terminating TLS (e.g. an emdata server), run gunicorn directly under systemd instead of the dev runserver:

pixi run serve-prod   # gunicorn, DEBUG=False, WhiteNoise static; binds 127.0.0.1:8200

For a supervised process that restarts on crash/reboot, install the systemd unit template at scripts/datasetgateway.service (see the header comments). Point your nginx location / at http://127.0.0.1:8200 and forward X-Forwarded-Proto $scheme so Django's SECURE_PROXY_SSL_HEADER sees the original HTTPS scheme. WhiteNoise serves /static/ from within gunicorn, so nginx needs no location /static/ block.

The Django admin is at /admin/.

Google OAuth setup

Login requires a Google OAuth 2.0 client. Without one the server runs but all login/authorize links will fail with a client_id error. The setup wizard (pixi run setup) will walk you through creating one if secrets/client_credentials.json is missing.

Alternatively, you can set it up manually:

1. Go to the Google Cloud Console and create an OAuth 2.0 Client ID (type: Web application).
2. Add http://localhost:8200/accounts/google/login/callback/ as an authorized redirect URI (and your production URI if known).
3. Download the JSON credentials and save them:

mkdir -p dsg/secrets
cp ~/Downloads/client_secret_*.json dsg/secrets/client_credentials.json

The secrets/ directory is gitignored. For Docker production, prefer GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET in .env, or mount a credentials file and set CLIENT_CREDENTIALS_PATH; dsg/.dockerignore excludes local secrets/ from the image. Alternatively, you can set environment variables instead of using the JSON file:

export GOOGLE_CLIENT_ID="your-client-id.apps.googleusercontent.com"
export GOOGLE_CLIENT_SECRET="your-client-secret"

Authentication

All users authenticate via Google OpenID Connect. On successful login, the server creates a DB-stored API key and sets it as the dsg_token cookie. This single cookie is shared by all services in the ecosystem.

API requests are authenticated by checking for the token in this order:

1. dsg_token cookie
2. Authorization: Bearer {token} header
3. ?dsg_token= query parameter

How each platform authenticates

CAVE services (MaterializationEngine, AnnotationEngine, etc.) call DatasetGateway's /api/v1/user/cache endpoint on every request to validate the user's token and retrieve their permissions. DatasetGateway has a preliminary implementation of the middle_auth-compatible endpoints, but it is not yet declared supported until tested with a real CAVE deployment. For a fresh deployment or planned migration where clients obtain DSG-minted Bearer tokens and CAVE services point AUTH_URL / STICKY_AUTH_URL at DatasetGateway, existing middle_auth_client Bearer-token flows should not require service code changes. Existing cookie/query-token flows that depend on middle_auth_token need a DSG login/token transition or compatibility configuration because DatasetGateway uses dsg_token.

Neuroglancer uses the ngauth protocol. Users log in via a popup that hits /auth/login → Google OAuth → dsg_token cookie. Because Neuroglancer runs on a different origin (e.g., neuroglancer.org), it cannot read the cookie directly. Instead it calls POST /token, which reads the cookie server-side and returns a short-lived token. Neuroglancer then exchanges that token for a time-limited GCS access credential via POST /gcs_token, which grants read access to the specific cloud storage bucket holding the dataset. Before minting, DatasetGateway resolves the bucket through every matching DatasetBucket row and applies its own unscoped grant, group, version, bucket, and TOS rules. It does not use the caller's per-user bucket IAM as an authorization probe. Public datasets need no grant but still require any applicable TOS acceptance.

Other services (neuPrint, celltyping-light, Clio) validate users by calling /api/v1/user/cache with the dsg_token value, the same way CAVE services do. When all services share a cookie domain (configured via AUTH_COOKIE_DOMAIN), users log in once and are authenticated everywhere.

Running tests

cd dsg
pixi run -e dev python -m pytest

Production deployment

DatasetGateway is designed for a single-server Docker deployment behind a reverse proxy that handles TLS.

cd dsg
pixi run setup    # generates .env interactively (set DJANGO_DEBUG=False for production)
pixi run deploy   # builds Docker image, starts container, runs migrations + seeds

Then create an admin user:

docker compose -f docker-compose.yml exec dsg python manage.py make_admin user@example.com

Put a reverse proxy (nginx or Caddy) in front for TLS, pointed at localhost:8080. The setup wizard defaults SECURE_SSL_REDIRECT=False since most deployments terminate TLS at the proxy.

The SQLite database and static files are stored in Docker volumes (dsg-data and dsg-static) so they survive container restarts. If you need PostgreSQL or Redis, swap the DATABASES / CACHES settings and add services to docker-compose.yml.

Without Docker (gunicorn + systemd)

On a host with its own nginx, skip Docker and run gunicorn under systemd (see Option C above and scripts/datasetgateway.service). Production prerequisites are the same either way:

• DJANGO_DEBUG=False — enables Secure cookies, HSTS, and generic error pages; Django then refuses to start unless DJANGO_SECRET_KEY and DJANGO_ALLOWED_HOSTS are set.
• DJANGO_SECRET_KEY — a strong random secret used to sign sessions, CSRF tokens, and password-reset/signed values. Generate one with python -c "import secrets; print(secrets.token_urlsafe(64))" and keep it out of source control.
• collectstatic runs automatically in serve-prod.sh; WhiteNoise serves the result, so the admin UI is styled without runserver's DEBUG-only static.
• Keep GUNICORN_WORKERS=1 while CACHES is the per-process LocMemCache (the permission cache is not shared across workers); raise it only after moving to a shared cache backend.
• Install the scheduled-maintenance systemd timer templates: scripts/datasetgateway-backup.{service,timer} (hourly encrypted DB backups; see docs/backups.md) and scripts/datasetgateway-iamsync.{service,timer} (daily GCS bucket-IAM reconcile backstop; see docs/admin-manual.md).

Environment variables

Variable	Default	Description
DJANGO_SECRET_KEY	insecure dev key	Secret key for sessions and CSRF. Set in production.
DJANGO_DEBUG	True	Set to False in production.
DJANGO_ALLOWED_HOSTS	*	Comma-separated list of allowed hostnames.
DATABASE_PATH	db.sqlite3	Path to SQLite database file.
SECURE_SSL_REDIRECT	True (prod)	Set to False if reverse proxy handles TLS.
DSG_ORIGIN	(empty)	Public origin for CSRF trusted origins (e.g., https://dataset-gateway.mydomain.org).
DSG_PORT	8200	Port for the development server.
GOOGLE_CLIENT_ID	(empty)	Google OAuth 2.0 client ID (overrides client_credentials.json).
GOOGLE_CLIENT_SECRET	(empty)	Google OAuth 2.0 client secret (overrides client_credentials.json).
CLIENT_CREDENTIALS_PATH	secrets/client_credentials.json	Alternative OAuth client credentials path. Useful when mounting credentials into Docker.
NGAUTH_ALLOWED_ORIGINS	^https?://.*\.neuroglancer\.org$	Regex for allowed CORS origins.
TOS_RETURN_ALLOWED_ORIGINS	(empty)	Comma-separated exact origins allowed as TOS service-check return targets.
AUTH_COOKIE_DOMAIN	(empty)	Cookie domain for cross-subdomain auth (e.g., .example.org).
PORT	8080	Port for gunicorn (Docker).
GUNICORN_WORKERS	2	Number of gunicorn worker processes.
LOG_LEVEL	info	Gunicorn log level.

Documentation

• Documentation index — status markers for living reference docs vs historical design records.
• User manual — setup, admin workflows, user workflows, management commands
• CAVE auth endpoints — CAVE API compatibility reference and SCIM 2.0 provisioning
• Admin manual — administration and operational reference
• Service accounts — non-human identity and token workflows
• Design archive — historical architecture and implementation records, not automatically synchronized with code changes