Skip to content

Update lapis and pgmoon to utilize connection pooling - #482

Open
cycomachead wants to merge 1 commit into
mainfrom
cycomachead/47-update-lapis-pgmoon-pooling/8
Open

cycomachead wants to merge 1 commit into
mainfrom
cycomachead/47-update-lapis-pgmoon-pooling/8

Conversation

@cycomachead

Copy link
Copy Markdown
Member

Update lapis/pgmoon and cap database connections per worker

Upgrades lapis (1.19.0) and pgmoon (1.18.0), and uses pgmoon's pool_size/backlog connect options (built on OpenResty's existing cosocket keepalive pool) to bound the total number of Postgres connections the app can hold per nginx worker. This addresses connection exhaustion on prod/staging that pgbouncer wasn't solving.

Note: pgmoon's PostgresPool class (referenced in the ticket) does not actually help here — it multiplexes coroutines within a single request over several connections and can't be shared across requests via OpenResty cosockets. The real fix is enforcing a hard per-worker connection cap via pool_size/backlog, which is what's implemented.

Changes

  • Dependency bump: snapcloud-dev-0.rockspec and luarocks.lock now both pin lapis 1.19.0 and pgmoon 1.18.0 (previously mismatched between rockspec and lockfile).
  • Connection pool sizing (config.lua): per-worker pool_size is derived from (DATABASE_MAX_CONNECTIONS - DATABASE_RESERVED_CONNECTIONS) / num_workers, with sane defaults (100 / 20) and env var overrides (DATABASE_MAX_CONNECTIONS, DATABASE_RESERVED_CONNECTIONS, DATABASE_POOL_SIZE, DATABASE_POOL_BACKLOG). Worker counts are now named constants shared between num_workers and the pool calculation. Connections are tagged with application_name = 'snapcloud' for monitoring.
  • Dev pooling support: code_cache can now be enabled via CODE_CACHE=on, since the connection pool lives in the Lua VM and is discarded when the code cache is off (the dev default).
  • nginx.conf: passes the new pool env vars through to Lua workers, and adds DATABASE_HOST/DATABASE_PORT which were previously missing (silently falling back to 127.0.0.1:5432).
  • Health check (api.lua): reports current_db_connections, max_db_connections, and connection_capacity (based on pg_stat_activity counts by application_name), sets status: warning and sends a Sentry alert when usage exceeds 90%. Removed the old check that looked up a snapcloud DB user.
  • Docs: docs/DEPLOYMENT.md gains a "Database connections" section (sizing formula, env vars, failure modes, monitoring query, code-cache caveat). docs/INSTALL.md documents where pool_size/backlog are defined across lapis/pgmoon/lua-nginx-module/Postgres, plus a local recipe for exercising the pool.

Resulting caps with defaults (max_connections=100, reserved=20):

Environment Workers pool_size Max app connections
production 8 10 80
staging 2 40 80
development 1 80 80

Reviewer notes

  • Nothing has been deployed. The lapis 1.19 rock upgrade still needs to happen on servers, and bin/deploy's luarocks step is currently disabled.
  • The formula assumes Postgres' default max_connections=100; verify with show max_connections; on prod/staging (production is reportedly on 200) and set DATABASE_MAX_CONNECTIONS accordingly.
  • backlog requires a modern-enough OpenResty (ngx_lua); confirmed compatible with production's OpenResty 1.15.8.3 (ngx_lua 0.10.15).
  • Overflow requests surface as 500s in Sentry with "too many waiting connect operations" — this is the intended signal of DB saturation, in addition to the new health-check warning at >90% capacity.
  • Verified locally: with CODE_CACHE=on and DATABASE_POOL_SIZE=2, 40 concurrent requests never exceeded 2 snapcloud connections in pg_stat_activity; with the code cache off, the same burst opened 4 connections, confirming the dev caveat.

Superconductor Ticket Implementation | App Preview | Guided Review

Upgrade lapis (1.19.0) and pgmoon (1.18.0) and configure per-worker connection pooling using pgmoon's `pool_size` and `backlog` options. This bounds total database connections based on environment capacity and prevents exhaustion. Also updates health check endpoint to report connection pool metrics and send Sentry alerts when approaching capacity.

https://www.superconductor.com/tickets/FJNJ7MNRjRpn/implementations/kKw9Np7kJQBB#message_MRqpW8fLRJR6

Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant