Update lapis and pgmoon to utilize connection pooling - #482
Open
cycomachead wants to merge 1 commit into
Open
cycomachead wants to merge 1 commit into
cycomachead wants to merge 1 commit into
Conversation
Upgrade lapis (1.19.0) and pgmoon (1.18.0) and configure per-worker connection pooling using pgmoon's `pool_size` and `backlog` options. This bounds total database connections based on environment capacity and prevents exhaustion. Also updates health check endpoint to report connection pool metrics and send Sentry alerts when approaching capacity. https://www.superconductor.com/tickets/FJNJ7MNRjRpn/implementations/kKw9Np7kJQBB#message_MRqpW8fLRJR6 Co-authored-by: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update lapis/pgmoon and cap database connections per worker
Upgrades lapis (1.19.0) and pgmoon (1.18.0), and uses pgmoon's
pool_size/backlogconnect options (built on OpenResty's existing cosocket keepalive pool) to bound the total number of Postgres connections the app can hold per nginx worker. This addresses connection exhaustion on prod/staging that pgbouncer wasn't solving.Note: pgmoon's
PostgresPoolclass (referenced in the ticket) does not actually help here — it multiplexes coroutines within a single request over several connections and can't be shared across requests via OpenResty cosockets. The real fix is enforcing a hard per-worker connection cap viapool_size/backlog, which is what's implemented.Changes
snapcloud-dev-0.rockspecandluarocks.locknow both pin lapis 1.19.0 and pgmoon 1.18.0 (previously mismatched between rockspec and lockfile).config.lua): per-workerpool_sizeis derived from(DATABASE_MAX_CONNECTIONS - DATABASE_RESERVED_CONNECTIONS) / num_workers, with sane defaults (100 / 20) and env var overrides (DATABASE_MAX_CONNECTIONS,DATABASE_RESERVED_CONNECTIONS,DATABASE_POOL_SIZE,DATABASE_POOL_BACKLOG). Worker counts are now named constants shared betweennum_workersand the pool calculation. Connections are tagged withapplication_name = 'snapcloud'for monitoring.code_cachecan now be enabled viaCODE_CACHE=on, since the connection pool lives in the Lua VM and is discarded when the code cache is off (the dev default).DATABASE_HOST/DATABASE_PORTwhich were previously missing (silently falling back to127.0.0.1:5432).api.lua): reportscurrent_db_connections,max_db_connections, andconnection_capacity(based onpg_stat_activitycounts byapplication_name), setsstatus: warningand sends a Sentry alert when usage exceeds 90%. Removed the old check that looked up asnapcloudDB user.docs/DEPLOYMENT.mdgains a "Database connections" section (sizing formula, env vars, failure modes, monitoring query, code-cache caveat).docs/INSTALL.mddocuments wherepool_size/backlogare defined across lapis/pgmoon/lua-nginx-module/Postgres, plus a local recipe for exercising the pool.Resulting caps with defaults (max_connections=100, reserved=20):
Reviewer notes
bin/deploy's luarocks step is currently disabled.max_connections=100; verify withshow max_connections;on prod/staging (production is reportedly on 200) and setDATABASE_MAX_CONNECTIONSaccordingly.backlogrequires a modern-enough OpenResty (ngx_lua); confirmed compatible with production's OpenResty 1.15.8.3 (ngx_lua 0.10.15).CODE_CACHE=onandDATABASE_POOL_SIZE=2, 40 concurrent requests never exceeded 2snapcloudconnections inpg_stat_activity; with the code cache off, the same burst opened 4 connections, confirming the dev caveat.Superconductor Ticket Implementation | App Preview | Guided Review