feat(config): expand environment variables in pgdog.toml and users.toml

[ChrisRx]

Configuration files can now reference the process environment, so secrets and per-environment values no longer have to be baked into the files on disk:

[admin]
password = "${PGDOG_ADMIN_PASSWORD}"

[general]
shutdown_timeout = ${PGDOG_SHUTDOWN_TIMEOUT:-60000}

Only the braced form is a reference: ${VAR} is substituted from the environment, ${VAR:-value} supplies a fallback, and $${VAR} is a literal ${VAR}. A bare $VAR is left alone.

Lookups are lenient, so a reference to a variable that isn't set is left in the document verbatim rather than failing the load. Between that and ignoring the unbraced form, existing values keep working even in users.toml where it is most likely to contain a stray $ in passwords. This means passwords like sup$rsecret or p$$w0rd passes through untouched instead of turning into a startup failure or a silently truncated credential.

Expansion runs on the document source before it is parsed, so a variable is interpolated as TOML rather than as a string. ${PASSWORD} in value position still needs its surrounding quotes, and a value containing " or a newline will change how the rest of the document parses. This is what allows bare shutdown_timeout = ${VAR} to work, and it is documented on expand.

Implementation notes:

• New pgdog-config::expand module. expand() is infallible and returns Cow::Borrowed when there is nothing to substitute, so the common case costs no allocation.
• Initially this used shellexpand but was replaced with a simple scanner since shellexpand doesn't have optional configuration that would only allow the terminated/braced form of variables for lookups.
• FromToml::from_toml replaces bare toml::from_str at the three sites that parse config text read from disk: both branches of ConfigAndUsers::load and bootstrap_logger. Every other toml::from_str in the tree parses a test literal, where expansion is unwanted, and is untouched.
• The trait carries a blanket impl over DeserializeOwned, so no per-type boilerplate is needed. from_toml, not from_str, to avoid colliding with the crate's many std::str::FromStr impls.
• Error::config now receives the expanded text, so the line numbers it reports stay correct when a variable's value contains a newline.
• ConfigAndUsers keeps config_text/users_text as the raw, unexpanded source. Resolved secrets must not be written back to disk when the config is reloaded or backed up.

Fixes #1479

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 98.27586% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog-config/src/core.rs	50.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[levkk]

I think all characters are allowed in a Postgres password, e.g., $, { and }, so this is a valid password which will be expanded to an empty string:

${hello}

Curious if you have any thoughts. Maybe we should only expand settings that are entirely covered by an env var, e.g.:

password = "${PASSWORD}" # setting value starts with `${` and ends with `}`

That would require us to perform shellexpand on each value after deserialization (or write a custom serializer).

Just thinking out loud, let me know what you think.

[ChrisRx]

In the case with ${hello} it would need to be set as the password value and also be set in the environment, so unless it has an environment variable for hello= it will keep it as the original string, ultimately leaving as ${hello}.

I was definitely concerned with using shellexpand, I could imagine a situation where generated passwords or especially some longer tokens could easily contain something where it would match shorter commonly set environment variables, like CC, where a randomly generated value like ....$CC..... would get expanded to the value of CC with something like gcc. But I feel a lot better with the new non-shellexpand approach being that it requires:

1. The environment variable must still be set in the process environment
2. It must be a sequence of ${ followed by a }
3. The variable name itself can only contain contain characters [a-zA-Z0-9_] and cannot start with _ or a digit (I based it off of POSIX standard, but including lowercase letters)

I don't know how to go about calculating a probability on it myself, but it seems like it would be practically impossible given the confluence of things that would need to happen coupled with password generators usually don't create passwords that include { or } and tokens like JWTs are usually base64 encoded which would exclude that as well.

I really appreciate the discussion with this btw, I think this kind of thing IME is not something you can think too much about for sure!