172 lines
8.3 KiB
Markdown
172 lines
8.3 KiB
Markdown
# PostgreSQL and MinIO public data stack
|
|
|
|
This Compose stack publishes four encrypted endpoints through Caddy, all on **one host name**
|
|
(`DATA_DOMAIN`, here `data.example.com`), each on its own port:
|
|
|
|
| Service | Address |
|
|
|---|---|
|
|
| MinIO S3 API | `https://data.example.com` (443) |
|
|
| PostgreSQL | `data.example.com:5432`, using the native PostgreSQL TLS negotiation |
|
|
| MinIO console | `https://data.example.com:9443` |
|
|
| pgAdmin (multi-user) | `https://data.example.com:5443` |
|
|
|
|
One name means one DNS record and one certificate: a certificate is issued for a name, not a port,
|
|
so every port above presents the same one. The S3 API is the one on the standard port because it is
|
|
what other systems call and MinIO serves buckets from the root of its host. Opening
|
|
`https://data.example.com/` in a browser therefore shows an XML error from MinIO; that is the API
|
|
answering, not a fault.
|
|
|
|
PostgreSQL and MinIO have no directly published container ports. Caddy is the only public entry
|
|
point; traffic from Caddy to the services stays on an internal Docker network. Caddy persists its
|
|
ACME account, certificates and private keys in `caddy-data`, so restarts do not trigger unnecessary
|
|
certificate reissuance.
|
|
|
|
Two of the images are not stock ones, so they are built once, from `caddy.Dockerfile` and
|
|
`minio.Dockerfile`, and published to Docker Hub; the server only pulls them and compiles nothing.
|
|
|
|
- `luciolelii/caddy-l4` is Caddy with the pinned `caddy-l4` module, because stock Caddy only proxies
|
|
HTTP. The module understands PostgreSQL's initial `SSLRequest`, terminates TLS with Caddy's
|
|
automatically managed certificate, and sends the decrypted connection to PostgreSQL only on the
|
|
private network.
|
|
- `luciolelii/minio` is MinIO built from its latest upstream security release. Upstream did not
|
|
publish a container for that release, so the Dockerfile builds the pinned source tag.
|
|
|
|
Each image's tag is the version its Dockerfile pins (`RELEASE.2025-10-15T17-29-55Z`, and
|
|
`2.11.4-l4-v0.1.2` for Caddy version then module version), and `docker-compose.yml` names that tag.
|
|
Both are built for amd64 and arm64.
|
|
|
|
pgAdmin is pinned to version 9.18 and runs in server mode. Its users, sessions, preferences and
|
|
saved server definitions live in the `pgadmin-data` volume.
|
|
|
|
## Start
|
|
|
|
1. Create public `A`/`AAAA` records for `DATA_DOMAIN` and point them to the server.
|
|
2. Allow inbound TCP ports `80`, `443`, `5432`, `9443` and `5443` in the host/cloud firewall. Restrict
|
|
`5432`, `9443` and `5443` to known client address ranges whenever possible: only `80` and `443`
|
|
have to be open to the whole Internet, for certificate issuance.
|
|
3. Copy the environment template and replace every placeholder:
|
|
|
|
```sh
|
|
cp .env.example .env
|
|
openssl rand -base64 36
|
|
docker compose config --quiet
|
|
docker compose pull
|
|
docker compose up -d
|
|
```
|
|
|
|
4. Follow certificate issuance and startup:
|
|
|
|
```sh
|
|
docker compose logs -f caddy postgres minio pgadmin
|
|
```
|
|
|
|
Ports 80 and 443 must reach Caddy from the public Internet for the usual ACME HTTP/TLS challenges.
|
|
The name must be eligible for Let's Encrypt issuance; a restrictive DNS CAA record can refuse it.
|
|
|
|
## Connect
|
|
|
|
PostgreSQL requires TLS at the public listener. `verify-full` both encrypts the connection and checks
|
|
that the certificate matches the hostname:
|
|
|
|
```sh
|
|
psql "host=data.example.com port=5432 dbname=humainflow user=humainflow sslmode=verify-full"
|
|
```
|
|
|
|
MinIO/S3 clients use `https://data.example.com` with path-style addressing
|
|
(`https://data.example.com/<bucket>/<key>`); administrators open `https://data.example.com:9443`.
|
|
`MINIO_SERVER_URL` is set to the public S3 address so presigned URLs remain valid behind the reverse
|
|
proxy, and `MINIO_BROWSER_REDIRECT_URL` carries the console's port.
|
|
|
|
Open `https://data.example.com:5443` and sign in with `PGADMIN_DEFAULT_EMAIL` and
|
|
`PGADMIN_DEFAULT_PASSWORD`. On the first login, register the database with these values:
|
|
|
|
- host: `postgres` (the Compose service name, not the public hostname);
|
|
- port: `5432`;
|
|
- maintenance database: the value of `POSTGRES_DB`;
|
|
- username/password: a PostgreSQL role and its password.
|
|
|
|
The initial pgAdmin administrator can create additional pgAdmin accounts from User Management.
|
|
pgAdmin accounts only control access to the web interface: create separate least-privilege
|
|
PostgreSQL roles for database authorization. Each person should use their own database role rather
|
|
than sharing `POSTGRES_USER`. Changing the default pgAdmin password in `.env` after the first start
|
|
does not update the account already stored in `pgadmin-data`; change it from pgAdmin instead.
|
|
|
|
## Operations
|
|
|
|
The persistent volumes are `postgres-data`, `minio-data`, `pgadmin-data`, and `caddy-data`. Back up
|
|
the first three; do not treat Docker volumes as backups. Never run `docker compose down -v` unless
|
|
permanent deletion of both data stores, pgAdmin's configuration and Caddy's certificate state is
|
|
intended.
|
|
|
|
Upgrade PostgreSQL one major version at a time using the PostgreSQL upgrade procedure. Updating an
|
|
image tag alone does not migrate an existing database volume.
|
|
|
|
## Storing the data on another disk
|
|
|
|
By default PostgreSQL and MinIO keep their data in the Docker volumes `postgres-data` and
|
|
`minio-data`, which live under `/var/lib/docker` on the system disk. To use a larger disk, mount it
|
|
on the host and point the two variables at directories on it:
|
|
|
|
```env
|
|
POSTGRES_DATA_PATH=/mnt/bigdisk/postgres
|
|
MINIO_DATA_PATH=/mnt/bigdisk/minio
|
|
```
|
|
|
|
The two are independent: set one or both. Unset (or empty) means the named volume, as before.
|
|
|
|
1. Create the directories **inside** the disk, not at its mount point, and give them to the user the
|
|
container runs as. On ext4 the mount point holds `lost+found`, and PostgreSQL will not start in
|
|
a directory that is not empty.
|
|
|
|
```sh
|
|
sudo mkdir -p /mnt/bigdisk/postgres /mnt/bigdisk/minio
|
|
sudo chown 70:70 /mnt/bigdisk/postgres && sudo chmod 700 /mnt/bigdisk/postgres # postgres (alpine)
|
|
sudo chown 1000:1000 /mnt/bigdisk/minio # MinIO, see minio.Dockerfile
|
|
```
|
|
|
|
2. **A new installation** needs nothing more: set the variables and `docker compose up -d`.
|
|
|
|
3. **An installation that already holds data** must copy it first. Do not use `-v` anywhere:
|
|
|
|
```sh
|
|
docker compose down
|
|
docker run --rm -v humainflow-data-stack_postgres-data:/from -v /mnt/bigdisk/postgres:/to alpine cp -a /from/. /to/
|
|
docker run --rm -v humainflow-data-stack_minio-data:/from -v /mnt/bigdisk/minio:/to alpine cp -a /from/. /to/
|
|
# set the two variables in .env, then
|
|
docker compose up -d
|
|
```
|
|
|
|
The volume names carry the project name from `name:` in the compose file. Keep the old volumes
|
|
until everything works, and remove them only then.
|
|
|
|
Cautions:
|
|
|
|
- Use a local or block disk. PostgreSQL on a network filesystem such as NFS risks corruption.
|
|
- Mount the disk at boot (`/etc/fstab`, with `nofail`) and start Docker after it. If the disk is
|
|
missing when the stack starts, Docker creates the directory on the system disk and PostgreSQL
|
|
starts **empty** - it looks like the data is gone, and it is only somewhere else.
|
|
- A backup of `POSTGRES_DATA_PATH` taken while PostgreSQL runs is not consistent; use `pg_dump` or a
|
|
proper base backup.
|
|
|
|
## Upgrading from the four-name layout
|
|
|
|
An earlier version used `POSTGRES_DOMAIN`, `MINIO_API_DOMAIN`, `MINIO_CONSOLE_DOMAIN` and
|
|
`PGADMIN_DOMAIN`. Replace them in `.env` with a single `DATA_DOMAIN` (any one of the old names will
|
|
do, as long as it resolves to this server), open ports `9443` and `5443`, and run
|
|
`docker compose up -d`. Data volumes are untouched. Addresses change: the console moves from
|
|
its own name to `:9443`, pgAdmin to `:5443`, and PostgreSQL clients connect to `DATA_DOMAIN`.
|
|
|
|
## Publishing the MinIO and Caddy images
|
|
|
|
Only needed to change a version. Edit the version in the Dockerfile (`ARG MINIO_VERSION` in
|
|
`minio.Dockerfile`; the `caddy:` tag and the `caddy-l4@` version in `caddy.Dockerfile`), then, from a
|
|
machine with Docker and a Docker Hub login (`docker login -u luciolelii`, with an access token that can
|
|
write):
|
|
|
|
```sh
|
|
./publish-images.sh # both; or: ./publish-images.sh minio ./publish-images.sh caddy
|
|
```
|
|
|
|
and set the new tag in `docker-compose.yml` (or `MINIO_IMAGE` / `CADDY_IMAGE` in `.env`). If the Docker
|
|
Hub repositories are private, run `docker login` on the server before `docker compose pull`.
|