docs: rewrite README with structured overview and quick start guide
Replaces the minimal project description with a comprehensive README including a component overview table, quick start instructions, common Ansible operations, and links to detailed documentation. Aligns with Red Panda Approval™ standards.
This commit is contained in:
287
docs/postgresql.md
Normal file
287
docs/postgresql.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# PostgreSQL - Dual-Deployment Database Layer
|
||||
|
||||
## Overview
|
||||
|
||||
PostgreSQL 17 serves as the primary relational database engine for the Agathos sandbox. There are **two separate deployment playbooks**, each targeting a different host with a distinct purpose:
|
||||
|
||||
| Playbook | Host | Purpose |
|
||||
|----------|------|---------|
|
||||
| `postgresql/deploy.yml` | **Portia** | Shared multi-tenant database with **pgvector** for AI/vector workloads |
|
||||
| `postgresql_ssl/deploy.yml` | **Titania** | Dedicated SSL-enabled database for the **Casdoor** identity provider |
|
||||
|
||||
**Portia** acts as the central database server for most applications, while **Titania** runs an isolated PostgreSQL instance exclusively for Casdoor, hardened with self-signed SSL certificates for secure external connections.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Portia (postgresql) │
|
||||
┌──────────┐ │ ┌──────────────────────────────────────────────┐ │
|
||||
│ Arke │───────────▶│ │ PostgreSQL 17 + pgvector v0.8.0 │ │
|
||||
│(Caliban) │ │ │ │ │
|
||||
├──────────┤ │ │ Databases: │ │
|
||||
│ Gitea │───────────▶│ │ arke ─── openwebui ─── spelunker │ │
|
||||
│(Rosalind)│ │ │ gitea ── lobechat ──── nextcloud │ │
|
||||
├──────────┤ │ │ anythingllm ────────── hass │ │
|
||||
│ Open │───────────▶│ │ │ │
|
||||
│ WebUI │ │ │ pgvector enabled in: │ │
|
||||
├──────────┤ │ │ arke, lobechat, openwebui, │ │
|
||||
│ LobeChat │───────────▶│ │ spelunker, anythingllm │ │
|
||||
├──────────┤ │ └──────────────────────────────────────────────┘ │
|
||||
│ HASS │───────────▶│ │
|
||||
│ + others │ │ PgAdmin available on :25555 │
|
||||
└──────────┘ └────────────────────────────────────────────────────┘
|
||||
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Titania (postgresql_ssl) │
|
||||
┌──────────┐ │ ┌──────────────────────────────────────────────┐ │
|
||||
│ Casdoor │──SSL──────▶│ │ PostgreSQL 17 + SSL (self-signed) │ │
|
||||
│(Titania) │ (local) │ │ │ │
|
||||
└──────────┘ │ │ Database: casdoor (single-purpose) │ │
|
||||
│ └──────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Terraform Resources
|
||||
|
||||
### Portia – Shared Database Host
|
||||
|
||||
Defined in `terraform/containers.tf`:
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| Image | noble |
|
||||
| Role | database |
|
||||
| Security Nesting | false |
|
||||
| Proxy Devices | `25555 → 80` (PgAdmin web UI) |
|
||||
|
||||
PostgreSQL port 5432 is **not** exposed externally—applications connect over the private Incus network (`10.10.0.0/16`).
|
||||
|
||||
### Titania – Proxy & SSO Host
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| Image | noble |
|
||||
| Role | proxy_sso |
|
||||
| Security Nesting | true |
|
||||
| Proxy Devices | `443 → 8443`, `80 → 8080` (HAProxy) |
|
||||
|
||||
Titania runs PostgreSQL alongside Casdoor on the same host. Casdoor connects via localhost, so SSL is not required for the local connection despite being available for external clients.
|
||||
|
||||
## Ansible Deployment
|
||||
|
||||
### Playbook 1: Shared PostgreSQL with pgvector (Portia)
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-playbook postgresql/deploy.yml
|
||||
```
|
||||
|
||||
#### Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `postgresql/deploy.yml` | Multi-tenant PostgreSQL with pgvector |
|
||||
|
||||
#### Deployment Steps
|
||||
|
||||
1. **Install build dependencies** — `curl`, `git`, `build-essential`, `vim`, `python3-psycopg2`
|
||||
2. **Add PGDG repository** — Official PostgreSQL APT repository
|
||||
3. **Install PostgreSQL 17** — Client, server, docs, `libpq-dev`, `server-dev`
|
||||
4. **Clone & build pgvector v0.8.0** — Compiled from source against the installed PG version
|
||||
5. **Start PostgreSQL** and restart after pgvector installation
|
||||
6. **Set data directory permissions** — `700` owned by `postgres:postgres`
|
||||
7. **Configure networking** — `listen_addresses = '*'`
|
||||
8. **Configure authentication** — `host all all 0.0.0.0/0 md5` in `pg_hba.conf`
|
||||
9. **Set admin password** — `postgres` superuser password from vault
|
||||
10. **Create application users** — 9 database users (see table below)
|
||||
11. **Create application databases** — 9 databases with matching owners
|
||||
12. **Enable pgvector** — `CREATE EXTENSION vector` in 5 databases
|
||||
|
||||
### Playbook 2: SSL-Enabled PostgreSQL (Titania)
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-playbook postgresql_ssl/deploy.yml
|
||||
```
|
||||
|
||||
#### Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `postgresql_ssl/deploy.yml` | Single-purpose SSL PostgreSQL for Casdoor |
|
||||
|
||||
#### Deployment Steps
|
||||
|
||||
1. **Install dependencies** — `curl`, `python3-psycopg2`, `python3-cryptography`
|
||||
2. **Add PGDG repository** — Official PostgreSQL APT repository
|
||||
3. **Install PostgreSQL 17** — Client and server only (no dev packages needed)
|
||||
4. **Generate SSL certificates** — 4096-bit RSA key, self-signed, 10-year validity
|
||||
5. **Configure networking** — `listen_addresses = '*'`
|
||||
6. **Enable SSL** — `ssl = on` with cert/key file paths
|
||||
7. **Configure tiered authentication** in `pg_hba.conf`:
|
||||
- `local` → `peer` (Unix socket, no password)
|
||||
- `host 127.0.0.1/32` → `md5` (localhost, no SSL)
|
||||
- `host 10.10.0.0/16` → `md5` (Incus network, no SSL)
|
||||
- `hostssl 0.0.0.0/0` → `md5` (external, SSL required)
|
||||
8. **Set admin password** — `postgres` superuser password from vault
|
||||
9. **Create Casdoor user and database** — Single-purpose
|
||||
|
||||
## User & Database Creation via Host Variables
|
||||
|
||||
Both playbooks derive all database names, usernames, and passwords from **host variables** defined in the Ansible inventory. No database credentials appear in `group_vars`—everything is scoped to the host that runs PostgreSQL.
|
||||
|
||||
### Portia Host Variables (`inventory/host_vars/portia.incus.yml`)
|
||||
|
||||
The `postgresql/deploy.yml` playbook loops over variable pairs to create users and databases. Each application gets three variables defined in Portia's host_vars:
|
||||
|
||||
| Variable Pattern | Example | Description |
|
||||
|-----------------|---------|-------------|
|
||||
| `{app}_db_name` | `arke_db_name: arke` | Database name |
|
||||
| `{app}_db_user` | `arke_db_user: arke` | Database owner/user |
|
||||
| `{app}_db_password` | `arke_db_password: "{{ vault_arke_db_password }}"` | Password (from vault) |
|
||||
|
||||
#### Application Database Matrix (Portia)
|
||||
|
||||
| Application | DB Name Variable | DB User Variable | pgvector |
|
||||
|-------------|-----------------|-----------------|----------|
|
||||
| Arke | `arke_db_name` | `arke_db_user` | ✔ |
|
||||
| Open WebUI | `openwebui_db_name` | `openwebui_db_user` | ✔ |
|
||||
| Spelunker | `spelunker_db_name` | `spelunker_db_user` | ✔ |
|
||||
| Gitea | `gitea_db_name` | `gitea_db_user` | |
|
||||
| LobeChat | `lobechat_db_name` | `lobechat_db_user` | ✔ |
|
||||
| Nextcloud | `nextcloud_db_name` | `nextcloud_db_user` | |
|
||||
| AnythingLLM | `anythingllm_db_name` | `anythingllm_db_user` | ✔ |
|
||||
| HASS | `hass_db_name` | `hass_db_user` | |
|
||||
| Nike | `nike_db_name` | `nike_db_user` | |
|
||||
|
||||
#### Additional Portia Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `postgres_user` | System user (`postgres`) |
|
||||
| `postgres_group` | System group (`postgres`) |
|
||||
| `postgresql_port` | Port (`5432`) |
|
||||
| `postgresql_data_dir` | Data directory (`/var/lib/postgresql`) |
|
||||
| `postgres_password` | Admin password (`{{ vault_postgres_password }}`) |
|
||||
|
||||
### Titania Host Variables (`inventory/host_vars/titania.incus.yml`)
|
||||
|
||||
The `postgresql_ssl/deploy.yml` playbook creates a single database for Casdoor:
|
||||
|
||||
| Variable | Value | Description |
|
||||
|----------|-------|-------------|
|
||||
| `postgresql_ssl_postgres_password` | `{{ vault_postgresql_ssl_postgres_password }}` | Admin password |
|
||||
| `postgresql_ssl_port` | `5432` | PostgreSQL port |
|
||||
| `postgresql_ssl_cert_path` | `/etc/postgresql/17/main/ssl/server.crt` | SSL certificate |
|
||||
| `casdoor_db_name` | `casdoor` | Database name |
|
||||
| `casdoor_db_user` | `casdoor` | Database user |
|
||||
| `casdoor_db_password` | `{{ vault_casdoor_db_password }}` | Password (from vault) |
|
||||
| `casdoor_db_sslmode` | `disable` | Local connection skips SSL |
|
||||
|
||||
### Adding a New Application Database
|
||||
|
||||
To add a new application database on Portia:
|
||||
|
||||
1. **Add variables** to `inventory/host_vars/portia.incus.yml`:
|
||||
```yaml
|
||||
myapp_db_name: myapp
|
||||
myapp_db_user: myapp
|
||||
myapp_db_password: "{{ vault_myapp_db_password }}"
|
||||
```
|
||||
|
||||
2. **Add the vault secret** to `inventory/group_vars/all/vault.yml`:
|
||||
```yaml
|
||||
vault_myapp_db_password: "s3cure-passw0rd"
|
||||
```
|
||||
|
||||
3. **Add the user** to the `Create application database users` loop in `postgresql/deploy.yml`:
|
||||
```yaml
|
||||
- { user: "{{ myapp_db_user }}", password: "{{ myapp_db_password }}" }
|
||||
```
|
||||
|
||||
4. **Add the database** to the `Create application databases with owners` loop:
|
||||
```yaml
|
||||
- { name: "{{ myapp_db_name }}", owner: "{{ myapp_db_user }}" }
|
||||
```
|
||||
|
||||
5. **(Optional)** If the application uses vector embeddings, add the database to the `Enable pgvector extension in databases` loop:
|
||||
```yaml
|
||||
- "{{ myapp_db_name }}"
|
||||
```
|
||||
|
||||
## Operations
|
||||
|
||||
### Start/Stop
|
||||
|
||||
```bash
|
||||
# On either host
|
||||
sudo systemctl start postgresql
|
||||
sudo systemctl stop postgresql
|
||||
sudo systemctl restart postgresql
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
```bash
|
||||
# From any Incus host → Portia
|
||||
psql -h portia.incus -U postgres -c "SELECT 1;"
|
||||
|
||||
# From Titania localhost
|
||||
sudo -u postgres psql -c "SELECT 1;"
|
||||
|
||||
# Check pgvector availability
|
||||
sudo -u postgres psql -c "SELECT * FROM pg_available_extensions WHERE name = 'vector';"
|
||||
```
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
# Systemd journal
|
||||
journalctl -u postgresql -f
|
||||
|
||||
# PostgreSQL log files
|
||||
tail -f /var/log/postgresql/postgresql-17-main.log
|
||||
|
||||
# Loki (via Grafana Explore)
|
||||
{job="postgresql"}
|
||||
```
|
||||
|
||||
### Backup
|
||||
|
||||
```bash
|
||||
# Dump a single database
|
||||
sudo -u postgres pg_dump myapp > myapp_backup.sql
|
||||
|
||||
# Dump all databases
|
||||
sudo -u postgres pg_dumpall > full_backup.sql
|
||||
```
|
||||
|
||||
### Restore
|
||||
|
||||
```bash
|
||||
# Restore a single database
|
||||
sudo -u postgres psql myapp < myapp_backup.sql
|
||||
|
||||
# Restore all databases
|
||||
sudo -u postgres psql < full_backup.sql
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Symptom | Cause | Resolution |
|
||||
|---------|-------|------------|
|
||||
| Connection refused from app host | `pg_hba.conf` missing entry | Verify client IP is covered by HBA rules |
|
||||
| pgvector extension not found | Built against wrong PG version | Re-run the `Build pgvector with correct pg_config` task |
|
||||
| SSL handshake failure (Titania) | Expired or missing certificate | Check `/etc/postgresql/17/main/ssl/server.crt` validity |
|
||||
| `FATAL: password authentication failed` | Wrong password in host_vars | Verify vault variable matches and re-run playbook |
|
||||
| PgAdmin unreachable on :25555 | Incus proxy device missing | Check `terraform/containers.tf` proxy for Portia |
|
||||
|
||||
## References
|
||||
|
||||
- [PostgreSQL 17 Documentation](https://www.postgresql.org/docs/17/)
|
||||
- [pgvector GitHub](https://github.com/pgvector/pgvector)
|
||||
- [Terraform Practices](terraform.md)
|
||||
- [Ansible Practices](ansible.md)
|
||||
Reference in New Issue
Block a user