# PostgreSQL - Dual-Deployment Database Layer

## Overview

PostgreSQL 17 serves as the primary relational database engine for the Agathos sandbox. There are **two separate deployment playbooks**, each targeting a different host with a distinct purpose:

| Playbook | Host | Purpose |
|----------|------|---------|
| `postgresql/deploy.yml` | **Portia** | Shared multi-tenant database with **pgvector** for AI/vector workloads |
| `postgresql_ssl/deploy.yml` | **Titania** | Dedicated SSL-enabled database for the **Casdoor** identity provider |

**Portia** acts as the central database server for most applications, while **Titania** runs an isolated PostgreSQL instance exclusively for Casdoor, hardened with self-signed SSL certificates for secure external connections.

## Architecture

```
                          ┌────────────────────────────────────────────────────┐
                          │              Portia (postgresql)                   │
  ┌──────────┐            │  ┌──────────────────────────────────────────────┐  │
  │  Arke    │───────────▶│  │  PostgreSQL 17 + pgvector v0.8.0            │  │
  │(Caliban) │            │  │                                              │  │
  ├──────────┤            │  │  Databases:                                  │  │
  │  Gitea   │───────────▶│  │    arke ─── openwebui ─── spelunker         │  │
  │(Rosalind)│            │  │    gitea ── lobechat ──── nextcloud          │  │
  ├──────────┤            │  │    anythingllm ────────── hass               │  │
  │ Open     │───────────▶│  │                                              │  │
  │ WebUI    │            │  │  pgvector enabled in:                        │  │
  ├──────────┤            │  │    arke, lobechat, openwebui,                │  │
  │ LobeChat │───────────▶│  │    spelunker, anythingllm                    │  │
  ├──────────┤            │  └──────────────────────────────────────────────┘  │
  │  HASS    │───────────▶│                                                    │
  │ + others │            │  PgAdmin available on :25555                       │
  └──────────┘            └────────────────────────────────────────────────────┘

                          ┌────────────────────────────────────────────────────┐
                          │             Titania (postgresql_ssl)               │
  ┌──────────┐            │  ┌──────────────────────────────────────────────┐  │
  │ Casdoor  │──SSL──────▶│  │  PostgreSQL 17 + SSL (self-signed)          │  │
  │(Titania) │  (local)   │  │                                              │  │
  └──────────┘            │  │  Database: casdoor (single-purpose)          │  │
                          │  └──────────────────────────────────────────────┘  │
                          └────────────────────────────────────────────────────┘
```

## Terraform Resources

### Portia – Shared Database Host

Defined in `terraform/containers.tf`:

| Attribute | Value |
|-----------|-------|
| Image | noble |
| Role | database |
| Security Nesting | false |
| Proxy Devices | `25555 → 80` (PgAdmin web UI) |

PostgreSQL port 5432 is **not** exposed externally—applications connect over the private Incus network (`10.10.0.0/16`).

### Titania – Proxy & SSO Host

| Attribute | Value |
|-----------|-------|
| Image | noble |
| Role | proxy_sso |
| Security Nesting | true |
| Proxy Devices | `443 → 8443`, `80 → 8080` (HAProxy) |

Titania runs PostgreSQL alongside Casdoor on the same host. Casdoor connects via localhost, so SSL is not required for the local connection despite being available for external clients.

## Ansible Deployment

### Playbook 1: Shared PostgreSQL with pgvector (Portia)

```bash
cd ansible
ansible-playbook postgresql/deploy.yml
```

#### Files

| File | Purpose |
|------|---------|
| `postgresql/deploy.yml` | Multi-tenant PostgreSQL with pgvector |

#### Deployment Steps

1. **Install build dependencies** — `curl`, `git`, `build-essential`, `vim`, `python3-psycopg2`
2. **Add PGDG repository** — Official PostgreSQL APT repository
3. **Install PostgreSQL 17** — Client, server, docs, `libpq-dev`, `server-dev`
4. **Clone & build pgvector v0.8.0** — Compiled from source against the installed PG version
5. **Start PostgreSQL** and restart after pgvector installation
6. **Set data directory permissions** — `700` owned by `postgres:postgres`
7. **Configure networking** — `listen_addresses = '*'`
8. **Configure authentication** — `host all all 0.0.0.0/0 md5` in `pg_hba.conf`
9. **Set admin password** — `postgres` superuser password from vault
10. **Create application users** — 9 database users (see table below)
11. **Create application databases** — 9 databases with matching owners
12. **Enable pgvector** — `CREATE EXTENSION vector` in 5 databases

### Playbook 2: SSL-Enabled PostgreSQL (Titania)

```bash
cd ansible
ansible-playbook postgresql_ssl/deploy.yml
```

#### Files

| File | Purpose |
|------|---------|
| `postgresql_ssl/deploy.yml` | Single-purpose SSL PostgreSQL for Casdoor |

#### Deployment Steps

1. **Install dependencies** — `curl`, `python3-psycopg2`, `python3-cryptography`
2. **Add PGDG repository** — Official PostgreSQL APT repository
3. **Install PostgreSQL 17** — Client and server only (no dev packages needed)
4. **Generate SSL certificates** — 4096-bit RSA key, self-signed, 10-year validity
5. **Configure networking** — `listen_addresses = '*'`
6. **Enable SSL** — `ssl = on` with cert/key file paths
7. **Configure tiered authentication** in `pg_hba.conf`:
   - `local` → `peer` (Unix socket, no password)
   - `host 127.0.0.1/32` → `md5` (localhost, no SSL)
   - `host 10.10.0.0/16` → `md5` (Incus network, no SSL)
   - `hostssl 0.0.0.0/0` → `md5` (external, SSL required)
8. **Set admin password** — `postgres` superuser password from vault
9. **Create Casdoor user and database** — Single-purpose

## User & Database Creation via Host Variables

Both playbooks derive all database names, usernames, and passwords from **host variables** defined in the Ansible inventory. No database credentials appear in `group_vars`—everything is scoped to the host that runs PostgreSQL.

### Portia Host Variables (`inventory/host_vars/portia.incus.yml`)

The `postgresql/deploy.yml` playbook loops over variable pairs to create users and databases. Each application gets three variables defined in Portia's host_vars:

| Variable Pattern | Example | Description |
|-----------------|---------|-------------|
| `{app}_db_name` | `arke_db_name: arke` | Database name |
| `{app}_db_user` | `arke_db_user: arke` | Database owner/user |
| `{app}_db_password` | `arke_db_password: "{{ vault_arke_db_password }}"` | Password (from vault) |

#### Application Database Matrix (Portia)

| Application | DB Name Variable | DB User Variable | pgvector |
|-------------|-----------------|-----------------|----------|
| Arke | `arke_db_name` | `arke_db_user` | ✔ |
| Open WebUI | `openwebui_db_name` | `openwebui_db_user` | ✔ |
| Spelunker | `spelunker_db_name` | `spelunker_db_user` | ✔ |
| Gitea | `gitea_db_name` | `gitea_db_user` | |
| LobeChat | `lobechat_db_name` | `lobechat_db_user` | ✔ |
| Nextcloud | `nextcloud_db_name` | `nextcloud_db_user` | |
| AnythingLLM | `anythingllm_db_name` | `anythingllm_db_user` | ✔ |
| HASS | `hass_db_name` | `hass_db_user` | |
| Nike | `nike_db_name` | `nike_db_user` | |

#### Additional Portia Variables

| Variable | Description |
|----------|-------------|
| `postgres_user` | System user (`postgres`) |
| `postgres_group` | System group (`postgres`) |
| `postgresql_port` | Port (`5432`) |
| `postgresql_data_dir` | Data directory (`/var/lib/postgresql`) |
| `postgres_password` | Admin password (`{{ vault_postgres_password }}`) |

### Titania Host Variables (`inventory/host_vars/titania.incus.yml`)

The `postgresql_ssl/deploy.yml` playbook creates a single database for Casdoor:

| Variable | Value | Description |
|----------|-------|-------------|
| `postgresql_ssl_postgres_password` | `{{ vault_postgresql_ssl_postgres_password }}` | Admin password |
| `postgresql_ssl_port` | `5432` | PostgreSQL port |
| `postgresql_ssl_cert_path` | `/etc/postgresql/17/main/ssl/server.crt` | SSL certificate |
| `casdoor_db_name` | `casdoor` | Database name |
| `casdoor_db_user` | `casdoor` | Database user |
| `casdoor_db_password` | `{{ vault_casdoor_db_password }}` | Password (from vault) |
| `casdoor_db_sslmode` | `disable` | Local connection skips SSL |

### Adding a New Application Database

To add a new application database on Portia:

1. **Add variables** to `inventory/host_vars/portia.incus.yml`:
   ```yaml
   myapp_db_name: myapp
   myapp_db_user: myapp
   myapp_db_password: "{{ vault_myapp_db_password }}"
   ```

2. **Add the vault secret** to `inventory/group_vars/all/vault.yml`:
   ```yaml
   vault_myapp_db_password: "s3cure-passw0rd"
   ```

3. **Add the user** to the `Create application database users` loop in `postgresql/deploy.yml`:
   ```yaml
   - { user: "{{ myapp_db_user }}", password: "{{ myapp_db_password }}" }
   ```

4. **Add the database** to the `Create application databases with owners` loop:
   ```yaml
   - { name: "{{ myapp_db_name }}", owner: "{{ myapp_db_user }}" }
   ```

5. **(Optional)** If the application uses vector embeddings, add the database to the `Enable pgvector extension in databases` loop:
   ```yaml
   - "{{ myapp_db_name }}"
   ```

## Operations

### Start/Stop

```bash
# On either host
sudo systemctl start postgresql
sudo systemctl stop postgresql
sudo systemctl restart postgresql
```

### Health Check

```bash
# From any Incus host → Portia
psql -h portia.incus -U postgres -c "SELECT 1;"

# From Titania localhost
sudo -u postgres psql -c "SELECT 1;"

# Check pgvector availability
sudo -u postgres psql -c "SELECT * FROM pg_available_extensions WHERE name = 'vector';"
```

### Logs

```bash
# Systemd journal
journalctl -u postgresql -f

# PostgreSQL log files
tail -f /var/log/postgresql/postgresql-17-main.log

# Loki (via Grafana Explore)
{job="postgresql"}
```

### Backup

```bash
# Dump a single database
sudo -u postgres pg_dump myapp > myapp_backup.sql

# Dump all databases
sudo -u postgres pg_dumpall > full_backup.sql
```

### Restore

```bash
# Restore a single database
sudo -u postgres psql myapp < myapp_backup.sql

# Restore all databases
sudo -u postgres psql < full_backup.sql
```

## Troubleshooting

### Common Issues

| Symptom | Cause | Resolution |
|---------|-------|------------|
| Connection refused from app host | `pg_hba.conf` missing entry | Verify client IP is covered by HBA rules |
| pgvector extension not found | Built against wrong PG version | Re-run the `Build pgvector with correct pg_config` task |
| SSL handshake failure (Titania) | Expired or missing certificate | Check `/etc/postgresql/17/main/ssl/server.crt` validity |
| `FATAL: password authentication failed` | Wrong password in host_vars | Verify vault variable matches and re-run playbook |
| PgAdmin unreachable on :25555 | Incus proxy device missing | Check `terraform/containers.tf` proxy for Portia |

## References

- [PostgreSQL 17 Documentation](https://www.postgresql.org/docs/17/)
- [pgvector GitHub](https://github.com/pgvector/pgvector)
- [Terraform Practices](terraform.md)
- [Ansible Practices](ansible.md)