ouranos/docs/ouranos.md

# Ouranos Lab

Infrastructure-as-Code project managing the **Ouranos Lab** — a development sandbox at [ouranos.helu.ca](https://ouranos.helu.ca). Uses **Terraform** for container provisioning and **Ansible** for configuration management, themed around the moons of Uranus.

---

## Project Overview

| Component | Purpose |
|-----------|---------|
| **Terraform** | Provisions 10 specialised Incus containers (LXC) with DNS-resolved networking, security policies, and resource dependencies |
| **Ansible** | Deploys Docker, databases (PostgreSQL, Neo4j), observability stack (Prometheus, Grafana, Loki), and application runtimes across all hosts |

> **DNS Domain**: Incus resolves containers via the `.incus` domain suffix (e.g., `oberon.incus`, `portia.incus`). IPv4 addresses are dynamically assigned — always use DNS names, never hardcode IPs.

## Project Numbers
- External Apps
  - Well known: Postgresl, ssh, web, prometheus
  - 220: External Apps (legacy)
  - 290: External App 1
  - 299: External App 9
- Django Projects: 
  - 221: Zelus
  - 222: Angelia
  - 224: Athena
  - 225: Kairos
  - 226: Icarlos
  - 227: MCP Switchboard (227), Spelunker (228), Peitho (229), Mnemosyne (230)
- FastAgent Projects:
  - 240: Pallas Iolaus
  - 241: Pallas Kottos
  - 242: Pallas Mentor
- FastAPI Projects: 
  - 200: Daedalus
  - 201: Arke
  - 202: Kernos
  - 203: Rommie
  - 204: Orpheus
  - 205: Periplus
  - 206: Nike 
  - 207: Stentor 
  - 208: Argos
  - 209: Hecate
  - 210: Rhema 
  - 211: Synesis

## Port Numbering

Well-known ports running as a service may be used: Postgresql 5432, Prometheus Metrics 9100.

However inside a docker project, the number plan needs to be followed to avoid port conflicts and confusion:
XXXYZ
XXX Project Number or 290-299 for external project (host specific)
Y Service: 0 reserved, 1-4 flexible, 5 database, 6 MCP, 7 API, 8 Web App, 9 Prometheus metrics
Z Instance: The running instance of this app on the same host, starting at 1.  May also be used to handle exceptions.

255 Incus port forwarding: Ports in this range are forwarded from the Incus host to Incus containers (defined in Terraform), but HAProxy through Titania

| Range | Host | Purpose |
|-------|------|---------|
| 25510–25519 | caliban | 25512→22 SSH, 25515→5432 Postgres, 25516→8006 web, 25517→8007 web, 25518→8008 web, 25519→3389 RDP |
| 25530–25539 | miranda | MCP containers |
| 25540–25544 | sycorax | Arke LLM proxy |
| 25554 | ariel | Neo4j |
| 25555 | umbriel | Neo4j (Mnemosyne) |
| 25560–25569 | miranda | MCPO ports |
| 25570–25589 | puck | 25570–25588 app ports, 25589→3389 RDP |
| 25590–25599 | oberon | App ports |

514ZZ is the syslog port.  Docker containers send their syslog to an Alloy syslog collector port.  ZZ is the application instance, they just need to be different on the same host and increment from 01.

## Uranian Host Architecture

All containers are named after moons of Uranus and resolved via the `.incus` DNS suffix.

| Name | Role | Description | Nesting |
|------|------|-------------|---------|
| **ariel** | graph_database | Neo4j — Ethereal graph connections | ✔ |
| **caliban** | agent_automation | Agent S MCP Server with MATE Desktop | ✔ |
| **miranda** | mcp_docker_host | Dedicated Docker Host for MCP Servers | ✔ |
| **oberon** | container_orchestration | Docker Host — MCP Switchboard, RabbitMQ, Open WebUI | ✔ |
| **portia** | database | PostgreSQL — Relational database host | ❌ |
| **prospero** | observability | PPLG stack — Prometheus, Grafana, Loki, PgAdmin | ❌ |
| **puck** | application_runtime | Python App Host — JupyterLab, Django apps, Gitea Runner | ✔ |
| **rosalind** | collaboration | Gitea, LobeChat, Nextcloud, AnythingLLM | ✔ |
| **sycorax** | language_models | Arke LLM Proxy | ✔ |
| **titania** | proxy_sso | HAProxy TLS termination + Casdoor SSO | ✔ |
| **umbriel** | graph_database | Neo4j (Mnemosyne) — dedicated memory graph | ✔ |

### puck — Project Application Runtime

Shape-shifting trickster embodying Python's versatility.
This is the host that runs Python projects in the Ouranos sandbox.
It has an RDP server and is generally where application development happens.
Each project has a number that is used to determine port numbers.


### caliban — Agent Automation

Autonomous computer agent learning through environmental interaction.

- Docker engine
- Agent S MCP Server (MATE desktop, AT-SPI automation)
- Kernos MCP Shell Server
- Rommie MCP Server — agent-to-agent GUI automation via Agent S
- FreeCAD Robust MCP Server — CAD automation via FreeCAD XML-RPC
- GPU passthrough
- RDP access

### oberon — Container Orchestration & Dockerized Shared Services

King of the Fairies orchestrating containers and managing MCP infrastructure.

- Docker engine
- RabbitMQ message queue
- smtp4dev SMTP test server

### portia — Relational Database

Intelligent and resourceful — the reliability of relational databases.

- PostgreSQL 17 (port 5432)
- Databases: `arke`, `anythingllm`, `gitea`, `hass`, `lobechat`, `mcp_switchboard`, `mnemosyne`, `nextcloud`, `openwebui`, `periplus`, `spelunker`

### ariel — Graph Database

Air spirit — ethereal, interconnected nature mirroring graph relationships.
- Neo4j (Docker)

### umbriel — Graph Database (Mnemosyne)

Dusky melancholy sprite from Pope's *Rape of the Lock* — keeper of the Cave of
Spleen, naturally paired with Mnemosyne the Titan of memory. Dedicated Neo4j
instance so Mnemosyne's `Library`/`Collection`/`Item`/`Chunk`/`Concept` labels,
vector indexes, and schema migrations can't collide with another tenant's
graph on Ariel.

- Neo4j (Docker)

### miranda — MCP Docker Host

Curious bridge between worlds — hosting MCP server containers.

- Docker engine 
- MCPO OpenAI-compatible MCP 
- Argos MCP Server — web search via SearXNG
- Grafana MCP Server
- Neo4j MCP Server
- Gitea MCP Server

### prospero — Observability Stack

Master magician observing all events.

- PPLG stack via Docker Compose: Prometheus, Loki, Grafana, PgAdmin
- Internal HAProxy with OAuth2-Proxy for all dashboards
- AlertManager with Pushover notifications
- Prometheus metrics collection (`node-exporter`, HAProxy, Loki)
- Loki log aggregation via Alloy (all hosts)
- Grafana dashboard suite with Casdoor SSO integration

### rosalind — Third Party Applications for testing and evaluation

Witty and resourceful moon for PHP, Go, and Node.js runtimes.

- SearXNG privacy search
- Gitea self-hosted Git
- Nextcloud file sharing and collaboration 
- Jellyfin media server (port 22086, NVIDIA transcoding, Casdoor SSO)
- Nextcloud data on dedicated Incus storage volume
- Open WebUI LLM interface (port 22088, PostgreSQL backend on Portia
- Home Assistant (port 8123)

### sycorax — Language Models

Original magical power wielding language magic.

- Arke LLM API Proxy
- Multi-provider support (OpenAI, Anthropic, etc.)
- Session management with Memcached
- Database backend on Portia

### titania — Proxy & SSO Services

Queen of the Fairies managing access control and authentication.

- HAProxy 3.x with TLS termination 
- Let's Encrypt wildcard certificate via certbot DNS-01 (Namecheap)
- HTTP to HTTPS redirect (port 80)
- Gitea SSH proxy (port 22022)
- Casdoor SSO (port 22081, local PostgreSQL)
- Prometheus metrics at `:8404/metrics`

---


## Application Conventions

Standards that all services deployed in Ouranos MUST follow. For full logging standards and anti-patterns, see [red_panda_standards.md](red_panda_standards.md).

### Health Check Endpoints

All services MUST expose Kubernetes-style health endpoints:

| Endpoint | Purpose | Auth |
|----------|---------|------|
| `GET /live` | **Liveness** — process is running and accepting connections | None |
| `GET /ready` | **Readiness** — process is running AND all dependencies (DB, cache, upstream APIs) are healthy | None |
| `GET /metrics` | Prometheus metrics (see below) | IP-restricted |

- HAProxy checks `health_path` (typically `/ready/`) for backend health — return HTTP 200 when healthy
- Health endpoints MUST NOT require authentication (no JWT, no session)
- Third-party services use their native health paths (e.g., `/api/health`, `/api/healthz`, `/-/healthy`)

### Health Checks in Docker Compose

Use `curl -f` for Docker Compose healthchecks. Install curl in images if needed.

```yaml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/live"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s
```

### Logging Conventions

Log output flows through: **App → syslog (RFC3164) → Alloy → Loki → Grafana**

| Level | Usage |
|-------|-------|
| **ERROR** | Broken state requiring human action — always include `exc_info=True`, error type, and context |
| **WARNING** | Degraded but recovering — client disconnects, performance outliers, client-side exceptions, leaked markup |
| **INFO** | Lifecycle events — service start/stop, connections, requests completed, jobs finished |
| **DEBUG** | Diagnostic detail — SSE events, keepalive pings, health check 200 responses, negotiation steps |

**Health check responses MUST be logged at DEBUG only.** HAProxy and Prometheus probe endpoints every 15-30 seconds. Logging these at INFO floods syslog with thousands of identical `200 OK` lines per hour, burying real events.

### Protected vs Unprotected Endpoints

| Protected (require valid JWT) | Unprotected |
|-------------------------------|-------------|
| All `/api/v1/*` routes | `GET /live` |
| | `GET /ready` |
| | `GET /metrics` (IP-restricted to internal networks) |
| | `GET /api/auth/login-url` |
| | `POST /api/auth/token` |
| | `POST /api/v1/telemetry` (sendBeacon cannot set headers) |

### Prometheus Metrics

All services SHOULD expose `GET /metrics` in Prometheus exposition format, scraped by Prospero's Prometheus (default 15s interval).

- **IP-restricted** to internal networks only (`10.10.0.0/24`, `172.16.0.0/12`, `127.0.0.0/8`)
- Consider exposing: request counts/durations, error rates, active connections, queue depths, dependency health

### Browser Telemetry

Frontend/browser code MUST send telemetry data and errors back to the application's telemetry API:

- `POST /api/v1/telemetry` — unprotected (browser `sendBeacon` cannot set Authorization headers)
- Capture and report: JavaScript exceptions, performance metrics, user-facing errors
- Client-side exceptions should log as **WARNING** on the server (they indicate a problem but not a server-side failure)

### Docker Networking

- Use the **default Docker bridge network** for simple deployments
- Add additional named networks only when required (e.g., isolating database traffic) or explicitly requested
- Do not create custom network definitions for single-service Docker Compose stacks

---

## External Access via HAProxy

Titania provides TLS termination and reverse proxy for all services.

- **Base domain**: `ouranos.helu.ca`
- **HTTPS**: port 443 (standard)
- **HTTP**: port 80 (redirects to HTTPS)
- **Certificate**: Let's Encrypt wildcard via certbot DNS-01

### Subdomains

Refer to the Ansible Titania host inventory (`inventory/host_vars/titania.incus.yml`) for current backend routing configuration.

---

## Infrastructure Management

### Quick Start

```bash
# Provision containers
cd terraform
terraform init
terraform plan
terraform apply

# Start all containers
cd ../ansible
source ~/env/ouranos/bin/activate
ansible-playbook sandbox_up.yml

# Deploy all services
ansible-playbook site.yml

# Stop all containers
ansible-playbook sandbox_down.yml
```

### Python Virtual Environment Setup

The Ansible automation requires a Python virtual environment with the `ansible` package installed. Create and activate the environment from the `~` directory:

```bash
# Create virtual environment
cd ~
python3 -m venv env/ouranos

# Activate environment
source ~/env/ouranos/bin/activate

# Install Ansible
pip install ansible
pip install ansible-core
pip install ansible-community.postgresql
```

### Ansible Playbook Syntax Check

Before running playbooks, use the `apsc.sh` utility (in PATH) to quickly validate YAML syntax:

```bash
# From the ansible directory
apsc.sh

# This will check all YAML files in the current directory for syntax errors
```

### Terraform Workflow

1. **Define** — Containers, networks, and resources in `*.tf` files
2. **Plan** — Review changes with `terraform plan`
3. **Apply** — Provision with `terraform apply`
4. **Verify** — Check outputs and container status

### Terraform Import

When containers or other resources are created manually (outside Terraform) or need to be re-imported after recreation, use `terraform import` to sync the Terraform state with existing infrastructure.

#### Import Syntax

The correct import format for Incus resources requires quoting resource addresses with `for_each` keys and using the full ID including image fingerprints:

```bash
# Import a container with correct syntax
terraform import 'incus_instance.uranian_hosts["<name>"]' ouranos/<name>,image=<fingerprint>
```

#### Getting Image Fingerprints

First, get the fingerprint of the image resource from Terraform state:

```bash
cd terraform
terraform state show incus_image.noble | grep fingerprint
# Output: fingerprint = "75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644"

terraform state show incus_image.questing | grep fingerprint
# Output: fingerprint = "e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf"
```

#### Importing All Uranian Hosts

Replace containers missing from state (or re-import after manual recreation):

```bash
# Containers using noble image
terraform import 'incus_instance.uranian_hosts["ariel"]' ouranos/ariel,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["miranda"]' ouranos/miranda,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["oberon"]' ouranos/oberon,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["portia"]' ouranos/portia,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["prospero"]' ouranos/prospero,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["rosalind"]' ouranos/rosalind,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["sycorax"]' ouranos/sycorax,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["titania"]' ouranos/titania,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
terraform import 'incus_instance.uranian_hosts["umbriel"]' ouranos/umbriel,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644

# Containers using questing image
terraform import 'incus_instance.uranian_hosts["caliban"]' ouranos/caliban,image=e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf
terraform import 'incus_instance.uranian_hosts["puck"]' ouranos/puck,image=e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf
```

#### Storage Bucket Import

For storage buckets, use the `<project>/<pool>/<name>` format:

```bash
terraform import incus_storage_bucket.<name> ouranos/default/<bucket-name>
```

#### Common Issues

1. **Import ID format errors**: Use quotes around resource addresses with `for_each` keys: `'incus_instance.uranian_hosts["name"]'`

2. **Image replacement on import**: Importing without specifying the image fingerprint will cause Terraform to replace the container on next apply. Always include `image=<fingerprint>` in the import ID.

3. **Tainted state**: If a resource shows "will be created" but already exists, it may be tainted. Remove from state and re-import:
   ```bash
   terraform state rm 'incus_instance.uranian_hosts["name"]'
   terraform import 'incus_instance.uranian_hosts["name"]' ouranos/name,image=<fingerprint>
   ```

#### Verify Import

After importing, verify with `terraform plan`:

```bash
terraform plan
# Should show: Plan: 0 to add, 0 to change, 0 to destroy
# (Minor "update in-place" changes are normal for state sync of computed attributes)
```

### Ansible Workflow

1. **Bootstrap** — Update packages, install essentials (`apt_update.yml`)
2. **Agents** — Deploy Alloy (log/metrics) and Node Exporter on all hosts
3. **Services** — Configure databases, Docker, applications, observability
4. **Verify** — Check service health and connectivity

### Vault Management

```bash
# Edit secrets
ansible-vault edit inventory/group_vars/all/vault.yml

# View secrets
ansible-vault view inventory/group_vars/all/vault.yml

# Encrypt a new file
ansible-vault encrypt new_secrets.yml
```

---

## S3 Storage Provisioning

Terraform provisions Incus S3 buckets for services requiring object storage:

| Name                | Description                      |
|---------------------|----------------------------------|
| `casdoor`           | Casdoor file storage bucket      |
| `daedalus`          | Daedalus file storage bucket     |
| `lobechat`          | Lobechat file storage bucket     |
| `mnemosyne-content` | Mnemosyne content storage bucket |
| `spelunker`         | Spelunker file storage bucket    |

> S3 credentials (access key, secret key, endpoint) are stored as sensitive Terraform outputs and managed in Ansible Vault with the `vault_*_s3_*` prefix.

### Retrieving S3 Bucket Credentials

The bucket credentials are declared as **sensitive** Terraform outputs, so a plain
`terraform output` will mask them. Use the `-json` (or `-raw`) flag to reveal the
values:

```bash
cd terraform

# List all outputs (sensitive values shown as <sensitive>)
terraform output

# Show a specific bucket's credentials as JSON
terraform output -json casdoor_s3_credentials
terraform output -json daedalus_s3_credentials
terraform output -json lobechat_s3_credentials
terraform output -json mnemosyne_s3_credentials
terraform output -json spelunker_s3_credentials

# Extract a single field (e.g. access_key) with jq
terraform output -json casdoor_s3_credentials | jq -r .access_key
terraform output -json casdoor_s3_credentials | jq -r .secret_key
terraform output -json casdoor_s3_credentials | jq -r .endpoint
```

Each `*_s3_credentials` output contains `bucket`, `access_key`, `secret_key`, and
`endpoint`. Copy these into `inventory/group_vars/all/vault.yml` as
`vault_<service>_s3_access_key`, `vault_<service>_s3_secret_key`, etc.

---

## Ansible Automation

### Full Deployment (`site.yml`)

Playbooks run in dependency order:

| Playbook | Hosts | Purpose |
|----------|-------|---------|
| `apt_update.yml` | All | Update packages and install essentials |
| `alloy/deploy.yml` | All | Grafana Alloy log/metrics collection |
| `prometheus/node_deploy.yml` | All | Node Exporter metrics |
| `docker/deploy.yml` | Oberon, Ariel, Miranda, Puck, Rosalind, Sycorax, Caliban, Titania | Docker engine |
| `smtp4dev/deploy.yml` | Oberon | SMTP test server |
| `pplg/deploy.yml` | Prospero | Full observability stack + HAProxy + OAuth2-Proxy |
| `postgresql/deploy.yml` | Portia | PostgreSQL with all databases |
| `postgresql_ssl/deploy.yml` | Titania | Dedicated PostgreSQL for Casdoor |
| `neo4j/deploy.yml` | Ariel, Umbriel | Neo4j graph database (Umbriel is the dedicated Mnemosyne instance) |
| `searxng/deploy.yml` | Oberon | SearXNG privacy search |
| `haproxy/deploy.yml` | Titania | HAProxy TLS termination and routing |
| `casdoor/deploy.yml` | Titania | Casdoor SSO |
| `mcpo/deploy.yml` | Miranda | MCPO MCP proxy |
| `openwebui/deploy.yml` | Oberon | Open WebUI LLM interface |
| `hass/deploy.yml` | Oberon | Home Assistant |
| `gitea/deploy.yml` | Rosalind | Gitea self-hosted Git |
| `nextcloud/deploy.yml` | Rosalind | Nextcloud collaboration |

### Individual Service Deployments

Services with standalone deploy playbooks (not in `site.yml`):

| Playbook | Host | Service |
|----------|------|---------|
| `anythingllm/deploy.yml` | Rosalind | AnythingLLM document AI |
| `arke/deploy.yml` | Sycorax | Arke LLM proxy |
| `argos/deploy.yml` | Miranda | Argos MCP web search server |
| `caliban/deploy.yml` | Caliban | Agent S MCP Server |
| `certbot/deploy.yml` | Titania | Let's Encrypt certificate renewal |
| `gitea_mcp/deploy.yml` | Miranda | Gitea MCP Server |
| `gitea_runner/deploy.yml` | Puck | Gitea CI/CD runner |
| `grafana_mcp/deploy.yml` | Miranda | Grafana MCP Server |
| `jellyfin/deploy.yml` | Rosalind | Jellyfin media server |
| `jupyterlab/deploy.yml` | Puck | JupyterLab + OAuth2-Proxy |
| `kernos/deploy.yml` | Caliban | Kernos MCP shell server |
| `lobechat/deploy.yml` | Rosalind | LobeChat AI chat |
| `rommie/deploy.yml` | Caliban | Rommie MCP server (Agent S GUI automation) |
| `neo4j_mcp/deploy.yml` | Miranda | Neo4j MCP Server |
| `freecad_mcp/deploy.yml` | Caliban | FreeCAD Robust MCP Server |
| `rabbitmq/deploy.yml` | Oberon | RabbitMQ message queue |

### Lifecycle Playbooks

| Playbook | Purpose |
|----------|---------|
| `sandbox_up.yml` | Start all Uranian host containers |
| `sandbox_down.yml` | Gracefully stop all containers |
| `apt_update.yml` | Update packages on all hosts |
| `site.yml` | Full deployment orchestration |

---

## Data Flow Architecture

### Observability Pipeline

```
All Hosts                      Prospero                         Alerts
Alloy + Node Exporter     →   Prometheus + Loki + Grafana   →  AlertManager + Pushover
collect metrics & logs         storage & visualisation           notifications
```

### Integration Points

| Consumer | Provider | Connection |
|----------|----------|-----------|
| All LLM apps | Arke (Sycorax) | `http://sycorax.incus:25540` |
| Open WebUI, Arke, Gitea, Nextcloud, LobeChat | PostgreSQL (Portia) | `portia.incus:5432` |
| Neo4j MCP | Neo4j (Ariel) | `ariel.incus:7687` (Bolt) |
| Mnemosyne | Neo4j (Umbriel) | `umbriel.incus:7687` (Bolt) — dedicated tenant |
| MCP Switchboard | Docker API (Miranda) | `tcp://miranda.incus:2375` |
| MCP Switchboard | RabbitMQ (Oberon) | `oberon.incus:5672` |
| Kairos, Spelunker | RabbitMQ (Oberon) | `oberon.incus:5672` |
| SMTP (all apps) | smtp4dev (Oberon) | `oberon.incus:22025` |
| All hosts | Loki (Prospero) | `http://prospero.incus:3100` |
| All hosts | Prometheus (Prospero) | `http://prospero.incus:9090` |

---

## Important Notes

⚠️ **Alloy Host Variables Required** — Every host with `alloy` in its `services` list must define `alloy_log_level` in `inventory/host_vars/<host>.incus.yml`. The playbook will fail with an undefined variable error if this is missing.

⚠️ **Alloy Syslog Listeners Required for Docker Services** — Any Docker Compose service using the syslog logging driver must have a corresponding `loki.source.syslog` listener in the host's Alloy config template (`ansible/alloy/<hostname>/config.alloy.j2`). Missing listeners cause Docker containers to fail on start.

⚠️ **Local Terraform State** — This project uses local Terraform state (no remote backend). Do not run `terraform apply` from multiple machines simultaneously.

⚠️ **Nested Docker** — Docker runs inside Incus containers (nested), requiring `security.nesting = true` and `lxc.apparmor.profile=unconfined` AppArmor override on all Docker-enabled hosts.

⚠️ **Deployment Order** — Prospero (observability) must be fully deployed before other hosts, as Alloy on every host pushes logs and metrics to `prospero.incus`. Run `pplg/deploy.yml` before `site.yml` on a fresh environment.