docs(env): expand .env.example into full compose interpolation template

Replace the minimal placeholder .env.example with a comprehensive template
documenting every variable consumed by docker-compose.yaml, organized by
service (Django core, HTTP, Postgres, Neo4j, Memcached, S3/MinIO, Daedalus,
Celery/RabbitMQ, etc.). Clarifies that this file is rendered from an Ansible
Jinja2 template with vaulted secrets in production, and distinguishes it
from the in-tree mnemosyne/.env used for bare-Python development.

.env.example

@@ -1,13 +1,133 @@
 # =============================================================================
-# Mnemosyne — Docker Compose environment
+# Mnemosyne — docker compose interpolation template
 # =============================================================================
-# This file documents variables consumed by docker-compose.yaml itself
+# This file is consumed by `docker compose` as the source for `${VAR}`
-# (image tags, port overrides, etc.). It is NOT the application config.
+# interpolations in docker-compose.yaml. In production it is generated from
+# a Jinja2 template by an Ansible role, with secrets pulled from the Ansible
+# vault — do not commit a populated copy.
 #
-# Application config lives in mnemosyne/.env — copy mnemosyne/.env\ example
+# Copy to `.env` (at the repo root, NOT inside `mnemosyne/`) and fill in the
-# to mnemosyne/.env and fill in your values before running `docker compose up`.
+# blanks before running `docker compose up -d`. The in-tree `mnemosyne/.env`
+# file (used by bare-Python development on caliban) is a separate concern
+# and is NOT read by the compose stack.
 #
-# This file has no required variables for a default deployment: the compose
+# Every variable below is referenced by at least one service in
-# file uses a fixed image tag and port. Add overrides here if you parameterise
+# docker-compose.yaml. Per-service scoping (which container sees which var)
-# those in docker-compose.yaml (e.g. MNEMOSYNE_IMAGE, MNEMOSYNE_PORT).
+# is defined by the `environment:` blocks in that file; this template just
+# provides the values.
 # =============================================================================
+
+# --- Django core ------------------------------------------------------------
+# Consumed by: app, mcp, worker
+SECRET_KEY=change-me-to-a-real-secret-key
+DEBUG=False
+TIME_ZONE=UTC
+LANGUAGE_CODE=en-us
+
+# --- HTTP surface -----------------------------------------------------------
+# Consumed by: app (CSRF_TRUSTED_ORIGINS: app only; ALLOWED_HOSTS: app + mcp)
+# Include every hostname HAProxy routes to this stack, plus localhost for the
+# inter-container health probes.
+ALLOWED_HOSTS=localhost,127.0.0.1,mnemosyne.ouranos.helu.ca
+CSRF_TRUSTED_ORIGINS=https://mnemosyne.ouranos.helu.ca
+
+# --- PostgreSQL (Portia) ----------------------------------------------------
+# Consumed by: app, mcp, worker
+APP_DB_NAME=mnemosyne
+APP_DB_USER=mnemosyne
+APP_DB_PASSWORD=change-me
+DB_HOST=portia.incus
+DB_PORT=5432
+
+# --- Neo4j (Umbriel — dedicated Mnemosyne instance) -------------------------
+# Consumed by: app, mcp, worker
+# Umbriel MUST be dedicated to Mnemosyne; do not share with Spelunker or any
+# other graph workload. See README.md for the full rationale.
+NEOMODEL_NEO4J_BOLT_URL=bolt://neo4j:change-me@umbriel.incus:7687
+
+# --- Memcached --------------------------------------------------------------
+# Consumed by: app, mcp, worker
+# Must resolve from inside containers — 127.0.0.1 will NOT work.
+KVDB_LOCATION=oberon.incus:11211
+KVDB_PREFIX=mnemosyne
+
+# --- S3 / MinIO (Nyx) — Mnemosyne's own bucket ------------------------------
+# Consumed by: app, mcp, worker
+# Mnemosyne writes chunk text and item files here. Set USE_LOCAL_STORAGE=False
+# in production so the S3Boto3Storage backend is used instead of the local
+# FileSystemStorage fallback.
+USE_LOCAL_STORAGE=False
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_STORAGE_BUCKET_NAME=mnemosyne-content
+AWS_S3_ENDPOINT_URL=https://nyx.helu.ca:8555
+AWS_S3_USE_SSL=True
+AWS_S3_VERIFY=False
+AWS_S3_REGION_NAME=us-east-1
+
+# --- Daedalus S3 (cross-bucket reads for ingest) ----------------------------
+# Consumed by: worker only
+# Mnemosyne's ingest Celery task reads files from Daedalus's bucket and
+# copies them into AWS_STORAGE_BUCKET_NAME for processing. These creds
+# should be scoped read-only to the Daedalus bucket in your secret manager.
+DAEDALUS_S3_ENDPOINT_URL=https://nyx.helu.ca:8555
+DAEDALUS_S3_ACCESS_KEY_ID=
+DAEDALUS_S3_SECRET_ACCESS_KEY=
+DAEDALUS_S3_BUCKET_NAME=daedalus
+DAEDALUS_S3_REGION_NAME=us-east-1
+DAEDALUS_S3_USE_SSL=True
+DAEDALUS_S3_VERIFY=True
+
+# --- Celery / RabbitMQ (Oberon) ---------------------------------------------
+# Consumed by: app (producer), worker (consumer). NOT mcp.
+# Remember to percent-encode any password characters that have meaning in a
+# URL (`@ : / # % + ? & =` and space). Kombu's AMQP URL parser is strict —
+# an unencoded password is the most common cause of PLAIN 403 failures when
+# the bare-Python client happens to connect fine.
+CELERY_BROKER_URL=amqp://mnemosyne:change-me@oberon.incus:5672/mnemosyne
+CELERY_RESULT_BACKEND=rpc://
+CELERY_TASK_ALWAYS_EAGER=False
+
+# --- Worker tuning ---------------------------------------------------------
+# Consumed by: worker only (read by entrypoint.sh → `celery -A mnemosyne worker`)
+# Override per host if you want to dedicate a worker to a single queue.
+CELERY_QUEUES=celery,embedding,batch
+CELERY_CONCURRENCY=2
+
+# --- MCP server -------------------------------------------------------------
+# Consumed by: mcp only
+MCP_REQUIRE_AUTH=True
+
+# --- LLM API encryption -----------------------------------------------------
+# Consumed by: app (admin pages), worker (ingest vision pass). NOT mcp.
+# Generate once per deployment, store in the vault, never rotate without
+# re-encrypting every stored provider key first.
+# python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
+LLM_API_SECRETS_ENCRYPTION_KEY=
+
+# --- Email (smtp4dev on Oberon) --------------------------------------------
+# Consumed by: app only
+EMAIL_HOST=oberon.incus
+EMAIL_PORT=22025
+EMAIL_USE_TLS=False
+
+# --- Embedding pipeline -----------------------------------------------------
+# Consumed by: worker only
+EMBEDDING_BATCH_SIZE=8
+EMBEDDING_TIMEOUT=120
+
+# --- Search & re-ranker -----------------------------------------------------
+# Consumed by: app, mcp. Not worker (workers never serve queries).
+SEARCH_VECTOR_TOP_K=50
+SEARCH_FULLTEXT_TOP_K=30
+SEARCH_GRAPH_MAX_DEPTH=2
+SEARCH_RRF_K=60
+SEARCH_DEFAULT_LIMIT=20
+RERANKER_MAX_CANDIDATES=32
+RERANKER_TIMEOUT=30
+
+# --- Logging ----------------------------------------------------------------
+# Consumed by: app, mcp, worker (each picks the levels it cares about)
+LOGGING_LEVEL=INFO
+CELERY_LOGGING_LEVEL=INFO
+DJANGO_LOGGING_LEVEL=WARNING

README.md

@@ -168,11 +168,34 @@ Production runs as four containers from a single image (built and pushed by [`.g
 
 Plus a one-shot `static-init` service that copies `/app/staticfiles` (baked into the image at build time via `collectstatic`) into the shared volume nginx reads from. It runs to completion on every `up`, so static-file changes propagate on each deploy without manual intervention.
 
-External services (NOT spun up by compose): Postgres on Portia, Neo4j on Umbriel (dedicated Mnemosyne instance), RabbitMQ on Oberon, S3/MinIO on Nyx, Memcached, embedder + reranker. All reached over the internal 10.10.0.0/24 network. URLs and credentials live in `mnemosyne/.env`.
+External services (NOT spun up by compose): Postgres on Portia, Neo4j on Umbriel (dedicated Mnemosyne instance), RabbitMQ on Oberon, S3/MinIO on Nyx, Memcached, embedder + reranker. All reached over the internal 10.10.0.0/24 network.
+
+### Environment scoping
+
+Each compose service declares *only* the environment variables it actually needs — there is no shared `env_file:`. The rationale:
+
+- The MCP server (the most exposed surface, because it talks to outside LLMs) should never see the Celery broker URL or the LLM API encryption key. It only needs Postgres, Neo4j, Memcached, S3, and the MCP-specific auth toggle.
+- The Celery worker has no business knowing `ALLOWED_HOSTS`, `CSRF_TRUSTED_ORIGINS`, `MCP_REQUIRE_AUTH`, or the email backend — it doesn't serve HTTP.
+- The Django app doesn't need the Daedalus S3 credentials — only the ingest Celery task reads that bucket.
+- When a shared secret (like the broker password) is mis-configured, the blast radius is limited to the services that actually need that secret, so you can still observe the rest of the stack while debugging.
+
+Values are interpolated from a `.env` file at the **repo root** (not `mnemosyne/.env`, which is the dev config for bare-Python runs). Copy `.env.example` to `.env` and fill in the blanks, or — in production — have your Ansible role render `.env` from a Jinja2 template with secrets from the vault.
+
+```bash
+cp .env.example .env
+$EDITOR .env       # fill in SECRET_KEY, DB/RabbitMQ/S3 creds, LLM_API_SECRETS_ENCRYPTION_KEY
+```
+
+The per-service surface is defined by the `environment:` blocks in `docker-compose.yaml`; `.env.example` documents every variable with which service(s) consume it.
+
+> **Broker URL gotcha.** If the RabbitMQ password contains any of `@ : / # % + ? & =` or a space, it must be percent-encoded in `CELERY_BROKER_URL`. Kombu's URL parser is strict, and this is the most common cause of a `PLAIN 403 ACCESS_REFUSED` at worker startup when the same credentials work fine under bare-Python `celery` invocations (because you were probably passing them as kwargs, not a URL).
 
 ### First-time bring-up
 
 ```bash
+# Generate the root .env from the template (or let Ansible do it)
+cp .env.example .env && $EDITOR .env
+
 # Pull the image (or build locally with `docker compose build`)
 docker compose pull
 
@@ -199,17 +222,35 @@ docker compose restart mcp         # restart just the MCP server
 docker compose pull && docker compose up -d
 ```
 
-### Things to verify in `mnemosyne/.env` before bringing up
+### Things to verify in `.env` before bringing up
 
-The development `.env` has a few values that need adjusting for production:
+The root `.env` (the one compose interpolates from — not `mnemosyne/.env`) needs the following set for a working production deploy:
 
 - `DEBUG=False`
-- `USE_LOCAL_STORAGE=False` (already set; just confirm)
+- `USE_LOCAL_STORAGE=False`
 - `KVDB_LOCATION=<external-memcached-host>:11211` — `127.0.0.1` does not resolve from inside containers
-- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` filled in
+- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` filled in (Mnemosyne's own MinIO bucket)
-- `DAEDALUS_S3_*` filled in for cross-bucket reads from the Daedalus bucket
+- `DAEDALUS_S3_ACCESS_KEY_ID` / `DAEDALUS_S3_SECRET_ACCESS_KEY` filled in for cross-bucket ingest reads
+- `CELERY_BROKER_URL` with the RabbitMQ password **percent-encoded** if it contains URL-special characters
 - `ALLOWED_HOSTS` includes the public hostname HAProxy routes to (e.g. `mnemosyne.ouranos.helu.ca`)
-- `LLM_API_SECRETS_ENCRYPTION_KEY` set to a real Fernet key
+- `CSRF_TRUSTED_ORIGINS` includes `https://<same-hostname>`
+- `LLM_API_SECRETS_ENCRYPTION_KEY` set to a real Fernet key (generated once per environment)
+
+### Verifying the environment reached a container
+
+If a service misbehaves on startup — typically the worker with an `AccessRefused` from RabbitMQ, or the app with a DB auth error — the fastest diagnostic is to print what Django actually parsed, since that removes every layer of env-file / interpolation / URL-encoding ambiguity:
+
+```bash
+# What broker URL did the worker actually receive?
+docker compose run --rm --no-deps worker \
+    python -c "from django.conf import settings; print(repr(settings.CELERY_BROKER_URL))"
+
+# What DB host/user?
+docker compose run --rm --no-deps app \
+    python -c "from django.conf import settings; print(settings.DATABASES['default'])"
+```
+
+The `repr(...)` form surfaces CRLF, trailing whitespace, stray quotes, or characters that should have been percent-encoded.
 
 ### Health probes
 

docker-compose.yaml

@@ -12,6 +12,16 @@
 # and reranker on Nyx, smtp4dev on Oberon. All reached over the internal
 # 10.10.0.0/24 network.
 #
+# Environment scoping
+# -------------------
+# Every service lists ONLY the environment variables it actually needs, with
+# values interpolated from the shell (typically `.env` at the project root,
+# which an Ansible role generates from a j2 template + vault secrets). No
+# `env_file:` sharing — a compromised MCP container should not see the Celery
+# broker creds or the LLM API encryption key, and the Celery worker has no
+# business knowing `ALLOWED_HOSTS`. If you add a new Django setting, decide
+# which services need it and add it only to those `environment:` blocks.
+#
 # Run:
 #   docker compose up -d
 #   docker compose run --rm app migrate    # one-shot DB migrate
@@ -32,11 +42,63 @@ services:
     restart: "no"
 
   # ── App: Django REST API + admin ──────────────────────────────────────────
+  # Serves /library/api/*, /admin/, /live/, /ready/, /metrics. Enqueues
+  # Celery tasks (hence CELERY_BROKER_URL is required here too — Django is
+  # the producer, the worker is the consumer).
   app:
     image: git.helu.ca/r/mnemosyne:latest
     pull_policy: always
     command: ["web"]
-    env_file: mnemosyne/.env
+    environment:
+      # Django core
+      - DJANGO_SETTINGS_MODULE=mnemosyne.settings
+      - SECRET_KEY=${SECRET_KEY}
+      - DEBUG=${DEBUG}
+      - ALLOWED_HOSTS=${ALLOWED_HOSTS}
+      - CSRF_TRUSTED_ORIGINS=${CSRF_TRUSTED_ORIGINS}
+      - TIME_ZONE=${TIME_ZONE}
+      - LANGUAGE_CODE=${LANGUAGE_CODE}
+      # Postgres (Django ORM)
+      - APP_DB_NAME=${APP_DB_NAME}
+      - APP_DB_USER=${APP_DB_USER}
+      - APP_DB_PASSWORD=${APP_DB_PASSWORD}
+      - DB_HOST=${DB_HOST}
+      - DB_PORT=${DB_PORT}
+      # Neo4j (knowledge graph + vectors)
+      - NEOMODEL_NEO4J_BOLT_URL=${NEOMODEL_NEO4J_BOLT_URL}
+      # Memcached (readiness probe, theme/notification cache)
+      - KVDB_LOCATION=${KVDB_LOCATION}
+      - KVDB_PREFIX=${KVDB_PREFIX}
+      # S3 (Django storage backend — chunk text, item files)
+      - USE_LOCAL_STORAGE=${USE_LOCAL_STORAGE}
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_STORAGE_BUCKET_NAME=${AWS_STORAGE_BUCKET_NAME}
+      - AWS_S3_ENDPOINT_URL=${AWS_S3_ENDPOINT_URL}
+      - AWS_S3_USE_SSL=${AWS_S3_USE_SSL}
+      - AWS_S3_VERIFY=${AWS_S3_VERIFY}
+      - AWS_S3_REGION_NAME=${AWS_S3_REGION_NAME}
+      # Celery (Django enqueues tasks; does NOT consume)
+      - CELERY_BROKER_URL=${CELERY_BROKER_URL}
+      - CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
+      - CELERY_TASK_ALWAYS_EAGER=${CELERY_TASK_ALWAYS_EAGER}
+      # LLM API secrets (admin + DRF pages decrypt stored provider API keys)
+      - LLM_API_SECRETS_ENCRYPTION_KEY=${LLM_API_SECRETS_ENCRYPTION_KEY}
+      # Email
+      - EMAIL_HOST=${EMAIL_HOST}
+      - EMAIL_PORT=${EMAIL_PORT}
+      - EMAIL_USE_TLS=${EMAIL_USE_TLS}
+      # Search & re-ranker (serves /library/api/search)
+      - SEARCH_VECTOR_TOP_K=${SEARCH_VECTOR_TOP_K}
+      - SEARCH_FULLTEXT_TOP_K=${SEARCH_FULLTEXT_TOP_K}
+      - SEARCH_GRAPH_MAX_DEPTH=${SEARCH_GRAPH_MAX_DEPTH}
+      - SEARCH_RRF_K=${SEARCH_RRF_K}
+      - SEARCH_DEFAULT_LIMIT=${SEARCH_DEFAULT_LIMIT}
+      - RERANKER_MAX_CANDIDATES=${RERANKER_MAX_CANDIDATES}
+      - RERANKER_TIMEOUT=${RERANKER_TIMEOUT}
+      # Logging
+      - LOGGING_LEVEL=${LOGGING_LEVEL}
+      - DJANGO_LOGGING_LEVEL=${DJANGO_LOGGING_LEVEL}
     restart: unless-stopped
     depends_on:
       static-init:
@@ -51,11 +113,63 @@ services:
       start_period: 30s
 
   # ── MCP server: FastMCP Streamable HTTP at /mcp/ ───────────────────────────
+  # Read-only LLM-facing surface. Intentionally excluded:
+  #   CELERY_BROKER_URL         — MCP must not enqueue tasks
+  #   LLM_API_SECRETS_ENCRYPTION_KEY — MCP must not decrypt stored provider keys
+  #   DAEDALUS_S3_*             — MCP does not ingest
+  #   CSRF_TRUSTED_ORIGINS      — MCP does not accept browser forms
+  #   EMAIL_*                   — MCP does not send mail
+  #   EMBEDDING_* (batch/timeout) — MCP does not embed
+  # S3 vars ARE passed so STORAGES initialises identically to the app container
+  # (simpler to reason about than having mcp use FileSystemStorage while the
+  # rest of the stack uses S3). MCP is read-only at the application layer so
+  # the S3 key here only matters if someone exploits a write path in the
+  # future — keep the credential scoped to read-only in your secret manager.
   mcp:
     image: git.helu.ca/r/mnemosyne:latest
     pull_policy: always
     command: ["mcp"]
-    env_file: mnemosyne/.env
+    environment:
+      # Django core (ASGI still imports settings)
+      - DJANGO_SETTINGS_MODULE=mnemosyne.settings
+      - SECRET_KEY=${SECRET_KEY}
+      - DEBUG=${DEBUG}
+      - ALLOWED_HOSTS=${ALLOWED_HOSTS}
+      - TIME_ZONE=${TIME_ZONE}
+      - LANGUAGE_CODE=${LANGUAGE_CODE}
+      # Postgres (McpToken lookup lives in Django ORM)
+      - APP_DB_NAME=${APP_DB_NAME}
+      - APP_DB_USER=${APP_DB_USER}
+      - APP_DB_PASSWORD=${APP_DB_PASSWORD}
+      - DB_HOST=${DB_HOST}
+      - DB_PORT=${DB_PORT}
+      # Neo4j (search + get_chunk)
+      - NEOMODEL_NEO4J_BOLT_URL=${NEOMODEL_NEO4J_BOLT_URL}
+      # Memcached
+      - KVDB_LOCATION=${KVDB_LOCATION}
+      - KVDB_PREFIX=${KVDB_PREFIX}
+      # S3 (same block as app — STORAGES must initialise identically)
+      - USE_LOCAL_STORAGE=${USE_LOCAL_STORAGE}
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_STORAGE_BUCKET_NAME=${AWS_STORAGE_BUCKET_NAME}
+      - AWS_S3_ENDPOINT_URL=${AWS_S3_ENDPOINT_URL}
+      - AWS_S3_USE_SSL=${AWS_S3_USE_SSL}
+      - AWS_S3_VERIFY=${AWS_S3_VERIFY}
+      - AWS_S3_REGION_NAME=${AWS_S3_REGION_NAME}
+      # MCP-specific
+      - MCP_REQUIRE_AUTH=${MCP_REQUIRE_AUTH}
+      # Search & re-ranker (the `search` MCP tool uses these)
+      - SEARCH_VECTOR_TOP_K=${SEARCH_VECTOR_TOP_K}
+      - SEARCH_FULLTEXT_TOP_K=${SEARCH_FULLTEXT_TOP_K}
+      - SEARCH_GRAPH_MAX_DEPTH=${SEARCH_GRAPH_MAX_DEPTH}
+      - SEARCH_RRF_K=${SEARCH_RRF_K}
+      - SEARCH_DEFAULT_LIMIT=${SEARCH_DEFAULT_LIMIT}
+      - RERANKER_MAX_CANDIDATES=${RERANKER_MAX_CANDIDATES}
+      - RERANKER_TIMEOUT=${RERANKER_TIMEOUT}
+      # Logging
+      - LOGGING_LEVEL=${LOGGING_LEVEL}
+      - DJANGO_LOGGING_LEVEL=${DJANGO_LOGGING_LEVEL}
     restart: unless-stopped
     volumes:
       - mnemosyne-media:/app/media
@@ -67,11 +181,66 @@ services:
       start_period: 30s
 
   # ── Celery worker: embedding + ingest + batch queues ───────────────────────
+  # Consumer side of the queue. Needs the full S3 block (reads Daedalus's
+  # bucket, writes to Mnemosyne's), the LLM API encryption key (ingest calls
+  # vision models via stored provider keys), and both broker URL + result
+  # backend. Does NOT need HTTP-layer settings (ALLOWED_HOSTS, CSRF, MCP auth)
+  # or search tuning (the worker never serves queries).
   worker:
     image: git.helu.ca/r/mnemosyne:latest
     pull_policy: always
     command: ["worker"]
-    env_file: mnemosyne/.env
+    environment:
+      # Django core (Celery imports settings)
+      - DJANGO_SETTINGS_MODULE=mnemosyne.settings
+      - SECRET_KEY=${SECRET_KEY}
+      - DEBUG=${DEBUG}
+      - TIME_ZONE=${TIME_ZONE}
+      - LANGUAGE_CODE=${LANGUAGE_CODE}
+      # Postgres
+      - APP_DB_NAME=${APP_DB_NAME}
+      - APP_DB_USER=${APP_DB_USER}
+      - APP_DB_PASSWORD=${APP_DB_PASSWORD}
+      - DB_HOST=${DB_HOST}
+      - DB_PORT=${DB_PORT}
+      # Neo4j (graph writes during embed/ingest)
+      - NEOMODEL_NEO4J_BOLT_URL=${NEOMODEL_NEO4J_BOLT_URL}
+      # Memcached (task progress cache)
+      - KVDB_LOCATION=${KVDB_LOCATION}
+      - KVDB_PREFIX=${KVDB_PREFIX}
+      # S3 — Mnemosyne's own bucket (chunk text writes, item file storage)
+      - USE_LOCAL_STORAGE=${USE_LOCAL_STORAGE}
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_STORAGE_BUCKET_NAME=${AWS_STORAGE_BUCKET_NAME}
+      - AWS_S3_ENDPOINT_URL=${AWS_S3_ENDPOINT_URL}
+      - AWS_S3_USE_SSL=${AWS_S3_USE_SSL}
+      - AWS_S3_VERIFY=${AWS_S3_VERIFY}
+      - AWS_S3_REGION_NAME=${AWS_S3_REGION_NAME}
+      # Daedalus S3 — cross-bucket reads for ingest (worker-only)
+      - DAEDALUS_S3_ENDPOINT_URL=${DAEDALUS_S3_ENDPOINT_URL}
+      - DAEDALUS_S3_ACCESS_KEY_ID=${DAEDALUS_S3_ACCESS_KEY_ID}
+      - DAEDALUS_S3_SECRET_ACCESS_KEY=${DAEDALUS_S3_SECRET_ACCESS_KEY}
+      - DAEDALUS_S3_BUCKET_NAME=${DAEDALUS_S3_BUCKET_NAME}
+      - DAEDALUS_S3_REGION_NAME=${DAEDALUS_S3_REGION_NAME}
+      - DAEDALUS_S3_USE_SSL=${DAEDALUS_S3_USE_SSL}
+      - DAEDALUS_S3_VERIFY=${DAEDALUS_S3_VERIFY}
+      # Celery / RabbitMQ
+      - CELERY_BROKER_URL=${CELERY_BROKER_URL}
+      - CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
+      - CELERY_TASK_ALWAYS_EAGER=${CELERY_TASK_ALWAYS_EAGER}
+      # Worker tuning (entrypoint.sh reads these)
+      - CELERY_QUEUES=${CELERY_QUEUES}
+      - CELERY_CONCURRENCY=${CELERY_CONCURRENCY}
+      - CELERY_LOG_LEVEL=${CELERY_LOGGING_LEVEL}
+      # LLM API secrets (ingest vision pass decrypts stored provider keys)
+      - LLM_API_SECRETS_ENCRYPTION_KEY=${LLM_API_SECRETS_ENCRYPTION_KEY}
+      # Embedding pipeline
+      - EMBEDDING_BATCH_SIZE=${EMBEDDING_BATCH_SIZE}
+      - EMBEDDING_TIMEOUT=${EMBEDDING_TIMEOUT}
+      # Logging
+      - LOGGING_LEVEL=${LOGGING_LEVEL}
+      - CELERY_LOGGING_LEVEL=${CELERY_LOGGING_LEVEL}
     restart: unless-stopped
     depends_on:
       app:
@@ -85,7 +254,9 @@ services:
       retries: 3
       start_period: 60s
 
-  # ── Web: nginx reverse proxy, public port 23090 ───────────────────────────
+  # ── Web: nginx reverse proxy, public port 23181 ────────────────────────────
+  # No Django env — nginx only knows how to route. Public listener is
+  # templated into the conf file by Ansible if the port ever needs to change.
   web:
     image: nginx:alpine
     restart: unless-stopped
@@ -106,8 +277,9 @@ services:
       retries: 3
 
 volumes:
-  # Static files baked into the image at /app/staticfiles. The app service
+  # Static files baked into the image at /app/staticfiles. The static-init
-  # mounts this volume, populating it on first start; nginx reads from it.
+  # service seeds this volume on every `up`, so nginx always serves the
+  # current image's static bundle.
   mnemosyne-static:
   # Local FileSystemStorage fallback. Production uses USE_LOCAL_STORAGE=False
   # so this is mostly empty — kept for parity with dev and for any path

docs/ouranos.md

@@ -31,16 +31,41 @@ All containers are named after moons of Uranus and resolved via the `.incus` DNS
 | **rosalind** | collaboration | Gitea, LobeChat, Nextcloud, AnythingLLM | ✔ |
 | **sycorax** | language_models | Arke LLM Proxy | ✔ |
 | **titania** | proxy_sso | HAProxy TLS termination + Casdoor SSO | ✔ |
+| **umbriel** | graph_database | Neo4j (Mnemosyne) — dedicated memory graph | ✔ |
 
-### oberon — Container Orchestration
+### puck — Project Application Runtime
+
+Shape-shifting trickster embodying Python's versatility.
+This is the host that runs Python projects in the Ouranos sandbox.
+It has an RDP server and is generally where application development happens.
+Each project has a number that is used to determine port numbers.
+
+- Docker engine
+- JupyterLab (port 22071 via OAuth2-Proxy)
+- Gitea Runner (CI/CD agent)
+- Django Projects: Zelus (221), Angelia (222), Athena (224), Kairos (225), Icarlos (226), MCP Switchboard (227), Spelunker (228), Peitho (229), Mnemosyne (230)
+- FastAgent Projects: Pallas (240)
+- FastAPI Projects: Daedalus (200), Arke (201) Kernos (202), Rommie (203), Orpheus (204), Periplus (205), Nike (206), Stentor (207)
+
+### caliban — Agent Automation
+
+Autonomous computer agent learning through environmental interaction.
+
+- Docker engine
+- Agent S MCP Server (MATE desktop, AT-SPI automation)
+- Kernos MCP Shell Server (port 22062)
+- Rommie MCP Server (port 22061) — agent-to-agent GUI automation via Agent S
+- FreeCAD Robust MCP Server (port 22063) — CAD automation via FreeCAD XML-RPC
+- GPU passthrough
+- RDP access (port 25521)
+
+### oberon — Container Orchestration & Dockerized Shared Services
 
 King of the Fairies orchestrating containers and managing MCP infrastructure.
 
 - Docker engine
-- MCP Switchboard (port 22785) — Django app routing MCP tool calls
+- MCP Switchboard (port 22781) — Django app routing MCP tool calls
 - RabbitMQ message queue
-- Open WebUI LLM interface (port 22088, PostgreSQL backend on Portia)
-- SearXNG privacy search (port 22083, behind OAuth2-Proxy)
 - smtp4dev SMTP test server (port 22025)
 
 ### portia — Relational Database
@@ -48,25 +73,38 @@ King of the Fairies orchestrating containers and managing MCP infrastructure.
 Intelligent and resourceful — the reliability of relational databases.
 
 - PostgreSQL 17 (port 5432)
-- Databases: `arke`, `anythingllm`, `gitea`, `hass`, `lobechat`, `mcp_switchboard`, `nextcloud`, `openwebui`, `periplus`, `spelunker`
+- Databases: `arke`, `anythingllm`, `gitea`, `hass`, `lobechat`, `mcp_switchboard`, `mnemosyne`, `nextcloud`, `openwebui`, `periplus`, `spelunker`
 
 ### ariel — Graph Database
 
 Air spirit — ethereal, interconnected nature mirroring graph relationships.
 
 - Neo4j 5.26.0 (Docker)
-- HTTP API: port 25584
+- HTTP API: port 25554
-- Bolt: port 25554
+- Bolt: port 7687 (reached as `ariel.incus:7687` on the internal network)
 
-### puck — Application Runtime
+### umbriel — Graph Database (Mnemosyne)
 
-Shape-shifting trickster embodying Python's versatility.
+Dusky melancholy sprite from Pope's *Rape of the Lock* — keeper of the Cave of
+Spleen, naturally paired with Mnemosyne the Titan of memory. Dedicated Neo4j
+instance so Mnemosyne's `Library`/`Collection`/`Item`/`Chunk`/`Concept` labels,
+vector indexes, and schema migrations can't collide with another tenant's
+graph on Ariel.
 
-- Docker engine
+- Neo4j 5.26.0 (Docker)
-- JupyterLab (port 22071 via OAuth2-Proxy)
+- HTTP Browser: port 25555
-- Gitea Runner (CI/CD agent)
+- Bolt: port 7687 (reached as `umbriel.incus:7687` on the internal network)
-- Home Assistant (port 8123)
+
-- Django applications: Angelia (22281), Athena (22481), Kairos (22581), Icarlos (22681), Spelunker (22881), Peitho (22981)
+### miranda — MCP Docker Host
+
+Curious bridge between worlds — hosting MCP server containers.
+
+- Docker engine (API exposed on port 2375 for MCP Switchboard)
+- MCPO OpenAI-compatible MCP proxy 22071
+- Argos MCP Server — web search via SearXNG (port 22062)
+- Grafana MCP Server (port 22063)
+- Neo4j MCP Server (port 22064)
+- Gitea MCP Server (port 22065)
 
 ### prospero — Observability Stack
 
@@ -79,16 +117,18 @@ Master magician observing all events.
 - Loki log aggregation via Alloy (all hosts)
 - Grafana dashboard suite with Casdoor SSO integration
 
-### miranda — MCP Docker Host
+### rosalind — Third Party Applications for testing and evaluation
 
-Curious bridge between worlds — hosting MCP server containers.
+Witty and resourceful moon for PHP, Go, and Node.js runtimes.
 
-- Docker engine (API exposed on port 2375 for MCP Switchboard)
+- SearXNG privacy search (port 22083, behind OAuth2-Proxy)
-- MCPO OpenAI-compatible MCP proxy
+- Gitea self-hosted Git (port 22082, SSH on 22022)
-- Grafana MCP Server (port 25533)
+- LobeChat AI chat interface (port 22081)
-- Gitea MCP Server (port 25535)
+- Nextcloud file sharing and collaboration (port 22083)
-- Neo4j MCP Server
+- AnythingLLM document AI workspace (port 22084)
-- Argos MCP Server — web search via SearXNG (port 25534)
+- Nextcloud data on dedicated Incus storage volume
+- Open WebUI LLM interface (port 22088, PostgreSQL backend on Portia
+- Home Assistant (port 8123)
 
 ### sycorax — Language Models
 
@@ -99,26 +139,6 @@ Original magical power wielding language magic.
 - Session management with Memcached
 - Database backend on Portia
 
-### caliban — Agent Automation
-
-Autonomous computer agent learning through environmental interaction.
-
-- Docker engine
-- Agent S MCP Server (MATE desktop, AT-SPI automation)
-- Kernos MCP Shell Server (port 22021)
-- GPU passthrough for vision tasks
-- RDP access (port 25521)
-
-### rosalind — Collaboration Services
-
-Witty and resourceful moon for PHP, Go, and Node.js runtimes.
-
-- Gitea self-hosted Git (port 22082, SSH on 22022)
-- LobeChat AI chat interface (port 22081)
-- Nextcloud file sharing and collaboration (port 22083)
-- AnythingLLM document AI workspace (port 22084)
-- Nextcloud data on dedicated Incus storage volume
-
 ### titania — Proxy & SSO Services
 
 Queen of the Fairies managing access control and authentication.
@@ -132,6 +152,100 @@ Queen of the Fairies managing access control and authentication.
 
 ---
 
+## Port Numbering
+
+Well-known ports running as a service may be used: Postgresql 5432, Prometheus Metrics 9100.
+
+However inside a docker project, the number plan needs to be followed to avoid port conflicts and confusion:
+XXXYZ
+XXX Project Number or 220 for external project 
+Y Service: 0 reserved, 1-4 flexible, 5 database, 6 MCP, 7 API, 8 Web App, 9 Prometheus metrics
+Z Instance: The running instance of this app on the same host, starting at 1.  May also be used to handle exceptions.
+
+255 Incus port forwarding: Ports in ths range are forwarded from the Incus host to Incus containers (defined in Terraform)
+
+514ZZ is the syslog port.  Docker containers send their syslog to an Alloy syslog collector port.  ZZ is the application instance, they just need to be different on the same host and increment from 01.
+
+---
+
+## Application Conventions
+
+Standards that all services deployed in Ouranos MUST follow. For full logging standards and anti-patterns, see [red_panda_standards.md](red_panda_standards.md).
+
+### Health Check Endpoints
+
+All services MUST expose Kubernetes-style health endpoints:
+
+| Endpoint | Purpose | Auth |
+|----------|---------|------|
+| `GET /live` | **Liveness** — process is running and accepting connections | None |
+| `GET /ready` | **Readiness** — process is running AND all dependencies (DB, cache, upstream APIs) are healthy | None |
+| `GET /metrics` | Prometheus metrics (see below) | IP-restricted |
+
+- HAProxy checks `health_path` (typically `/ready/`) for backend health — return HTTP 200 when healthy
+- Health endpoints MUST NOT require authentication (no JWT, no session)
+- Third-party services use their native health paths (e.g., `/api/health`, `/api/healthz`, `/-/healthy`)
+
+### Health Checks in Docker Compose
+
+Use `curl -f` for Docker Compose healthchecks. Install curl in images if needed.
+
+```yaml
+healthcheck:
+  test: ["CMD", "curl", "-f", "http://localhost:8000/live"]
+  interval: 30s
+  timeout: 10s
+  retries: 3
+  start_period: 40s
+```
+
+### Logging Conventions
+
+Log output flows through: **App → syslog (RFC3164) → Alloy → Loki → Grafana**
+
+| Level | Usage |
+|-------|-------|
+| **ERROR** | Broken state requiring human action — always include `exc_info=True`, error type, and context |
+| **WARNING** | Degraded but recovering — client disconnects, performance outliers, client-side exceptions, leaked markup |
+| **INFO** | Lifecycle events — service start/stop, connections, requests completed, jobs finished |
+| **DEBUG** | Diagnostic detail — SSE events, keepalive pings, health check 200 responses, negotiation steps |
+
+**Health check responses MUST be logged at DEBUG only.** HAProxy and Prometheus probe endpoints every 15-30 seconds. Logging these at INFO floods syslog with thousands of identical `200 OK` lines per hour, burying real events.
+
+### Protected vs Unprotected Endpoints
+
+| Protected (require valid JWT) | Unprotected |
+|-------------------------------|-------------|
+| All `/api/v1/*` routes | `GET /live` |
+| | `GET /ready` |
+| | `GET /metrics` (IP-restricted to internal networks) |
+| | `GET /api/auth/login-url` |
+| | `POST /api/auth/token` |
+| | `POST /api/v1/telemetry` (sendBeacon cannot set headers) |
+
+### Prometheus Metrics
+
+All services SHOULD expose `GET /metrics` in Prometheus exposition format, scraped by Prospero's Prometheus (default 15s interval).
+
+- **IP-restricted** to internal networks only (`10.10.0.0/24`, `172.16.0.0/12`, `127.0.0.0/8`)
+- Consider exposing: request counts/durations, error rates, active connections, queue depths, dependency health
+
+### Browser Telemetry
+
+Frontend/browser code MUST send telemetry data and errors back to the application's telemetry API:
+
+- `POST /api/v1/telemetry` — unprotected (browser `sendBeacon` cannot set Authorization headers)
+- Capture and report: JavaScript exceptions, performance metrics, user-facing errors
+- Client-side exceptions should log as **WARNING** on the server (they indicate a problem but not a server-side failure)
+
+### Docker Networking
+
+- Use the **default Docker bridge network** for simple deployments
+- Add additional named networks only when required (e.g., isolating database traffic) or explicitly requested
+- Do not create custom network definitions for single-service Docker Compose stacks
+
+---
+
 ## External Access via HAProxy
 
 Titania provides TLS termination and reverse proxy for all services.
@@ -160,14 +274,13 @@ Titania provides TLS termination and reverse proxy for all services.
 | `kairos.ouranos.helu.ca` | puck.incus:22581 | Kairos (Django) |
 | `lobechat.ouranos.helu.ca` | rosalind.incus:22081 | LobeChat |
 | `loki.ouranos.helu.ca` | prospero.incus:443 (SSL) | Loki |
-| `mcp-switchboard.ouranos.helu.ca` | oberon.incus:22785 | MCP Switchboard |
+| `mcp-switchboard.ouranos.helu.ca` | oberon.incus:22781 | MCP Switchboard |
 | `nextcloud.ouranos.helu.ca` | rosalind.incus:22083 | Nextcloud |
 | `openwebui.ouranos.helu.ca` | oberon.incus:22088 | Open WebUI |
 | `peitho.ouranos.helu.ca` | puck.incus:22981 | Peitho (Django) |
+| `periplus.ouranos.helu.ca` | puck.incus:20681 | Periplus (FastAPI + MCP via nginx) |
 | `pgadmin.ouranos.helu.ca` | prospero.incus:443 (SSL) | PgAdmin 4 |
 | `prometheus.ouranos.helu.ca` | prospero.incus:443 (SSL) | Prometheus |
-| `freecad-mcp.ouranos.helu.ca` | caliban.incus:22032 | FreeCAD Robust MCP Server |
-| `rommie.ouranos.helu.ca` | caliban.incus:22031 | Rommie MCP Server (Agent S GUI automation) |
 | `searxng.ouranos.helu.ca` | oberon.incus:22073 | SearXNG (OAuth2-Proxy) |
 | `smtp4dev.ouranos.helu.ca` | oberon.incus:22085 | smtp4dev |
 | `spelunker.ouranos.helu.ca` | puck.incus:22881 | Spelunker (Django) |
@@ -187,7 +300,7 @@ terraform apply
 
 # Start all containers
 cd ../ansible
-source ~/env/agathos/bin/activate
+source ~/env/ouranos/bin/activate
 ansible-playbook sandbox_up.yml
 
 # Deploy all services
@@ -197,6 +310,35 @@ ansible-playbook site.yml
 ansible-playbook sandbox_down.yml
 ```
 
+### Python Virtual Environment Setup
+
+The Ansible automation requires a Python virtual environment with the `ansible` package installed. Create and activate the environment from the `~` directory:
+
+```bash
+# Create virtual environment
+cd ~
+python3 -m venv env/ouranos
+
+# Activate environment
+source ~/env/ouranos/bin/activate
+
+# Install Ansible
+pip install ansible
+pip install ansible-core
+pip install ansible-community.postgresql
+```
+
+### Ansible Playbook Syntax Check
+
+Before running playbooks, use the `apsc.sh` utility (in PATH) to quickly validate YAML syntax:
+
+```bash
+# From the ansible directory
+apsc.sh
+
+# This will check all YAML files in the current directory for syntax errors
+```
+
 ### Terraform Workflow
 
 1. **Define** — Containers, networks, and resources in `*.tf` files
@@ -204,6 +346,83 @@ ansible-playbook sandbox_down.yml
 3. **Apply** — Provision with `terraform apply`
 4. **Verify** — Check outputs and container status
 
+### Terraform Import
+
+When containers or other resources are created manually (outside Terraform) or need to be re-imported after recreation, use `terraform import` to sync the Terraform state with existing infrastructure.
+
+#### Import Syntax
+
+The correct import format for Incus resources requires quoting resource addresses with `for_each` keys and using the full ID including image fingerprints:
+
+```bash
+# Import a container with correct syntax
+terraform import 'incus_instance.uranian_hosts["<name>"]' ouranos/<name>,image=<fingerprint>
+```
+
+#### Getting Image Fingerprints
+
+First, get the fingerprint of the image resource from Terraform state:
+
+```bash
+cd terraform
+terraform state show incus_image.noble | grep fingerprint
+# Output: fingerprint = "75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644"
+
+terraform state show incus_image.questing | grep fingerprint
+# Output: fingerprint = "e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf"
+```
+
+#### Importing All Uranian Hosts
+
+Replace containers missing from state (or re-import after manual recreation):
+
+```bash
+# Containers using noble image
+terraform import 'incus_instance.uranian_hosts["ariel"]' ouranos/ariel,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["miranda"]' ouranos/miranda,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["oberon"]' ouranos/oberon,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["portia"]' ouranos/portia,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["prospero"]' ouranos/prospero,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["rosalind"]' ouranos/rosalind,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["sycorax"]' ouranos/sycorax,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["titania"]' ouranos/titania,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+terraform import 'incus_instance.uranian_hosts["umbriel"]' ouranos/umbriel,image=75cde3e755b0e657c05f67e03a42683217b233b0339448be747845747df58644
+
+# Containers using questing image
+terraform import 'incus_instance.uranian_hosts["caliban"]' ouranos/caliban,image=e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf
+terraform import 'incus_instance.uranian_hosts["puck"]' ouranos/puck,image=e78dd4a406b7fa3592ed0a6048862260b3d2e50c76e32a6169930245c0a13fdf
+```
+
+#### Storage Bucket Import
+
+For storage buckets, use the `<project>/<pool>/<name>` format:
+
+```bash
+terraform import incus_storage_bucket.<name> ouranos/default/<bucket-name>
+```
+
+#### Common Issues
+
+1. **Import ID format errors**: Use quotes around resource addresses with `for_each` keys: `'incus_instance.uranian_hosts["name"]'`
+
+2. **Image replacement on import**: Importing without specifying the image fingerprint will cause Terraform to replace the container on next apply. Always include `image=<fingerprint>` in the import ID.
+
+3. **Tainted state**: If a resource shows "will be created" but already exists, it may be tainted. Remove from state and re-import:
+   ```bash
+   terraform state rm 'incus_instance.uranian_hosts["name"]'
+   terraform import 'incus_instance.uranian_hosts["name"]' ouranos/name,image=<fingerprint>
+   ```
+
+#### Verify Import
+
+After importing, verify with `terraform plan`:
+
+```bash
+terraform plan
+# Should show: Plan: 0 to add, 0 to change, 0 to destroy
+# (Minor "update in-place" changes are normal for state sync of computed attributes)
+```
+
 ### Ansible Workflow
 
 1. **Bootstrap** — Update packages, install essentials (`apt_update.yml`)
@@ -255,7 +474,7 @@ Playbooks run in dependency order:
 | `pplg/deploy.yml` | Prospero | Full observability stack + HAProxy + OAuth2-Proxy |
 | `postgresql/deploy.yml` | Portia | PostgreSQL with all databases |
 | `postgresql_ssl/deploy.yml` | Titania | Dedicated PostgreSQL for Casdoor |
-| `neo4j/deploy.yml` | Ariel | Neo4j graph database |
+| `neo4j/deploy.yml` | Ariel, Umbriel | Neo4j graph database (Umbriel is the dedicated Mnemosyne instance) |
 | `searxng/deploy.yml` | Oberon | SearXNG privacy search |
 | `haproxy/deploy.yml` | Titania | HAProxy TLS termination and routing |
 | `casdoor/deploy.yml` | Titania | Casdoor SSO |
@@ -282,7 +501,9 @@ Services with standalone deploy playbooks (not in `site.yml`):
 | `jupyterlab/deploy.yml` | Puck | JupyterLab + OAuth2-Proxy |
 | `kernos/deploy.yml` | Caliban | Kernos MCP shell server |
 | `lobechat/deploy.yml` | Rosalind | LobeChat AI chat |
+| `rommie/deploy.yml` | Caliban | Rommie MCP server (Agent S GUI automation) |
 | `neo4j_mcp/deploy.yml` | Miranda | Neo4j MCP Server |
+| `freecad_mcp/deploy.yml` | Caliban | FreeCAD Robust MCP Server |
 | `rabbitmq/deploy.yml` | Oberon | RabbitMQ message queue |
 
 ### Lifecycle Playbooks
@@ -313,6 +534,7 @@ collect metrics & logs         storage & visualisation           notifications
 | All LLM apps | Arke (Sycorax) | `http://sycorax.incus:25540` |
 | Open WebUI, Arke, Gitea, Nextcloud, LobeChat | PostgreSQL (Portia) | `portia.incus:5432` |
 | Neo4j MCP | Neo4j (Ariel) | `ariel.incus:7687` (Bolt) |
+| Mnemosyne | Neo4j (Umbriel) | `umbriel.incus:7687` (Bolt) — dedicated tenant |
 | MCP Switchboard | Docker API (Miranda) | `tcp://miranda.incus:2375` |
 | MCP Switchboard | RabbitMQ (Oberon) | `oberon.incus:5672` |
 | Kairos, Spelunker | RabbitMQ (Oberon) | `oberon.incus:5672` |