diff --git a/docs/Red Panda Standards_Django_V1-02.md b/docs/Red Panda Standards_Django_V1-02.md new file mode 100644 index 0000000..35eacd8 --- /dev/null +++ b/docs/Red Panda Standards_Django_V1-02.md @@ -0,0 +1,614 @@ +## 🐾 Red Panda Approvalβ„’ + +This project follows Red Panda Approval standards β€” our gold standard for Django application quality. Code must be elegant, reliable, and maintainable to earn the approval of our adorable red panda judges. + +### The 5 Sacred Django Criteria +1. **Fresh Migration Test** β€” Clean migrations from empty database +2. **Elegant Simplicity** β€” No unnecessary complexity +3. **Observable & Debuggable** β€” Proper logging and error handling +4. **Consistent Patterns** β€” Follow Django conventions +5. **Actually Works** β€” Passes all checks and serves real user needs + +### Changelog +- **V1-02** β€” Added Docker Compose environment-scoping standard (per-service `environment:` blocks), nginx reverse-proxy reference config (Docker DNS resolver, `X-Forwarded-Proto` preservation, access-log filtering, internal-network allowlists), and Memcached deployment note (bind to `0.0.0.0`, not `localhost`). +- **V1-01** β€” Initial published standards. + +## Environment Standards +- Virtual environment: ~/env/PROJECT/bin/activate +- Use pyproject.toml for project configuration (no setup.py, no requirements.txt) +- Python version: specified in pyproject.toml +- Dependencies: floor-pinned with ceiling (e.g. `Django>=5.2,<6.0`) + +### Dependency Pinning + +```toml +# Correct β€” floor pin with ceiling +dependencies = [ + "Django>=5.2,<6.0", + "djangorestframework>=3.14,<4.0", + "cryptography>=41.0,<45.0", +] + +# Wrong β€” exact pins in library packages +dependencies = [ + "Django==5.2.7", # too strict, breaks downstream +] +``` + +Exact pins (`==`) are only appropriate in application-level lock files, not in reusable library packages. + +## Directory Structure +myproject/ # Git repository root +β”œβ”€β”€ .gitignore +β”œβ”€β”€ README.md +β”œβ”€β”€ pyproject.toml # Project configuration (moved to repo root) +β”œβ”€β”€ docker-compose.yaml # Per-service environment scoping (see below) +β”œβ”€β”€ .env # Docker Compose interpolation source β€” NOT committed +β”œβ”€β”€ .env.example # Template listing every `${VAR}` with which service consumes it +β”‚ +β”œβ”€β”€ project/ # Django project root (manage.py lives here) +β”‚ β”œβ”€β”€ manage.py +β”‚ β”œβ”€β”€ Dockerfile +β”‚ β”œβ”€β”€ .env # Local bare-Python dev environment (runserver, celery, etc.) +β”‚ β”‚ # Only read by bare-Python runs; NOT by the compose stack +β”‚ β”œβ”€β”€ .env.example +β”‚ β”‚ +β”‚ β”œβ”€β”€ config/ # Django configuration module +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ settings.py +β”‚ β”‚ β”œβ”€β”€ urls.py +β”‚ β”‚ β”œβ”€β”€ wsgi.py +β”‚ β”‚ └── asgi.py +β”‚ β”‚ +β”‚ β”œβ”€β”€ accounts/ # Django app +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ models.py +β”‚ β”‚ β”œβ”€β”€ views.py +β”‚ β”‚ └── urls.py +β”‚ β”‚ +β”‚ β”œβ”€β”€ blog/ # Django app +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ models.py +β”‚ β”‚ β”œβ”€β”€ views.py +β”‚ β”‚ └── urls.py +β”‚ β”‚ +β”‚ β”œβ”€β”€ static/ +β”‚ β”‚ β”œβ”€β”€ css/ +β”‚ β”‚ └── js/ +β”‚ β”‚ +β”‚ └── templates/ +β”‚ └── base.html +β”‚ +β”œβ”€β”€ nginx/ # Nginx configuration (see Nginx Reverse Proxy below) +β”‚ └── PROJECT.conf +β”‚ +β”œβ”€β”€ db/ # PostgreSQL configuration (if customised) +β”‚ └── postgresql.conf +β”‚ +└── docs/ # Project documentation + └── index.md + +## Settings Structure +- Use a single settings.py file +- Use django-environ or python-dotenv for environment variables +- Never commit .env files to version control +- Provide .env.example with all required variables documented +- Create .gitignore file +- Create a .dockerignore file + +## Environment Variables + +### PostgreSQL settings (only if DB_ENGINE=postgresql) +``` +APP_DB_NAME=angelia2 +APP_DB_USER=angelia +APP_DB_PASSWORD=changeme +DB_HOST=db +DB_PORT=5432 +``` + +## Docker Compose β€” Per-Service Environment Scoping + +> **New in V1-02.** The monolithic `env_file:` pattern is deprecated. + +### Rule: every service declares only the env vars it actually needs + +In `docker-compose.yaml`, each service uses an `environment:` block listing only the variables that service consumes, with values interpolated from `.env` (at the repo root) using `${VAR}` syntax. Do **not** use `env_file: .env` shared across services. + +```yaml +services: + app: + image: git.helu.ca/r/myproject:latest + command: ["web"] + environment: + # Django core + - DJANGO_SETTINGS_MODULE=myproject.settings + - SECRET_KEY=${SECRET_KEY} + - DEBUG=${DEBUG} + - ALLOWED_HOSTS=${ALLOWED_HOSTS} + - CSRF_TRUSTED_ORIGINS=${CSRF_TRUSTED_ORIGINS} + # Postgres + - APP_DB_NAME=${APP_DB_NAME} + - APP_DB_USER=${APP_DB_USER} + - APP_DB_PASSWORD=${APP_DB_PASSWORD} + - DB_HOST=${DB_HOST} + - DB_PORT=${DB_PORT} + # ... + + worker: + image: git.helu.ca/r/myproject:latest + command: ["worker"] + environment: + - DJANGO_SETTINGS_MODULE=myproject.settings + - SECRET_KEY=${SECRET_KEY} + - APP_DB_NAME=${APP_DB_NAME} + # ... + # NO ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS, EMAIL_* β€” worker doesn't serve HTTP +``` + +### Why this matters + +1. **Least privilege / blast radius.** A compromised MCP container shouldn't see Celery broker credentials or encryption keys. A Celery worker shouldn't see `ALLOWED_HOSTS` or CSRF config β€” it doesn't serve HTTP. When every service shares one env file, a misconfigured secret takes down the whole stack instead of just the services that need that secret. + +2. **Self-documenting surface.** Reading `docker-compose.yaml` immediately tells you what each container depends on. With `env_file:`, every container has access to every secret and you can't tell from the compose file which service uses which variable. + +3. **Ansible rendering.** The compose file can be converted to a Jinja2 template and rendered per-host by an Ansible role, with secrets pulled from the vault. The `${VAR}` pattern is the natural interface between compose and Ansible. + +4. **Parsing correctness.** Docker Compose's `env_file:` parser does **not** strip inline `# comments`, honours CRLF `\r` as part of values, and handles quoting differently than `python-decouple`/`django-environ`. An `.env` that works with bare-Python `manage.py runserver` can silently feed a mangled URL (e.g. `CELERY_BROKER_URL` with a trailing `\r` or stray comment) to a container. Shell-style `${VAR}` interpolation avoids this because the value is unescaped by the shell the same way every time. + +### `.env.example` template convention + +Annotate each variable with which service(s) consume it: + +```bash +# --- Django core ------------------------------------------------------------ +# Consumed by: app, mcp, worker +SECRET_KEY=change-me-to-a-real-secret-key +DEBUG=False + +# --- PostgreSQL ------------------------------------------------------------ +# Consumed by: app, mcp, worker +APP_DB_NAME=myproject +APP_DB_USER=myproject +APP_DB_PASSWORD=change-me + +# --- Celery / RabbitMQ ----------------------------------------------------- +# Consumed by: app (producer), worker (consumer). NOT mcp. +# Percent-encode any password chars with URL meaning: @ : / # % + ? & = and space +CELERY_BROKER_URL=amqp://myproject:change-me@oberon.incus:5672/myproject +``` + +### Diagnostic: "what did Django actually parse?" + +When a service misbehaves on startup (broker 403, DB auth error, unreachable cache), the fastest diagnostic is to print what settings.py actually resolved to β€” that removes every layer of env-file / interpolation / URL-encoding ambiguity: + +```bash +docker compose run --rm --no-deps worker \ + python -c "from django.conf import settings; print(repr(settings.CELERY_BROKER_URL))" + +docker compose run --rm --no-deps app \ + python -c "from django.conf import settings; print(settings.DATABASES['default'])" +``` + +The `repr(...)` form surfaces CRLF, trailing whitespace, stray quotes, and characters that should have been percent-encoded but weren't. + +### Broker URL gotcha (documented for every new project) + +RabbitMQ connection URLs must percent-encode any password character with URL meaning (`@ : / # % + ? & =` and space). Kombu's URL parser is strict β€” an unencoded `#` in the password is read as the start of a URL fragment, and an unencoded `@` shifts the username/host boundary, both causing `ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN` at worker startup. Bare-Python tests that pass the password as a kwarg rather than a URL won't exhibit this and can mask the bug. + +## Nginx Reverse Proxy + +> **New in V1-02.** Standard reference config for any Red Panda project running behind HAProxy on Titania. + +Deployed as a sidecar container in the compose stack, fronting the Django app (gunicorn) and β€” where applicable β€” an MCP or streaming service. HAProxy handles TLS termination; nginx is plain HTTP on the internal network. + +### Required building blocks + +1. **Docker DNS resolver + variable-based `proxy_pass`.** `upstream` blocks resolve container hostnames **once at startup** and cache the IP forever. When `docker compose restart app` assigns a new IP, nginx returns 502 until fully reloaded. Use: + + ```nginx + resolver 127.0.0.11 valid=10s; + server { + set $backend_app http://app:8000; + location / { + proxy_pass $backend_app; # variable β†’ re-resolve each request + } + } + ``` + +2. **`$proxy_x_forwarded_proto` map.** Inside the compose network `$scheme` is always `http` because HAProxy already terminated TLS. Passing `$scheme` to Django breaks `request.is_secure()`, secure cookies, and `build_absolute_uri()`. Preserve the HAProxy header: + + ```nginx + map $http_x_forwarded_proto $proxy_x_forwarded_proto { + default $http_x_forwarded_proto; + "" $scheme; + } + # Then in every proxy block: + proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto; + ``` + +3. **Access-log suppression for probe paths.** HAProxy and Prometheus probe every 15–30 s; logging them floods Loki. The `nginx:alpine` image has a default http-level `access_log`, so a server-level `access_log` is *additive*, not replacing. You need both: + + ```nginx + map $request_uri $loggable { + default 1; + ~^/live(/|\?|$) 0; + ~^/ready(/|\?|$) 0; + ~^/metrics(/|\?|$) 0; + ~^/healthz(/|\?|$) 0; + ~^/nginx_status(/|\?|$) 0; + ~^/mcp/health(/|\?|$) 0; + } + + access_log off; # defeat inherited default + access_log /dev/stdout combined if=$loggable; # then install filtered version + ``` + +4. **Internal-network allowlist for all probe + metric endpoints.** RFC1918 + loopback, applied to `/live/`, `/ready/`, `/healthz`, `/metrics`, and `/nginx_status`: + + ```nginx + location = /metrics { + allow 127.0.0.0/8; # loopback + allow 10.0.0.0/8; # RFC1918 β€” primary internal (Incus, HAProxy) + allow 172.16.0.0/12; # RFC1918 β€” Docker bridge networks + allow 192.168.0.0/16; # RFC1918 + deny all; + proxy_pass $backend_app; + # ... + } + ``` + + All four RFC1918 ranges must be present β€” omitting `172.16.0.0/12` silently breaks scrapes from a Prometheus container on the default Docker bridge. + +5. **Security headers on the catch-all**, marked `always` so they apply to upstream 4xx/5xx: + + ```nginx + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-XSS-Protection "1; mode=block" always; + ``` + + Stronger policies (CSP, Referrer-Policy, HSTS) are set at HAProxy so they're consistent across every backend. + +6. **Catch-all proxies to Django.** nginx should intercept only the paths that need special handling (`/static/`, `/media/`, `/mcp/`, `/healthz`, `/metrics`, `/nginx_status`, the probes). Everything else flows through to Django, which returns its own themed 404 for unrouted paths β€” not nginx's bare default page. + +### Minimal reference config + +```nginx +resolver 127.0.0.11 valid=10s; + +map $http_x_forwarded_proto $proxy_x_forwarded_proto { + default $http_x_forwarded_proto; + "" $scheme; +} + +map $request_uri $loggable { + default 1; + ~^/live(/|\?|$) 0; + ~^/ready(/|\?|$) 0; + ~^/metrics(/|\?|$) 0; + ~^/healthz(/|\?|$) 0; + ~^/nginx_status(/|\?|$) 0; +} + +access_log off; +access_log /dev/stdout combined if=$loggable; + +server { + listen 80 default_server; + server_name _; + + client_max_body_size 64m; + + set $backend_app http://app:8000; + + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-XSS-Protection "1; mode=block" always; + + location /static/ { + alias /var/www/static/; + access_log off; + expires 30d; + add_header Cache-Control "public, immutable"; + } + + location /media/ { + alias /var/www/media/; + access_log off; + expires 7d; + } + + # Internal-only endpoints β€” allowlist applied uniformly + location = /live/ { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; } + location = /ready/ { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; } + location = /metrics { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; } + location = /nginx_status { + include /etc/nginx/snippets/internal-only.conf; + stub_status on; + access_log off; + } + + location / { + proxy_pass $backend_app; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto; + proxy_redirect off; + proxy_read_timeout 300s; + } +} +``` + +> Projects may inline the `proxy_set_header` block rather than using snippets; both are acceptable. The important thing is that every `proxy_pass` has the same four headers (`Host`, `X-Real-IP`, `X-Forwarded-For`, `X-Forwarded-Proto` using `$proxy_x_forwarded_proto`). + +## Memcached + +> **New in V1-02.** Memcached is a standard Red Panda dependency. Every Django service uses it for session storage, task-progress tracking, and cheap key-value caching. + +- Package: `pymemcache` (via `django.core.cache.backends.memcached.PyMemcacheCache`) +- Key prefix: per-app, configured via env var (e.g. `KVDB_PREFIX=mnemosyne`) +- Cache-key pattern: `{app}:{model}:{identifier}:{field}` + +### Deployment convention + +Memcached runs as a service on the application host (in Ouranos: a package install per Incus container). Configure it to bind to **all interfaces**, not just `localhost`, so: + +- Containers on the same host can reach it via the host's LAN name (e.g. `puck.incus:11211`). +- Other hosts in the lab can reach it for multi-host debugging. + +```bash +# /etc/memcached.conf on the Docker host +-l 0.0.0.0 +-p 11211 +-U 0 +``` + +The bare-Python "`localhost:11211` works" default is a dev-only convenience β€” it breaks as soon as Django moves into a container, because inside the container `127.0.0.1` is the container itself. The production `.env` must use the LAN-resolvable hostname: + +``` +KVDB_LOCATION=puck.incus:11211 +KVDB_PREFIX=myproject +``` + +### Health-check reachability + +The Django readiness probe (`GET /ready/`) must verify Memcached is reachable. If the probe returns 503 and the log shows no cause, hit the endpoint directly to read the JSON body which names the broken dependency: + +```bash +docker compose exec app curl -sS -o - -w "\nHTTP %{http_code}\n" http://localhost:8000/ready/ +``` + +## Code Organization +- Imports: PEP 8 ordering (stdlib, third-party, local) +- Type hints on function parameters +- CSS: External .css files only (no inline styles, no embedded `