mnemosyne/docs/Red Panda Standards_Django_V1-02.md

## 🐾 Red Panda Approval™

This project follows Red Panda Approval standards — our gold standard for Django application quality. Code must be elegant, reliable, and maintainable to earn the approval of our adorable red panda judges.

### The 5 Sacred Django Criteria
1. **Fresh Migration Test** — Clean migrations from empty database
2. **Elegant Simplicity** — No unnecessary complexity
3. **Observable & Debuggable** — Proper logging and error handling
4. **Consistent Patterns** — Follow Django conventions
5. **Actually Works** — Passes all checks and serves real user needs

### Changelog
- **V1-02** — Added Docker Compose environment-scoping standard (per-service `environment:` blocks), nginx reverse-proxy reference config (Docker DNS resolver, `X-Forwarded-Proto` preservation, access-log filtering, internal-network allowlists), and Memcached deployment note (bind to `0.0.0.0`, not `localhost`).
- **V1-01** — Initial published standards.

## Environment Standards
- Virtual environment: ~/env/PROJECT/bin/activate
- Use pyproject.toml for project configuration (no setup.py, no requirements.txt)
- Python version: specified in pyproject.toml
- Dependencies: floor-pinned with ceiling (e.g. `Django>=5.2,<6.0`)

### Dependency Pinning

```toml
# Correct — floor pin with ceiling
dependencies = [
    "Django>=5.2,<6.0",
    "djangorestframework>=3.14,<4.0",
    "cryptography>=41.0,<45.0",
]

# Wrong — exact pins in library packages
dependencies = [
    "Django==5.2.7",  # too strict, breaks downstream
]
```

Exact pins (`==`) are only appropriate in application-level lock files, not in reusable library packages.

## Directory Structure
myproject/                     # Git repository root
├── .gitignore
├── README.md
├── pyproject.toml             # Project configuration (moved to repo root)
├── docker-compose.yaml        # Per-service environment scoping (see below)
├── .env                       # Docker Compose interpolation source — NOT committed
├── .env.example               # Template listing every `${VAR}` with which service consumes it
│
├── project/                   # Django project root (manage.py lives here)
│   ├── manage.py
│   ├── Dockerfile
│   ├── .env                   # Local bare-Python dev environment (runserver, celery, etc.)
│   │                          #   Only read by bare-Python runs; NOT by the compose stack
│   ├── .env.example
│   │
│   ├── config/                # Django configuration module
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   ├── wsgi.py
│   │   └── asgi.py
│   │
│   ├── accounts/              # Django app
│   │   ├── __init__.py
│   │   ├── models.py
│   │   ├── views.py
│   │   └── urls.py
│   │
│   ├── blog/                  # Django app
│   │   ├── __init__.py
│   │   ├── models.py
│   │   ├── views.py
│   │   └── urls.py
│   │
│   ├── static/
│   │   ├── css/
│   │   └── js/
│   │
│   └── templates/
│       └── base.html
│
├── nginx/                     # Nginx configuration (see Nginx Reverse Proxy below)
│   └── PROJECT.conf
│
├── db/                        # PostgreSQL configuration (if customised)
│   └── postgresql.conf
│
└── docs/                      # Project documentation
    └── index.md

## Settings Structure
- Use a single settings.py file
- Use django-environ or python-dotenv for environment variables
- Never commit .env files to version control
- Provide .env.example with all required variables documented
- Create .gitignore file
- Create a .dockerignore file

## Environment Variables

### PostgreSQL settings (only if DB_ENGINE=postgresql)
```
APP_DB_NAME=angelia2
APP_DB_USER=angelia
APP_DB_PASSWORD=changeme
DB_HOST=db
DB_PORT=5432
```

## Docker Compose — Per-Service Environment Scoping

> **New in V1-02.** The monolithic `env_file:` pattern is deprecated.

### Rule: every service declares only the env vars it actually needs

In `docker-compose.yaml`, each service uses an `environment:` block listing only the variables that service consumes, with values interpolated from `.env` (at the repo root) using `${VAR}` syntax. Do **not** use `env_file: .env` shared across services.

```yaml
services:
  app:
    image: git.helu.ca/r/myproject:latest
    command: ["web"]
    environment:
      # Django core
      - DJANGO_SETTINGS_MODULE=myproject.settings
      - SECRET_KEY=${SECRET_KEY}
      - DEBUG=${DEBUG}
      - ALLOWED_HOSTS=${ALLOWED_HOSTS}
      - CSRF_TRUSTED_ORIGINS=${CSRF_TRUSTED_ORIGINS}
      # Postgres
      - APP_DB_NAME=${APP_DB_NAME}
      - APP_DB_USER=${APP_DB_USER}
      - APP_DB_PASSWORD=${APP_DB_PASSWORD}
      - DB_HOST=${DB_HOST}
      - DB_PORT=${DB_PORT}
      # ...

  worker:
    image: git.helu.ca/r/myproject:latest
    command: ["worker"]
    environment:
      - DJANGO_SETTINGS_MODULE=myproject.settings
      - SECRET_KEY=${SECRET_KEY}
      - APP_DB_NAME=${APP_DB_NAME}
      # ...
      # NO ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS, EMAIL_* — worker doesn't serve HTTP
```

### Why this matters

1. **Least privilege / blast radius.** A compromised MCP container shouldn't see Celery broker credentials or encryption keys. A Celery worker shouldn't see `ALLOWED_HOSTS` or CSRF config — it doesn't serve HTTP. When every service shares one env file, a misconfigured secret takes down the whole stack instead of just the services that need that secret.

2. **Self-documenting surface.** Reading `docker-compose.yaml` immediately tells you what each container depends on. With `env_file:`, every container has access to every secret and you can't tell from the compose file which service uses which variable.

3. **Ansible rendering.** The compose file can be converted to a Jinja2 template and rendered per-host by an Ansible role, with secrets pulled from the vault. The `${VAR}` pattern is the natural interface between compose and Ansible.

4. **Parsing correctness.** Docker Compose's `env_file:` parser does **not** strip inline `# comments`, honours CRLF `\r` as part of values, and handles quoting differently than `python-decouple`/`django-environ`. An `.env` that works with bare-Python `manage.py runserver` can silently feed a mangled URL (e.g. `CELERY_BROKER_URL` with a trailing `\r` or stray comment) to a container. Shell-style `${VAR}` interpolation avoids this because the value is unescaped by the shell the same way every time.

### `.env.example` template convention

Annotate each variable with which service(s) consume it:

```bash
# --- Django core ------------------------------------------------------------
# Consumed by: app, mcp, worker
SECRET_KEY=change-me-to-a-real-secret-key
DEBUG=False

# --- PostgreSQL ------------------------------------------------------------
# Consumed by: app, mcp, worker
APP_DB_NAME=myproject
APP_DB_USER=myproject
APP_DB_PASSWORD=change-me

# --- Celery / RabbitMQ -----------------------------------------------------
# Consumed by: app (producer), worker (consumer). NOT mcp.
# Percent-encode any password chars with URL meaning: @ : / # % + ? & = and space
CELERY_BROKER_URL=amqp://myproject:change-me@oberon.incus:5672/myproject
```

### Diagnostic: "what did Django actually parse?"

When a service misbehaves on startup (broker 403, DB auth error, unreachable cache), the fastest diagnostic is to print what settings.py actually resolved to — that removes every layer of env-file / interpolation / URL-encoding ambiguity:

```bash
docker compose run --rm --no-deps worker \
    python -c "from django.conf import settings; print(repr(settings.CELERY_BROKER_URL))"

docker compose run --rm --no-deps app \
    python -c "from django.conf import settings; print(settings.DATABASES['default'])"
```

The `repr(...)` form surfaces CRLF, trailing whitespace, stray quotes, and characters that should have been percent-encoded but weren't.

### Broker URL gotcha (documented for every new project)

RabbitMQ connection URLs must percent-encode any password character with URL meaning (`@ : / # % + ? & =` and space). Kombu's URL parser is strict — an unencoded `#` in the password is read as the start of a URL fragment, and an unencoded `@` shifts the username/host boundary, both causing `ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN` at worker startup. Bare-Python tests that pass the password as a kwarg rather than a URL won't exhibit this and can mask the bug.

## Nginx Reverse Proxy

> **New in V1-02.** Standard reference config for any Red Panda project running behind HAProxy on Titania.

Deployed as a sidecar container in the compose stack, fronting the Django app (gunicorn) and — where applicable — an MCP or streaming service. HAProxy handles TLS termination; nginx is plain HTTP on the internal network.

### Required building blocks

1. **Docker DNS resolver + variable-based `proxy_pass`.** `upstream` blocks resolve container hostnames **once at startup** and cache the IP forever. When `docker compose restart app` assigns a new IP, nginx returns 502 until fully reloaded. Use:

   ```nginx
   resolver 127.0.0.11 valid=10s;
   server {
       set $backend_app http://app:8000;
       location / {
           proxy_pass $backend_app;   # variable → re-resolve each request
       }
   }
   ```

2. **`$proxy_x_forwarded_proto` map.** Inside the compose network `$scheme` is always `http` because HAProxy already terminated TLS. Passing `$scheme` to Django breaks `request.is_secure()`, secure cookies, and `build_absolute_uri()`. Preserve the HAProxy header:

   ```nginx
   map $http_x_forwarded_proto $proxy_x_forwarded_proto {
       default $http_x_forwarded_proto;
       ""      $scheme;
   }
   # Then in every proxy block:
   proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
   ```

3. **Access-log suppression for probe paths.** HAProxy and Prometheus probe every 15–30 s; logging them floods Loki. The `nginx:alpine` image has a default http-level `access_log`, so a server-level `access_log` is *additive*, not replacing. You need both:

   ```nginx
   map $request_uri $loggable {
       default                     1;
       ~^/live(/|\?|$)             0;
       ~^/ready(/|\?|$)            0;
       ~^/metrics(/|\?|$)          0;
       ~^/healthz(/|\?|$)          0;
       ~^/nginx_status(/|\?|$)     0;
       ~^/mcp/health(/|\?|$)       0;
   }

   access_log off;                                   # defeat inherited default
   access_log /dev/stdout combined if=$loggable;     # then install filtered version
   ```

4. **Internal-network allowlist for all probe + metric endpoints.** RFC1918 + loopback, applied to `/live/`, `/ready/`, `/healthz`, `/metrics`, and `/nginx_status`:

   ```nginx
   location = /metrics {
       allow 127.0.0.0/8;     # loopback
       allow 10.0.0.0/8;      # RFC1918 — primary internal (Incus, HAProxy)
       allow 172.16.0.0/12;   # RFC1918 — Docker bridge networks
       allow 192.168.0.0/16;  # RFC1918
       deny  all;
       proxy_pass $backend_app;
       # ...
   }
   ```

   All four RFC1918 ranges must be present — omitting `172.16.0.0/12` silently breaks scrapes from a Prometheus container on the default Docker bridge.

5. **Security headers on the catch-all**, marked `always` so they apply to upstream 4xx/5xx:

   ```nginx
   add_header X-Frame-Options        "SAMEORIGIN" always;
   add_header X-Content-Type-Options "nosniff"    always;
   add_header X-XSS-Protection       "1; mode=block" always;
   ```

   Stronger policies (CSP, Referrer-Policy, HSTS) are set at HAProxy so they're consistent across every backend.

6. **Catch-all proxies to Django.** nginx should intercept only the paths that need special handling (`/static/`, `/media/`, `/mcp/`, `/healthz`, `/metrics`, `/nginx_status`, the probes). Everything else flows through to Django, which returns its own themed 404 for unrouted paths — not nginx's bare default page.

### Minimal reference config

```nginx
resolver 127.0.0.11 valid=10s;

map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    ""      $scheme;
}

map $request_uri $loggable {
    default                     1;
    ~^/live(/|\?|$)             0;
    ~^/ready(/|\?|$)            0;
    ~^/metrics(/|\?|$)          0;
    ~^/healthz(/|\?|$)          0;
    ~^/nginx_status(/|\?|$)     0;
}

access_log off;
access_log /dev/stdout combined if=$loggable;

server {
    listen 80 default_server;
    server_name _;

    client_max_body_size 64m;

    set $backend_app http://app:8000;

    add_header X-Frame-Options        "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff"    always;
    add_header X-XSS-Protection       "1; mode=block" always;

    location /static/ {
        alias /var/www/static/;
        access_log off;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    location /media/ {
        alias /var/www/media/;
        access_log off;
        expires 7d;
    }

    # Internal-only endpoints — allowlist applied uniformly
    location = /live/      { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
    location = /ready/     { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
    location = /metrics    { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
    location = /nginx_status {
        include /etc/nginx/snippets/internal-only.conf;
        stub_status on;
        access_log off;
    }

    location / {
        proxy_pass $backend_app;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
        proxy_redirect    off;
        proxy_read_timeout 300s;
    }
}
```

> Projects may inline the `proxy_set_header` block rather than using snippets; both are acceptable. The important thing is that every `proxy_pass` has the same four headers (`Host`, `X-Real-IP`, `X-Forwarded-For`, `X-Forwarded-Proto` using `$proxy_x_forwarded_proto`).

## Memcached

> **New in V1-02.** Memcached is a standard Red Panda dependency. Every Django service uses it for session storage, task-progress tracking, and cheap key-value caching.

- Package: `pymemcache` (via `django.core.cache.backends.memcached.PyMemcacheCache`)
- Key prefix: per-app, configured via env var (e.g. `KVDB_PREFIX=mnemosyne`)
- Cache-key pattern: `{app}:{model}:{identifier}:{field}`

### Deployment convention

Memcached runs as a service on the application host (in Ouranos: a package install per Incus container). Configure it to bind to **all interfaces**, not just `localhost`, so:

- Containers on the same host can reach it via the host's LAN name (e.g. `puck.incus:11211`).
- Other hosts in the lab can reach it for multi-host debugging.

```bash
# /etc/memcached.conf on the Docker host
-l 0.0.0.0
-p 11211
-U 0
```

The bare-Python "`localhost:11211` works" default is a dev-only convenience — it breaks as soon as Django moves into a container, because inside the container `127.0.0.1` is the container itself. The production `.env` must use the LAN-resolvable hostname:

```
KVDB_LOCATION=puck.incus:11211
KVDB_PREFIX=myproject
```

### Health-check reachability

The Django readiness probe (`GET /ready/`) must verify Memcached is reachable. If the probe returns 503 and the log shows no cause, hit the endpoint directly to read the JSON body which names the broken dependency:

```bash
docker compose exec app curl -sS -o - -w "\nHTTP %{http_code}\n" http://localhost:8000/ready/
```

## Code Organization
- Imports: PEP 8 ordering (stdlib, third-party, local)
- Type hints on function parameters
- CSS: External .css files only (no inline styles, no embedded `<style>` tags)
- JS: External .js files only (no inline handlers, no embedded `<script>` blocks)
- Maximum file length: 1000 lines
- If a file exceeds 500 lines, consider splitting by domain concept

## Database Conventions
- Migrations run cleanly from empty database
- Never edit deployed migrations
- Use meaningful migration names: --name add_email_to_profile
- One logical change per migration when possible
- Test migrations both forward and backward

### Development vs Production
- Development: SQLite
- Production: PostgreSQL

## Caching
- Expensive queries are cached
- Cache keys follow naming convention
- TTLs are appropriate (not infinite)
- Invalidation is documented
- Key Naming Pattern: {app}:{model}:{identifier}:{field}
- See the **Memcached** section above for deployment details

## Model Naming
- Model names: singular PascalCase (User, BlogPost, OrderItem)
- Correct English pluralization on related names
- All models have created_at and updated_at
- All models define __str__ and get_absolute_url
- TextChoices used for status fields
- related_name defined on ForeignKey fields
- Related names: plural snake_case with proper English pluralization

## Forms
- Use ModelForm with explicit fields list (never __all__)

## Field Naming
- Foreign keys: singular without _id suffix (author, category, parent)
- Boolean fields: use prefixes (is_active, has_permission, can_edit)
- Date fields: use suffixes (created_at, updated_at, published_on)
- Avoid abbreviations (use description, not desc)

## Required Model Fields
- All models should include:
  - created_at = models.DateTimeField(auto_now_add=True)
  - updated_at = models.DateTimeField(auto_now=True)
- Consider adding:
  - id = models.UUIDField(primary_key=True) for public-facing models
  - is_active = models.BooleanField(default=True) for soft deletes

## Indexing
- Add db_index=True to frequently queried fields
- Use Meta.indexes for composite indexes
- Document why each index exists

## Queries
- Use select_related() for foreign keys
- Use prefetch_related() for reverse relations and M2M
- Avoid queries in loops (N+1 problem)
- Use .only() and .defer() for large models
- Add comments explaining complex querysets

## Docstrings
- Use Sphinx style docstrings
- Document all public functions, classes, and modules
- Skip docstrings for obvious one-liners and standard Django overrides

## Views
- Use Function-Based Views (FBVs) exclusively
- Explicit logic is preferred over implicit inheritance
- Extract shared logic into utility functions

## URLs & Identifiers

- Public URLs use short UUIDs (12 characters) via `shortuuid`
- Never expose sequential IDs in URLs (security/enumeration risk)
- Internal references may use standard UUIDs or PKs

## URL Patterns
- Resource-based URLs (RESTful style)
- Namespaced URL names per app
- Trailing slashes (Django default)
- Flat structure preferred over deep nesting

## Background Tasks
- All tasks are run synchronously unless the design specifies background tasks are needed for long operations
- Long operations use Celery tasks
- Use Memcached, task progress pattern: {app}:task:{task_id}:progress
- Tasks are idempotent
- Tasks include retry logic
- Tasks live in app/tasks.py
- RabbitMQ is the Message Broker
- Flower Monitoring: Use for debugging failed tasks
- Per-service env scoping: the Celery worker container consumes `CELERY_BROKER_URL` + `LLM_API_SECRETS_ENCRYPTION_KEY` + `DAEDALUS_S3_*` but NOT `ALLOWED_HOSTS`/`CSRF_TRUSTED_ORIGINS`/`MCP_REQUIRE_AUTH`/`EMAIL_*` (see Docker Compose section)

## Testing
- Framework: Django TestCase (not pytest)
- Separate test files per module: test_models.py, test_views.py, test_forms.py

## Frontend Standards

### New Projects (DaisyUI + Tailwind)
- DaisyUI 4 via CDN for component classes
- Tailwind CSS via CDN for utility classes
- Theme management via Themis (DaisyUI `data-theme` attribute)
- All apps extend `themis/base.html` for consistent navigation
- No inline styles or scripts

### Existing Projects (Bootstrap 5)
- Bootstrap 5 via CDN
- Bootstrap Icons via CDN
- Bootswatch for theme variants (if applicable)
- django-bootstrap5 and crispy-bootstrap5 for form rendering

## Preferred Packages

### Core Django
- django>=5.2,<6.0
- django-environ — Environment variables

### Authentication & Security
- django-allauth — User management
- django-allauth-2fa — Two-factor authentication

### API Development
- djangorestframework>=3.14,<4.0 — REST APIs
- drf-spectacular — OpenAPI/Swagger documentation

### Encryption
- cryptography — Fernet encryption for secrets/API keys

### Background Tasks
- celery — Async task queue
- django-celery-progress — Progress bars
- flower — Celery monitoring

### Caching
- pymemcache — Memcached backend

### Database
- psycopg[binary] — PostgreSQL adapter
- shortuuid — Short UUIDs for public URLs

### Production
- gunicorn — WSGI server

### Shared Apps
- django-heluca-themis — User preferences, themes, key management, navigation

### Deprecated / Removed
- ~~pytz~~ — Use stdlib `zoneinfo` (Python 3.9+, Django 4+)
- ~~Pillow~~ — Only add if your app needs ImageField
- ~~django-heluca-core~~ — Replaced by Themis
- ~~dj-database-url~~ — Use individual Django DB env vars instead

## Anti-Patterns to Avoid

### Docker Compose
- Don't share a single `env_file:` across services (see per-service scoping above)
- Don't put secrets in the compose file's `environment:` block as literals — use `${VAR}` interpolation
- Don't commit a populated `.env` — only `.env.example`

### Nginx
- Don't use `upstream` blocks for container hostnames without `resolver` + variable `proxy_pass` (nginx caches the IP and returns 502 after container restart)
- Don't pass `$scheme` as `X-Forwarded-Proto` when behind an external TLS terminator — use `$proxy_x_forwarded_proto` via the map pattern
- Don't rely on server-level `access_log` to override the image default — explicitly `access_log off;` first
- Don't allowlist only `10.0.0.0/8` for `/metrics` — also include `172.16.0.0/12` for Docker bridge sources

### Memcached
- Don't bind to `127.0.0.1` only on a host that runs Docker services — containers can't reach it
- Don't use `KVDB_LOCATION=127.0.0.1:11211` in a containerised `.env` (127.0.0.1 is the container itself)

### Models
- Don't use `Model.objects.get()` without handling `DoesNotExist`
- Don't use `null=True` on `CharField` or `TextField` (use `blank=True, default=""`)
- Don't use `related_name='+'` unless you have a specific reason
- Don't override `save()` for business logic (use signals or service functions)
- Don't use `auto_now=True` on fields you might need to manually set
- Don't use `ForeignKey` without specifying `on_delete` explicitly
- Don't use `Meta.ordering` on large tables (specify ordering in queries)

### Queries
- Don't query inside loops (N+1 problem)
- Don't use `.all()` when you need a subset
- Don't use raw SQL unless absolutely necessary
- Don't forget `select_related()` and `prefetch_related()`

### Views
- Don't put business logic in views
- Don't use `request.POST.get()` without validation (use forms)
- Don't return sensitive data in error messages
- Don't forget `login_required` decorator on protected views

### Forms
- Don't use `fields = '__all__'` in ModelForm
- Don't trust client-side validation alone
- Don't use `exclude` in ModelForm (use explicit `fields`)

### Templates
- Don't use `{{ variable }}` for URLs (use `{% url %}` tag)
- Don't put logic in templates
- Don't use inline CSS or JavaScript (external files only)
- Don't forget `{% csrf_token %}` in forms

### Security
- Don't store secrets in `settings.py` (use environment variables)
- Don't commit `.env` files to version control
- Don't use `DEBUG=True` in production
- Don't expose sequential IDs in public URLs
- Don't use `mark_safe()` on user-supplied content
- Don't disable CSRF protection

### Imports & Code Style
- Don't use `from module import *`
- Don't use mutable default arguments
- Don't use bare `except:` clauses
- Don't ignore linter warnings without documented reason

### Migrations
- Don't edit migrations that have been deployed
- Don't use `RunPython` without a reverse function
- Don't add non-nullable fields without a default value

### Celery Tasks
- Don't pass model instances to tasks (pass IDs and re-fetch)
- Don't assume tasks run immediately
- Don't forget retry logic for external service calls
- Don't forget to percent-encode special characters in `CELERY_BROKER_URL` (`@ : / # % + ? & =` and space)