r/mnemosyne
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add Red Panda Django Standards V1-02
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 49s
Details
CVE Scan & Docker Build / build-and-push (push) Successful in 42s
Details

Browse Source 
Introduces the Red Panda Approval standards document for Django projects,
covering environment setup, directory structure, dependency pinning,
Docker Compose per-service environment scoping, nginx reverse-proxy
configuration (Docker DNS, X-Forwarded-Proto preservation, access-log
filtering, internal allowlists), and Memcached deployment notes.
This commit is contained in:
Robert Helewka
2026-05-04 07:47:08 -04:00
parent c9328c58fc
commit df2e495660
1 changed files with 614 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

614
docs/Red Panda Standards_Django_V1-02.md Normal file
View File

@@ -0,0 +1,614 @@
## 🐾 Red Panda Approval™
This project follows Red Panda Approval standards — our gold standard for Django application quality. Code must be elegant, reliable, and maintainable to earn the approval of our adorable red panda judges.
### The 5 Sacred Django Criteria
1. **Fresh Migration Test** — Clean migrations from empty database
2. **Elegant Simplicity** — No unnecessary complexity
3. **Observable & Debuggable** — Proper logging and error handling
4. **Consistent Patterns** — Follow Django conventions
5. **Actually Works** — Passes all checks and serves real user needs
### Changelog
- **V1-02** — Added Docker Compose environment-scoping standard (per-service `environment:` blocks), nginx reverse-proxy reference config (Docker DNS resolver, `X-Forwarded-Proto` preservation, access-log filtering, internal-network allowlists), and Memcached deployment note (bind to `0.0.0.0`, not `localhost`).
- **V1-01** — Initial published standards.
## Environment Standards
- Virtual environment: ~/env/PROJECT/bin/activate
- Use pyproject.toml for project configuration (no setup.py, no requirements.txt)
- Python version: specified in pyproject.toml
- Dependencies: floor-pinned with ceiling (e.g. `Django>=5.2,<6.0`)
### Dependency Pinning
```toml
# Correct — floor pin with ceiling
dependencies = [
"Django>=5.2,<6.0",
"djangorestframework>=3.14,<4.0",
"cryptography>=41.0,<45.0",
]
# Wrong — exact pins in library packages
dependencies = [
"Django==5.2.7", # too strict, breaks downstream
]
```
Exact pins (`==`) are only appropriate in application-level lock files, not in reusable library packages.
## Directory Structure
myproject/ # Git repository root
├── .gitignore
├── README.md
├── pyproject.toml # Project configuration (moved to repo root)
├── docker-compose.yaml # Per-service environment scoping (see below)
├── .env # Docker Compose interpolation source — NOT committed
├── .env.example # Template listing every `${VAR}` with which service consumes it
│
├── project/ # Django project root (manage.py lives here)
│ ├── manage.py
│ ├── Dockerfile
│ ├── .env # Local bare-Python dev environment (runserver, celery, etc.)
│ │ # Only read by bare-Python runs; NOT by the compose stack
│ ├── .env.example
│ │
│ ├── config/ # Django configuration module
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ ├── wsgi.py
│ │ └── asgi.py
│ │
│ ├── accounts/ # Django app
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── views.py
│ │ └── urls.py
│ │
│ ├── blog/ # Django app
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── views.py
│ │ └── urls.py
│ │
│ ├── static/
│ │ ├── css/
│ │ └── js/
│ │
│ └── templates/
│ └── base.html
│
├── nginx/ # Nginx configuration (see Nginx Reverse Proxy below)
│ └── PROJECT.conf
│
├── db/ # PostgreSQL configuration (if customised)
│ └── postgresql.conf
│
└── docs/ # Project documentation
└── index.md
## Settings Structure
- Use a single settings.py file
- Use django-environ or python-dotenv for environment variables
- Never commit .env files to version control
- Provide .env.example with all required variables documented
- Create .gitignore file
- Create a .dockerignore file
## Environment Variables
### PostgreSQL settings (only if DB_ENGINE=postgresql)
```
APP_DB_NAME=angelia2
APP_DB_USER=angelia
APP_DB_PASSWORD=changeme
DB_HOST=db
DB_PORT=5432
```
## Docker Compose — Per-Service Environment Scoping
> **New in V1-02.** The monolithic `env_file:` pattern is deprecated.
### Rule: every service declares only the env vars it actually needs
In `docker-compose.yaml`, each service uses an `environment:` block listing only the variables that service consumes, with values interpolated from `.env` (at the repo root) using `${VAR}` syntax. Do **not** use `env_file: .env` shared across services.
```yaml
services:
app:
image: git.helu.ca/r/myproject:latest
command: ["web"]
environment:
# Django core
- DJANGO_SETTINGS_MODULE=myproject.settings
- SECRET_KEY=${SECRET_KEY}
- DEBUG=${DEBUG}
- ALLOWED_HOSTS=${ALLOWED_HOSTS}
- CSRF_TRUSTED_ORIGINS=${CSRF_TRUSTED_ORIGINS}
# Postgres
- APP_DB_NAME=${APP_DB_NAME}
- APP_DB_USER=${APP_DB_USER}
- APP_DB_PASSWORD=${APP_DB_PASSWORD}
- DB_HOST=${DB_HOST}
- DB_PORT=${DB_PORT}
# ...
worker:
image: git.helu.ca/r/myproject:latest
command: ["worker"]
environment:
- DJANGO_SETTINGS_MODULE=myproject.settings
- SECRET_KEY=${SECRET_KEY}
- APP_DB_NAME=${APP_DB_NAME}
# ...
# NO ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS, EMAIL_* — worker doesn't serve HTTP
```
### Why this matters
1. **Least privilege / blast radius.** A compromised MCP container shouldn't see Celery broker credentials or encryption keys. A Celery worker shouldn't see `ALLOWED_HOSTS` or CSRF config — it doesn't serve HTTP. When every service shares one env file, a misconfigured secret takes down the whole stack instead of just the services that need that secret.
2. **Self-documenting surface.** Reading `docker-compose.yaml` immediately tells you what each container depends on. With `env_file:`, every container has access to every secret and you can't tell from the compose file which service uses which variable.
3. **Ansible rendering.** The compose file can be converted to a Jinja2 template and rendered per-host by an Ansible role, with secrets pulled from the vault. The `${VAR}` pattern is the natural interface between compose and Ansible.
4. **Parsing correctness.** Docker Compose's `env_file:` parser does **not** strip inline `# comments`, honours CRLF `\r` as part of values, and handles quoting differently than `python-decouple`/`django-environ`. An `.env` that works with bare-Python `manage.py runserver` can silently feed a mangled URL (e.g. `CELERY_BROKER_URL` with a trailing `\r` or stray comment) to a container. Shell-style `${VAR}` interpolation avoids this because the value is unescaped by the shell the same way every time.
### `.env.example` template convention
Annotate each variable with which service(s) consume it:
```bash
# --- Django core ------------------------------------------------------------
# Consumed by: app, mcp, worker
SECRET_KEY=change-me-to-a-real-secret-key
DEBUG=False
# --- PostgreSQL ------------------------------------------------------------
# Consumed by: app, mcp, worker
APP_DB_NAME=myproject
APP_DB_USER=myproject
APP_DB_PASSWORD=change-me
# --- Celery / RabbitMQ -----------------------------------------------------
# Consumed by: app (producer), worker (consumer). NOT mcp.
# Percent-encode any password chars with URL meaning: @ : / # % + ? & = and space
CELERY_BROKER_URL=amqp://myproject:change-me@oberon.incus:5672/myproject
```
### Diagnostic: "what did Django actually parse?"
When a service misbehaves on startup (broker 403, DB auth error, unreachable cache), the fastest diagnostic is to print what settings.py actually resolved to — that removes every layer of env-file / interpolation / URL-encoding ambiguity:
```bash
docker compose run --rm --no-deps worker \
python -c "from django.conf import settings; print(repr(settings.CELERY_BROKER_URL))"
docker compose run --rm --no-deps app \
python -c "from django.conf import settings; print(settings.DATABASES['default'])"
```
The `repr(...)` form surfaces CRLF, trailing whitespace, stray quotes, and characters that should have been percent-encoded but weren't.
### Broker URL gotcha (documented for every new project)
RabbitMQ connection URLs must percent-encode any password character with URL meaning (`@ : / # % + ? & =` and space). Kombu's URL parser is strict — an unencoded `#` in the password is read as the start of a URL fragment, and an unencoded `@` shifts the username/host boundary, both causing `ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN` at worker startup. Bare-Python tests that pass the password as a kwarg rather than a URL won't exhibit this and can mask the bug.
## Nginx Reverse Proxy
> **New in V1-02.** Standard reference config for any Red Panda project running behind HAProxy on Titania.
Deployed as a sidecar container in the compose stack, fronting the Django app (gunicorn) and — where applicable — an MCP or streaming service. HAProxy handles TLS termination; nginx is plain HTTP on the internal network.
### Required building blocks
1. **Docker DNS resolver + variable-based `proxy_pass`.** `upstream` blocks resolve container hostnames **once at startup** and cache the IP forever. When `docker compose restart app` assigns a new IP, nginx returns 502 until fully reloaded. Use:
```nginx
resolver 127.0.0.11 valid=10s;
server {
set $backend_app http://app:8000;
location / {
proxy_pass $backend_app; # variable → re-resolve each request
}
}
```
2. **`$proxy_x_forwarded_proto` map.** Inside the compose network `$scheme` is always `http` because HAProxy already terminated TLS. Passing `$scheme` to Django breaks `request.is_secure()`, secure cookies, and `build_absolute_uri()`. Preserve the HAProxy header:
```nginx
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
"" $scheme;
}
# Then in every proxy block:
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
```
3. **Access-log suppression for probe paths.** HAProxy and Prometheus probe every 15–30 s; logging them floods Loki. The `nginx:alpine` image has a default http-level `access_log`, so a server-level `access_log` is *additive*, not replacing. You need both:
```nginx
map $request_uri $loggable {
default 1;
~^/live(/|\?|$) 0;
~^/ready(/|\?|$) 0;
~^/metrics(/|\?|$) 0;
~^/healthz(/|\?|$) 0;
~^/nginx_status(/|\?|$) 0;
~^/mcp/health(/|\?|$) 0;
}
access_log off; # defeat inherited default
access_log /dev/stdout combined if=$loggable; # then install filtered version
```
4. **Internal-network allowlist for all probe + metric endpoints.** RFC1918 + loopback, applied to `/live/`, `/ready/`, `/healthz`, `/metrics`, and `/nginx_status`:
```nginx
location = /metrics {
allow 127.0.0.0/8; # loopback
allow 10.0.0.0/8; # RFC1918 — primary internal (Incus, HAProxy)
allow 172.16.0.0/12; # RFC1918 — Docker bridge networks
allow 192.168.0.0/16; # RFC1918
deny all;
proxy_pass $backend_app;
# ...
}
```
All four RFC1918 ranges must be present — omitting `172.16.0.0/12` silently breaks scrapes from a Prometheus container on the default Docker bridge.
5. **Security headers on the catch-all**, marked `always` so they apply to upstream 4xx/5xx:
```nginx
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
```
Stronger policies (CSP, Referrer-Policy, HSTS) are set at HAProxy so they're consistent across every backend.
6. **Catch-all proxies to Django.** nginx should intercept only the paths that need special handling (`/static/`, `/media/`, `/mcp/`, `/healthz`, `/metrics`, `/nginx_status`, the probes). Everything else flows through to Django, which returns its own themed 404 for unrouted paths — not nginx's bare default page.
### Minimal reference config
```nginx
resolver 127.0.0.11 valid=10s;
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
"" $scheme;
}
map $request_uri $loggable {
default 1;
~^/live(/|\?|$) 0;
~^/ready(/|\?|$) 0;
~^/metrics(/|\?|$) 0;
~^/healthz(/|\?|$) 0;
~^/nginx_status(/|\?|$) 0;
}
access_log off;
access_log /dev/stdout combined if=$loggable;
server {
listen 80 default_server;
server_name _;
client_max_body_size 64m;
set $backend_app http://app:8000;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
location /static/ {
alias /var/www/static/;
access_log off;
expires 30d;
add_header Cache-Control "public, immutable";
}
location /media/ {
alias /var/www/media/;
access_log off;
expires 7d;
}
# Internal-only endpoints — allowlist applied uniformly
location = /live/ { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
location = /ready/ { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
location = /metrics { include /etc/nginx/snippets/internal-only.conf; proxy_pass $backend_app; include /etc/nginx/snippets/proxy-headers.conf; access_log off; }
location = /nginx_status {
include /etc/nginx/snippets/internal-only.conf;
stub_status on;
access_log off;
}
location / {
proxy_pass $backend_app;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_redirect off;
proxy_read_timeout 300s;
}
}
```
> Projects may inline the `proxy_set_header` block rather than using snippets; both are acceptable. The important thing is that every `proxy_pass` has the same four headers (`Host`, `X-Real-IP`, `X-Forwarded-For`, `X-Forwarded-Proto` using `$proxy_x_forwarded_proto`).
## Memcached
> **New in V1-02.** Memcached is a standard Red Panda dependency. Every Django service uses it for session storage, task-progress tracking, and cheap key-value caching.
- Package: `pymemcache` (via `django.core.cache.backends.memcached.PyMemcacheCache`)
- Key prefix: per-app, configured via env var (e.g. `KVDB_PREFIX=mnemosyne`)
- Cache-key pattern: `{app}:{model}:{identifier}:{field}`
### Deployment convention
Memcached runs as a service on the application host (in Ouranos: a package install per Incus container). Configure it to bind to **all interfaces**, not just `localhost`, so:
- Containers on the same host can reach it via the host's LAN name (e.g. `puck.incus:11211`).
- Other hosts in the lab can reach it for multi-host debugging.
```bash
# /etc/memcached.conf on the Docker host
-l 0.0.0.0
-p 11211
-U 0
```
The bare-Python "`localhost:11211` works" default is a dev-only convenience — it breaks as soon as Django moves into a container, because inside the container `127.0.0.1` is the container itself. The production `.env` must use the LAN-resolvable hostname:
```
KVDB_LOCATION=puck.incus:11211
KVDB_PREFIX=myproject
```
### Health-check reachability
The Django readiness probe (`GET /ready/`) must verify Memcached is reachable. If the probe returns 503 and the log shows no cause, hit the endpoint directly to read the JSON body which names the broken dependency:
```bash
docker compose exec app curl -sS -o - -w "\nHTTP %{http_code}\n" http://localhost:8000/ready/
```
## Code Organization
- Imports: PEP 8 ordering (stdlib, third-party, local)
- Type hints on function parameters
- CSS: External .css files only (no inline styles, no embedded `<style>` tags)
- JS: External .js files only (no inline handlers, no embedded `<script>` blocks)
- Maximum file length: 1000 lines
- If a file exceeds 500 lines, consider splitting by domain concept
## Database Conventions
- Migrations run cleanly from empty database
- Never edit deployed migrations
- Use meaningful migration names: --name add_email_to_profile
- One logical change per migration when possible
- Test migrations both forward and backward
### Development vs Production
- Development: SQLite
- Production: PostgreSQL
## Caching
- Expensive queries are cached
- Cache keys follow naming convention
- TTLs are appropriate (not infinite)
- Invalidation is documented
- Key Naming Pattern: {app}:{model}:{identifier}:{field}
- See the **Memcached** section above for deployment details
## Model Naming
- Model names: singular PascalCase (User, BlogPost, OrderItem)
- Correct English pluralization on related names
- All models have created_at and updated_at
- All models define __str__ and get_absolute_url
- TextChoices used for status fields
- related_name defined on ForeignKey fields
- Related names: plural snake_case with proper English pluralization
## Forms
- Use ModelForm with explicit fields list (never __all__)
## Field Naming
- Foreign keys: singular without _id suffix (author, category, parent)
- Boolean fields: use prefixes (is_active, has_permission, can_edit)
- Date fields: use suffixes (created_at, updated_at, published_on)
- Avoid abbreviations (use description, not desc)
## Required Model Fields
- All models should include:
- created_at = models.DateTimeField(auto_now_add=True)
- updated_at = models.DateTimeField(auto_now=True)
- Consider adding:
- id = models.UUIDField(primary_key=True) for public-facing models
- is_active = models.BooleanField(default=True) for soft deletes
## Indexing
- Add db_index=True to frequently queried fields
- Use Meta.indexes for composite indexes
- Document why each index exists
## Queries
- Use select_related() for foreign keys
- Use prefetch_related() for reverse relations and M2M
- Avoid queries in loops (N+1 problem)
- Use .only() and .defer() for large models
- Add comments explaining complex querysets
## Docstrings
- Use Sphinx style docstrings
- Document all public functions, classes, and modules
- Skip docstrings for obvious one-liners and standard Django overrides
## Views
- Use Function-Based Views (FBVs) exclusively
- Explicit logic is preferred over implicit inheritance
- Extract shared logic into utility functions
## URLs & Identifiers
- Public URLs use short UUIDs (12 characters) via `shortuuid`
- Never expose sequential IDs in URLs (security/enumeration risk)
- Internal references may use standard UUIDs or PKs
## URL Patterns
- Resource-based URLs (RESTful style)
- Namespaced URL names per app
- Trailing slashes (Django default)
- Flat structure preferred over deep nesting
## Background Tasks
- All tasks are run synchronously unless the design specifies background tasks are needed for long operations
- Long operations use Celery tasks
- Use Memcached, task progress pattern: {app}:task:{task_id}:progress
- Tasks are idempotent
- Tasks include retry logic
- Tasks live in app/tasks.py
- RabbitMQ is the Message Broker
- Flower Monitoring: Use for debugging failed tasks
- Per-service env scoping: the Celery worker container consumes `CELERY_BROKER_URL` + `LLM_API_SECRETS_ENCRYPTION_KEY` + `DAEDALUS_S3_*` but NOT `ALLOWED_HOSTS`/`CSRF_TRUSTED_ORIGINS`/`MCP_REQUIRE_AUTH`/`EMAIL_*` (see Docker Compose section)
## Testing
- Framework: Django TestCase (not pytest)
- Separate test files per module: test_models.py, test_views.py, test_forms.py
## Frontend Standards
### New Projects (DaisyUI + Tailwind)
- DaisyUI 4 via CDN for component classes
- Tailwind CSS via CDN for utility classes
- Theme management via Themis (DaisyUI `data-theme` attribute)
- All apps extend `themis/base.html` for consistent navigation
- No inline styles or scripts
### Existing Projects (Bootstrap 5)
- Bootstrap 5 via CDN
- Bootstrap Icons via CDN
- Bootswatch for theme variants (if applicable)
- django-bootstrap5 and crispy-bootstrap5 for form rendering
## Preferred Packages
### Core Django
- django>=5.2,<6.0
- django-environ — Environment variables
### Authentication & Security
- django-allauth — User management
- django-allauth-2fa — Two-factor authentication
### API Development
- djangorestframework>=3.14,<4.0 — REST APIs
- drf-spectacular — OpenAPI/Swagger documentation
### Encryption
- cryptography — Fernet encryption for secrets/API keys
### Background Tasks
- celery — Async task queue
- django-celery-progress — Progress bars
- flower — Celery monitoring
### Caching
- pymemcache — Memcached backend
### Database
- psycopg[binary] — PostgreSQL adapter
- shortuuid — Short UUIDs for public URLs
### Production
- gunicorn — WSGI server
### Shared Apps
- django-heluca-themis — User preferences, themes, key management, navigation
### Deprecated / Removed
- ~~pytz~~ — Use stdlib `zoneinfo` (Python 3.9+, Django 4+)
- ~~Pillow~~ — Only add if your app needs ImageField
- ~~django-heluca-core~~ — Replaced by Themis
- ~~dj-database-url~~ — Use individual Django DB env vars instead
## Anti-Patterns to Avoid
### Docker Compose
- Don't share a single `env_file:` across services (see per-service scoping above)
- Don't put secrets in the compose file's `environment:` block as literals — use `${VAR}` interpolation
- Don't commit a populated `.env` — only `.env.example`
### Nginx
- Don't use `upstream` blocks for container hostnames without `resolver` + variable `proxy_pass` (nginx caches the IP and returns 502 after container restart)
- Don't pass `$scheme` as `X-Forwarded-Proto` when behind an external TLS terminator — use `$proxy_x_forwarded_proto` via the map pattern
- Don't rely on server-level `access_log` to override the image default — explicitly `access_log off;` first
- Don't allowlist only `10.0.0.0/8` for `/metrics` — also include `172.16.0.0/12` for Docker bridge sources
### Memcached
- Don't bind to `127.0.0.1` only on a host that runs Docker services — containers can't reach it
- Don't use `KVDB_LOCATION=127.0.0.1:11211` in a containerised `.env` (127.0.0.1 is the container itself)
### Models
- Don't use `Model.objects.get()` without handling `DoesNotExist`
- Don't use `null=True` on `CharField` or `TextField` (use `blank=True, default=""`)
- Don't use `related_name='+'` unless you have a specific reason
- Don't override `save()` for business logic (use signals or service functions)
- Don't use `auto_now=True` on fields you might need to manually set
- Don't use `ForeignKey` without specifying `on_delete` explicitly
- Don't use `Meta.ordering` on large tables (specify ordering in queries)
### Queries
- Don't query inside loops (N+1 problem)
- Don't use `.all()` when you need a subset
- Don't use raw SQL unless absolutely necessary
- Don't forget `select_related()` and `prefetch_related()`
### Views
- Don't put business logic in views
- Don't use `request.POST.get()` without validation (use forms)
- Don't return sensitive data in error messages
- Don't forget `login_required` decorator on protected views
### Forms
- Don't use `fields = '__all__'` in ModelForm
- Don't trust client-side validation alone
- Don't use `exclude` in ModelForm (use explicit `fields`)
### Templates
- Don't use `{{ variable }}` for URLs (use `{% url %}` tag)
- Don't put logic in templates
- Don't use inline CSS or JavaScript (external files only)
- Don't forget `{% csrf_token %}` in forms
### Security
- Don't store secrets in `settings.py` (use environment variables)
- Don't commit `.env` files to version control
- Don't use `DEBUG=True` in production
- Don't expose sequential IDs in public URLs
- Don't use `mark_safe()` on user-supplied content
- Don't disable CSRF protection
### Imports & Code Style
- Don't use `from module import *`
- Don't use mutable default arguments
- Don't use bare `except:` clauses
- Don't ignore linter warnings without documented reason
### Migrations
- Don't edit migrations that have been deployed
- Don't use `RunPython` without a reverse function
- Don't add non-nullable fields without a default value
### Celery Tasks
- Don't pass model instances to tasks (pass IDs and re-fetch)
- Don't assume tasks run immediately
- Don't forget retry logic for external service calls
- Don't forget to percent-encode special characters in `CELERY_BROKER_URL` (`@ : / # % + ? & =` and space)