docs: add Sphinx documentation build and deploy workflow

- Add Gitea Actions workflow to build and deploy docs on push to main - Generate Sphinx reference documentation for all apps and modules - Deploy versioned and latest docs via rsync over SSH
2026-05-23 06:11:05 -04:00
parent 50dffe688b
commit f8a2cf0c3d
114 changed files with 1797 additions and 2 deletions
--- a/docs/Pattern_Sphinx-Documentation_V1-00.md
+++ b/docs/Pattern_Sphinx-Documentation_V1-00.md
@@ -0,0 +1,521 @@
+# Sphinx Documentation Pattern v1.0.0
+
+Standardizes how Django projects build, configure, and deploy Sphinx documentation under a single `settings.py` — using the `TESTING` env-var flag to relax required-secret checks so docs build cleanly in CI without a real `.env`.
+
+## 🐾 Red Panda Approval™
+
+This pattern follows Red Panda Approval standards.
+
+---
+
+## Why a Pattern, Not a Shared Implementation
+
+Every Django project has its own:
+
+- **Required env vars** — one project needs `MCP_JWT_SECRET`, another needs `SLACK_TOKEN`, a third needs neither.
+- **App layout** — `apps/` vs. top-level packages; some projects ship one app, others fifteen.
+- **Autodoc-poisoning attributes** — DRF projects have class-level `queryset = Model.objects.filter(...)`; pure-Django projects may not.
+- **Deploy target** — different hosts, ports, paths, and SSH key names per environment.
+
+A shared library can't paper over those differences. Instead, this pattern defines:
+
+- **Required interface** — the four files every project must have.
+- **Recommended behaviours** — what most projects should include.
+- **Extension guidelines** — what to add or skip per project.
+- **Standard Sphinx extension set** — for consistency across projects.
+
+---
+
+## Required Interface
+
+The non-negotiable minimum every Django project must provide.
+
+### 1. `settings.py` — TESTING-gated safe defaults
+
+Every required env var (those without a `default=`) must have a `TESTING`-mode fallback. Read `TESTING` **first**, then branch every required `env('X')` call:
+
+```python
+# Test mode flag — read first so it can relax required-env-var checks below.
+TESTING = env.bool('TESTING', default=False)
+
+DEBUG = env.bool('DEBUG', default=False)
+
+# In TESTING mode (unit tests, docs build) required keys fall back to safe
+# dummies so the settings module imports without a real .env. In production
+# they remain required — missing values fail loud.
+if TESTING:
+    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
+    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS', default=['testserver', 'localhost', '127.0.0.1'])
+    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS', default=['http://localhost'])
+    # ...any other required secrets get a 'testing-insecure-*' default here
+else:
+    SECRET_KEY = env('SECRET_KEY')
+    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
+    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS')
+    # ...and the production no-default form here
+```
+
+Rule: **every** required env var read in `settings.py` (anything that uses `env('X')` without `default=`) gets paired branches like above. Production fails loud on missing; TESTING falls back.
+
+### 2. Database choice gated on `TESTING`
+
+```python
+if TESTING:
+    # Test/docs build: in-memory SQLite. No real DB needed.
+    DATABASES = {
+        'default': {
+            'ENGINE': 'django.db.backends.sqlite3',
+            'NAME': ':memory:',
+        }
+    }
+elif env('APP_DB_NAME', default=None):
+    # Production: PostgreSQL (or whatever the project uses)
+    DATABASES = {
+        'default': {
+            'ENGINE': 'django.db.backends.postgresql',
+            'NAME': env('APP_DB_NAME'),
+            'USER': env('APP_DB_USER'),
+            'PASSWORD': env('APP_DB_PASSWORD'),
+            'HOST': env('DB_HOST'),
+            'PORT': env('DB_PORT'),
+        }
+    }
+else:
+    # Local development: SQLite file
+    DATABASES = {
+        'default': {
+            'ENGINE': 'django.db.backends.sqlite3',
+            'NAME': BASE_DIR / 'db.sqlite3',
+        }
+    }
+```
+
+### 3. `docs/source/conf.py` — boot Django in TESTING mode + neuter QuerySet repr
+
+```python
+import os
+import sys
+
+import django
+
+# Adjust this path to point at your Django package directory.
+sys.path.insert(0, os.path.abspath('../../<project_package>'))
+
+os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_package>.settings')
+
+# Load real .env if present (local dev). In CI there is none and that's fine.
+_repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
+_env_file = os.path.join(_repo_root, '.env')
+if os.path.exists(_env_file):
+    with open(_env_file) as _f:
+        for _line in _f:
+            _line = _line.strip()
+            if not _line or _line.startswith('#') or '=' not in _line:
+                continue
+            _key, _val = _line.split('=', 1)
+            os.environ.setdefault(_key.strip(), _val.strip())
+
+# Force TESTING mode so settings.py uses its safe dummy defaults and the
+# in-memory SQLite database. The docs build never serves traffic or touches
+# real data, so the production "fail loud on missing secret" contract does
+# not apply here.
+os.environ['TESTING'] = 'true'
+
+django.setup()
+
+# Sphinx 9 autodoc calls repr() on every class attribute it documents.
+# Django's QuerySet.__repr__ executes a SELECT against the database — which
+# documentation has no business doing. Intercept object_description so
+# QuerySet instances render as a static string instead.
+from django.db.models.query import QuerySet  # noqa: E402
+import sphinx.util.inspect as _sphinx_inspect  # noqa: E402
+
+_orig_object_description = _sphinx_inspect.object_description
+
+
+def _safe_object_description(obj, *args, **kwargs):
+    if isinstance(obj, QuerySet):
+        return f'<QuerySet [{obj.model.__name__}]>'
+    return _orig_object_description(obj, *args, **kwargs)
+
+
+_sphinx_inspect.object_description = _safe_object_description
+
+# ── Sphinx configuration below ────────────────────────────────────────────
+project = '<Project Name>'
+copyright = '<year>, <Project Team>'
+author = '<Project Team>'
+release = '1.0'
+
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.viewcode',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.intersphinx',
+    'sphinx_autodoc_typehints',
+    'sphinxcontrib.httpdomain',
+    'sphinxcontrib.mermaid',
+    'myst_parser',
+]
+
+source_suffix = {'.rst': 'restructuredtext', '.md': 'markdown'}
+
+myst_enable_extensions = ['colon_fence', 'deflist', 'tasklist', 'attrs_inline']
+myst_heading_anchors = 4
+
+autodoc_default_options = {
+    'members': True,
+    'member-order': 'bysource',
+    'special-members': '__init__',
+    'undoc-members': True,
+    'exclude-members': '__weakref__',
+}
+autodoc_inherit_docstrings = False
+napoleon_use_ivar = True
+
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']
+html_theme_options = {
+    'navigation_depth': 4,
+    'collapse_navigation': False,
+    'sticky_navigation': True,
+    'includehidden': True,
+    'titles_only': False,
+}
+```
+
+### 4. `.gitea/workflows/docs.yml` — build + failure-debug + deploy
+
+The failure-debug trio (`continue-on-error` + log dump + explicit fail) is **required** — without it, the Sphinx `ValueError` traceback in `/tmp/sphinx-err-*.log` is invisible in the Gitea UI and the build is effectively undiagnosable.
+
+```yaml
+name: Build & Deploy Docs
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - '<project_package>/**'
+      - 'docs/**'
+      - 'pyproject.toml'
+      - '.gitea/workflows/docs.yml'
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install package + docs deps
+        run: |
+          pip install --upgrade pip
+          pip install -e ".[docs]"
+
+      - name: Read version from pyproject.toml
+        id: version
+        run: |
+          VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
+          echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+
+      # ─── Failure-debug trio (REQUIRED) ─────────────────────────────────
+      - name: Build HTML
+        id: build_html
+        run: |
+          cd docs
+          ./regenerate_docs.sh
+        continue-on-error: true
+
+      - name: Print Sphinx error log on failure
+        if: steps.build_html.outcome == 'failure'
+        run: |
+          echo "=== Sphinx error log ==="
+          cat /tmp/sphinx-err-*.log 2>/dev/null || echo "(no sphinx error log found)"
+
+      - name: Fail if build failed
+        if: steps.build_html.outcome == 'failure'
+        run: exit 1
+      # ───────────────────────────────────────────────────────────────────
+
+      - name: Install rsync + openssh
+        run: |
+          apt-get update
+          apt-get install -y --no-install-recommends rsync openssh-client
+
+      - name: Configure SSH
+        run: |
+          mkdir -p ~/.ssh
+          printf '%s\n' "${{ secrets.DOCS_DEPLOY_KEY }}" > ~/.ssh/id_ed25519
+          chmod 600 ~/.ssh/id_ed25519
+          ssh-keyscan -p ${{ vars.DOCS_HOST_PORT }} ${{ vars.DOCS_HOST }} >> ~/.ssh/known_hosts
+
+      - name: Test SSH connectivity
+        run: |
+          ssh -o BatchMode=yes -o ConnectTimeout=10 \
+            -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 \
+            git@${{ vars.DOCS_HOST }} "id && echo 'SSH OK'"
+
+      - name: Rsync to versioned path
+        run: |
+          rsync -av --delete \
+            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
+            docs/_build/html/ \
+            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/${{ steps.version.outputs.version }}/
+
+      - name: Rsync to latest
+        run: |
+          rsync -av --delete \
+            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
+            docs/_build/html/ \
+            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/latest/
+
+      - name: Regenerate versions index
+        run: |
+          ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 git@${{ vars.DOCS_HOST }} \
+            'python3 - <<PY
+          import pathlib
+          root = pathlib.Path("/var/www/docs/<project_slug>")
+          versions = sorted(
+              (p.name for p in root.iterdir() if p.is_dir()),
+              reverse=True,
+          )
+          html = ["<!DOCTYPE html><html><head><title><Project> Docs</title></head><body>",
+                  "<h1><Project> Documentation</h1><ul>"]
+          for v in versions:
+              html.append(f"<li><a href=\"{v}/\">{v}</a></li>")
+          html.append("</ul></body></html>")
+          (root / "index.html").write_text("\n".join(html))
+          PY'
+```
+
+Required Gitea secrets/variables:
+
+- `secrets.DOCS_DEPLOY_KEY` — SSH private key authorised on the deploy host.
+- `vars.DOCS_HOST` — deploy host (e.g. `docs.example.com`).
+- `vars.DOCS_HOST_PORT` — SSH port (typically `22`).
+
+---
+
+## Standard Sphinx Extensions
+
+Use this exact extension set for consistency across projects:
+
+```python
+extensions = [
+    'sphinx.ext.autodoc',           # Pull docs from Python docstrings
+    'sphinx.ext.viewcode',          # "[source]" links to highlighted source
+    'sphinx.ext.napoleon',          # Google / NumPy style docstring support
+    'sphinx.ext.intersphinx',       # Cross-link to other projects' Sphinx docs
+    'sphinx_autodoc_typehints',     # Render PEP 484 type hints in docs
+    'sphinxcontrib.httpdomain',     # ".. http:get::" etc. for REST APIs
+    'sphinxcontrib.mermaid',        # Mermaid diagrams in Markdown / RST
+    'myst_parser',                  # Markdown source files alongside RST
+]
+```
+
+And the matching `pyproject.toml` extras group:
+
+```toml
+[project.optional-dependencies]
+docs = [
+    "sphinx",
+    "sphinx-rtd-theme",
+    "sphinx-autodoc-typehints",
+    "sphinx-autobuild",
+    "sphinxcontrib-httpdomain",
+    "sphinxcontrib-mermaid",
+    "myst-parser",
+]
+```
+
+---
+
+## Recommended Behaviours
+
+Behaviours that most projects should include but are not strictly required:
+
+- **Live rebuild during authoring** — `make livehtml` (via `sphinx-autobuild`) for hot-reload editing.
+- **One-shot regen script** — `docs/regenerate_docs.sh` runs `make clean`, `sphinx-apidoc` over every app, then `make html`. Drives both local development and the CI pipeline.
+- **Mermaid for diagrams** — text-based, diffable, lives in the `.md` / `.rst` source. Avoid binary diagram assets.
+- **Static images in `source/_static/`** — referenced with relative paths.
+- **Hand-written prose in Markdown (MyST)** alongside autogenerated reference docs in RST. The two coexist via `myst_parser` + `source_suffix`.
+- **Project root `CLAUDE.md` (or equivalent) names docs as the single source of truth** — discourage parallel READMEs that drift.
+
+---
+
+## Pattern Variant 1: DRF / QuerySet Autodoc Poisoning
+
+**Problem.** Sphinx 9 autodoc renders class attributes by calling `repr()` on the live object. Django's `QuerySet.__repr__` triggers `_fetch_all()`, which opens a database connection and runs a `SELECT`. For DRF viewsets like:
+
+```python
+class CurrencyViewSet(viewsets.ReadOnlyModelViewSet):
+    queryset = Currency.objects.filter(is_active=True)  # ← autodoc tries to execute this
+    serializer_class = CurrencySerializer
+```
+
+…the docs build crashes with `psycopg.OperationalError: failed to resolve host 'postgres'` (or whatever DB hostname is configured), even in TESTING mode where the in-memory SQLite has no tables.
+
+**Solution.** Monkey-patch `sphinx.util.inspect.object_description` in `conf.py` to short-circuit QuerySets before `repr()` is called:
+
+```python
+from django.db.models.query import QuerySet
+import sphinx.util.inspect as _sphinx_inspect
+
+_orig_object_description = _sphinx_inspect.object_description
+
+
+def _safe_object_description(obj, *args, **kwargs):
+    if isinstance(obj, QuerySet):
+        return f'<QuerySet [{obj.model.__name__}]>'
+    return _orig_object_description(obj, *args, **kwargs)
+
+
+_sphinx_inspect.object_description = _safe_object_description
+```
+
+This must run **after** `django.setup()` (so `QuerySet` can be imported) but **before** Sphinx starts processing documents.
+
+---
+
+## Pattern Variant 2: Settings-Driven TESTING Mode
+
+**Problem.** Docs build needs to import `settings.py` but has no real `.env` in CI. Production-mode `env('SECRET_KEY')` calls (no default) raise `ImproperlyConfigured` and the build crashes before Sphinx even starts.
+
+**Solution.** Read `TESTING` first in `settings.py`, then gate every required `env('X')` behind it:
+
+```python
+TESTING = env.bool('TESTING', default=False)
+if TESTING:
+    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
+else:
+    SECRET_KEY = env('SECRET_KEY')
+```
+
+`conf.py` flips the switch:
+
+```python
+os.environ['TESTING'] = 'true'
+django.setup()
+```
+
+**Bonus.** This also fixes a latent bug where `python manage.py test` would fail in any environment without `.env`. The same defaults that unblock the docs build now unblock the test suite — one mechanism, two payoffs.
+
+---
+
+## Pattern Variant 3: Gitea Actions Deploy Workflow
+
+The workflow has four logical phases:
+
+1. **Setup** — checkout, Python, `pip install -e ".[docs]"`, read version from `pyproject.toml`.
+2. **Build with failure visibility** — the three-step trio shown above. The `continue-on-error: true` on the build step plus `if: steps.build_html.outcome == 'failure'` on the log-dump and fail steps ensures the Sphinx traceback reaches the Gitea log even when the build crashes.
+3. **SSH setup** — write the deploy key to `~/.ssh/id_ed25519`, scan the host into `known_hosts`, verify connectivity.
+4. **Deploy** — rsync to `/var/www/docs/<project>/<version>/`, rsync to `…/latest/`, regenerate the versions index page on the remote host via a heredoc Python script.
+
+The deploy host is expected to serve `/var/www/docs/` over HTTPS via nginx or similar. Each pushed version gets its own directory; `latest/` is a copy of the most recent build. The versions index lists every directory alphabetically.
+
+---
+
+## Domain Extension Examples
+
+### Project without DRF / class-level QuerySets
+
+If your project has no `queryset = Model.objects.filter(...)` attributes at module load time, the `_safe_object_description` monkey-patch is unnecessary. You can omit it. The `TESTING=true` switch is still required because settings.py still has required env vars.
+
+### Project with extra required secrets
+
+Add each extra key to the TESTING branch in `settings.py`:
+
+```python
+if TESTING:
+    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
+    SLACK_TOKEN = env('SLACK_TOKEN', default='testing-insecure-slack')
+    STRIPE_API_KEY = env('STRIPE_API_KEY', default='testing-insecure-stripe')
+else:
+    SECRET_KEY = env('SECRET_KEY')
+    SLACK_TOKEN = env('SLACK_TOKEN')
+    STRIPE_API_KEY = env('STRIPE_API_KEY')
+```
+
+No changes needed to `conf.py` — the single `TESTING=true` flip covers them all.
+
+### Project on a non-Postgres database (MySQL, MariaDB)
+
+No special handling needed. The `if TESTING:` branch in `settings.py` switches to in-memory SQLite regardless of what production uses. The MySQL driver is never imported during a docs build.
+
+---
+
+## Anti-Patterns
+
+- ❌ **Don't load `.env.example` as a runtime fallback.** It's a documentation file with placeholder values like `DB_HOST=postgres` — those placeholders will poison the docs build by making `settings.py` believe Postgres is available.
+- ❌ **Don't override `settings.DATABASES` after `django.setup()`.** Django's `ConnectionHandler.databases` is a `@cached_property` populated during app loading; mutating `settings.DATABASES` afterwards has no effect.
+- ❌ **Don't add a separate `settings_docs.py`.** Env-var toggles are the project convention. A separate settings module fragments the config surface and forces every dev to remember which settings file applies in which context.
+- ❌ **Don't hand-edit `docs/source/reference/apps/`.** That tree is regenerated by `sphinx-apidoc` on every CI run. Hand-edits get overwritten.
+- ❌ **Don't suppress build errors in CI without dumping `/tmp/sphinx-err-*.log` first.** Sphinx writes its full traceback there and nowhere else; without the dump, the Gitea UI shows a one-line `ValueError` with no useful context.
+- ❌ **Don't use `os.environ.setdefault('TESTING', 'true')` in `conf.py`.** A user with `TESTING=false` in their local `.env` will see the setdefault skipped and hit production-mode behaviour during docs build. Use plain `os.environ['TESTING'] = 'true'` so it always wins.
+
+---
+
+## Settings
+
+Document the `TESTING` env var contract:
+
+```python
+# settings.py
+TESTING = env.bool('TESTING', default=False)
+# When true, gates safe-default branches for:
+#   - Required secrets (SECRET_KEY and any other env('X') with no default)
+#   - Required lists (ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS)
+#   - DATABASES → in-memory SQLite
+#   - CACHES → dummy backend
+#   - DRF throttling → disabled
+#   - MIGRATION_MODULES → disabled (no DB schema)
+#   - PASSWORD_HASHERS → fast hashers
+#   - LOGGING → minimal
+#
+# Set true for: pytest, manage.py test, docs build.
+# Set false (or unset) for: production, local dev with real services.
+```
+
+---
+
+## Testing
+
+Two verification recipes every project should run before pushing.
+
+### 1. Local build with real `.env`
+
+```bash
+cd docs
+make clean && make html
+```
+
+Expected: `build succeeded.` with zero warnings. Open `_build/html/index.html` to spot-check rendering.
+
+### 2. CI simulation (no `.env`)
+
+```bash
+mv .env .env.bak
+cd docs && make clean && make html
+cd .. && mv .env.bak .env
+```
+
+Expected: `build succeeded.` again. Settings.py uses TESTING-mode dummies; the in-memory SQLite has no tables but autodoc never queries it because the monkey-patch short-circuits QuerySet repr().
+
+### 3. Latent test-suite bug check
+
+```bash
+mv .env .env.bak
+python manage.py test --keepdb 2>&1 | head -5
+mv .env.bak .env
+```
+
+Expected: tests start running normally (not `ImproperlyConfigured: Set the SECRET_KEY environment variable`). This confirms the TESTING-mode defaults are wired into `settings.py` correctly — the docs build and the test suite share the same fallback mechanism.
+
+### 4. CI dry-run (Gitea Actions)
+
+Push to a feature branch. The workflow's failure-debug trio means any crash surfaces with a full traceback in the Gitea Actions log. Read the trace, fix the cause, push again.