mnemosyne/docs/Pattern_Sphinx-Documentation_V1-00.md

# Sphinx Documentation Pattern v1.0.0

Standardizes how Django projects build, configure, and deploy Sphinx documentation under a single `settings.py` — using the `TESTING` env-var flag to relax required-secret checks so docs build cleanly in CI without a real `.env`.

## 🐾 Red Panda Approval™

This pattern follows Red Panda Approval standards.

---

## Why a Pattern, Not a Shared Implementation

Every Django project has its own:

- **Required env vars** — one project needs `MCP_JWT_SECRET`, another needs `SLACK_TOKEN`, a third needs neither.
- **App layout** — `apps/` vs. top-level packages; some projects ship one app, others fifteen.
- **Autodoc-poisoning attributes** — DRF projects have class-level `queryset = Model.objects.filter(...)`; pure-Django projects may not.
- **Deploy target** — different hosts, ports, paths, and SSH key names per environment.

A shared library can't paper over those differences. Instead, this pattern defines:

- **Required interface** — the four files every project must have.
- **Recommended behaviours** — what most projects should include.
- **Extension guidelines** — what to add or skip per project.
- **Standard Sphinx extension set** — for consistency across projects.

---

## Required Interface

The non-negotiable minimum every Django project must provide.

### 1. `settings.py` — TESTING-gated safe defaults

Every required env var (those without a `default=`) must have a `TESTING`-mode fallback. Read `TESTING` **first**, then branch every required `env('X')` call:

```python
# Test mode flag — read first so it can relax required-env-var checks below.
TESTING = env.bool('TESTING', default=False)

DEBUG = env.bool('DEBUG', default=False)

# In TESTING mode (unit tests, docs build) required keys fall back to safe
# dummies so the settings module imports without a real .env. In production
# they remain required — missing values fail loud.
if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS', default=['testserver', 'localhost', '127.0.0.1'])
    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS', default=['http://localhost'])
    # ...any other required secrets get a 'testing-insecure-*' default here
else:
    SECRET_KEY = env('SECRET_KEY')
    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS')
    # ...and the production no-default form here
```

Rule: **every** required env var read in `settings.py` (anything that uses `env('X')` without `default=`) gets paired branches like above. Production fails loud on missing; TESTING falls back.

### 2. Database choice gated on `TESTING`

```python
if TESTING:
    # Test/docs build: in-memory SQLite. No real DB needed.
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': ':memory:',
        }
    }
elif env('APP_DB_NAME', default=None):
    # Production: PostgreSQL (or whatever the project uses)
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': env('APP_DB_NAME'),
            'USER': env('APP_DB_USER'),
            'PASSWORD': env('APP_DB_PASSWORD'),
            'HOST': env('DB_HOST'),
            'PORT': env('DB_PORT'),
        }
    }
else:
    # Local development: SQLite file
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': BASE_DIR / 'db.sqlite3',
        }
    }
```

### 3. `docs/source/conf.py` — boot Django in TESTING mode + neuter QuerySet repr

```python
import os
import sys

import django

# Adjust this path to point at your Django package directory.
sys.path.insert(0, os.path.abspath('../../<project_package>'))

os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_package>.settings')

# Load real .env if present (local dev). In CI there is none and that's fine.
_repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
_env_file = os.path.join(_repo_root, '.env')
if os.path.exists(_env_file):
    with open(_env_file) as _f:
        for _line in _f:
            _line = _line.strip()
            if not _line or _line.startswith('#') or '=' not in _line:
                continue
            _key, _val = _line.split('=', 1)
            os.environ.setdefault(_key.strip(), _val.strip())

# Force TESTING mode so settings.py uses its safe dummy defaults and the
# in-memory SQLite database. The docs build never serves traffic or touches
# real data, so the production "fail loud on missing secret" contract does
# not apply here.
os.environ['TESTING'] = 'true'

django.setup()

# Sphinx 9 autodoc calls repr() on every class attribute it documents.
# Django's QuerySet.__repr__ executes a SELECT against the database — which
# documentation has no business doing. Intercept object_description so
# QuerySet instances render as a static string instead.
from django.db.models.query import QuerySet  # noqa: E402
import sphinx.util.inspect as _sphinx_inspect  # noqa: E402

_orig_object_description = _sphinx_inspect.object_description


def _safe_object_description(obj, *args, **kwargs):
    if isinstance(obj, QuerySet):
        return f'<QuerySet [{obj.model.__name__}]>'
    return _orig_object_description(obj, *args, **kwargs)


_sphinx_inspect.object_description = _safe_object_description

# ── Sphinx configuration below ────────────────────────────────────────────
project = '<Project Name>'
copyright = '<year>, <Project Team>'
author = '<Project Team>'
release = '1.0'

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
    'sphinx.ext.napoleon',
    'sphinx.ext.intersphinx',
    'sphinx_autodoc_typehints',
    'sphinxcontrib.httpdomain',
    'sphinxcontrib.mermaid',
    'myst_parser',
]

source_suffix = {'.rst': 'restructuredtext', '.md': 'markdown'}

myst_enable_extensions = ['colon_fence', 'deflist', 'tasklist', 'attrs_inline']
myst_heading_anchors = 4

autodoc_default_options = {
    'members': True,
    'member-order': 'bysource',
    'special-members': '__init__',
    'undoc-members': True,
    'exclude-members': '__weakref__',
}
autodoc_inherit_docstrings = False
napoleon_use_ivar = True

html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']
html_theme_options = {
    'navigation_depth': 4,
    'collapse_navigation': False,
    'sticky_navigation': True,
    'includehidden': True,
    'titles_only': False,
}
```

### 4. `.gitea/workflows/docs.yml` — build + failure-debug + deploy

The failure-debug trio (`continue-on-error` + log dump + explicit fail) is **required** — without it, the Sphinx `ValueError` traceback in `/tmp/sphinx-err-*.log` is invisible in the Gitea UI and the build is effectively undiagnosable.

```yaml
name: Build & Deploy Docs

on:
  push:
    branches: [main]
    paths:
      - '<project_package>/**'
      - 'docs/**'
      - 'pyproject.toml'
      - '.gitea/workflows/docs.yml'

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install package + docs deps
        run: |
          pip install --upgrade pip
          pip install -e ".[docs]"

      - name: Read version from pyproject.toml
        id: version
        run: |
          VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
          echo "version=$VERSION" >> "$GITHUB_OUTPUT"

      # ─── Failure-debug trio (REQUIRED) ─────────────────────────────────
      - name: Build HTML
        id: build_html
        run: |
          cd docs
          ./regenerate_docs.sh
        continue-on-error: true

      - name: Print Sphinx error log on failure
        if: steps.build_html.outcome == 'failure'
        run: |
          echo "=== Sphinx error log ==="
          cat /tmp/sphinx-err-*.log 2>/dev/null || echo "(no sphinx error log found)"

      - name: Fail if build failed
        if: steps.build_html.outcome == 'failure'
        run: exit 1
      # ───────────────────────────────────────────────────────────────────

      - name: Install rsync + openssh
        run: |
          apt-get update
          apt-get install -y --no-install-recommends rsync openssh-client

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh
          printf '%s\n' "${{ secrets.DOCS_DEPLOY_KEY }}" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          ssh-keyscan -p ${{ vars.DOCS_HOST_PORT }} ${{ vars.DOCS_HOST }} >> ~/.ssh/known_hosts

      - name: Test SSH connectivity
        run: |
          ssh -o BatchMode=yes -o ConnectTimeout=10 \
            -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 \
            git@${{ vars.DOCS_HOST }} "id && echo 'SSH OK'"

      - name: Rsync to versioned path
        run: |
          rsync -av --delete \
            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
            docs/_build/html/ \
            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/${{ steps.version.outputs.version }}/

      - name: Rsync to latest
        run: |
          rsync -av --delete \
            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
            docs/_build/html/ \
            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/latest/

      - name: Regenerate versions index
        run: |
          ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 git@${{ vars.DOCS_HOST }} \
            'python3 - <<PY
          import pathlib
          root = pathlib.Path("/var/www/docs/<project_slug>")
          versions = sorted(
              (p.name for p in root.iterdir() if p.is_dir()),
              reverse=True,
          )
          html = ["<!DOCTYPE html><html><head><title><Project> Docs</title></head><body>",
                  "<h1><Project> Documentation</h1><ul>"]
          for v in versions:
              html.append(f"<li><a href=\"{v}/\">{v}</a></li>")
          html.append("</ul></body></html>")
          (root / "index.html").write_text("\n".join(html))
          PY'
```

Required Gitea secrets/variables:

- `secrets.DOCS_DEPLOY_KEY` — SSH private key authorised on the deploy host.
- `vars.DOCS_HOST` — deploy host (e.g. `docs.example.com`).
- `vars.DOCS_HOST_PORT` — SSH port (typically `22`).

---

## Standard Sphinx Extensions

Use this exact extension set for consistency across projects:

```python
extensions = [
    'sphinx.ext.autodoc',           # Pull docs from Python docstrings
    'sphinx.ext.viewcode',          # "[source]" links to highlighted source
    'sphinx.ext.napoleon',          # Google / NumPy style docstring support
    'sphinx.ext.intersphinx',       # Cross-link to other projects' Sphinx docs
    'sphinx_autodoc_typehints',     # Render PEP 484 type hints in docs
    'sphinxcontrib.httpdomain',     # ".. http:get::" etc. for REST APIs
    'sphinxcontrib.mermaid',        # Mermaid diagrams in Markdown / RST
    'myst_parser',                  # Markdown source files alongside RST
]
```

And the matching `pyproject.toml` extras group:

```toml
[project.optional-dependencies]
docs = [
    "sphinx",
    "sphinx-rtd-theme",
    "sphinx-autodoc-typehints",
    "sphinx-autobuild",
    "sphinxcontrib-httpdomain",
    "sphinxcontrib-mermaid",
    "myst-parser",
]
```

---

## Recommended Behaviours

Behaviours that most projects should include but are not strictly required:

- **Live rebuild during authoring** — `make livehtml` (via `sphinx-autobuild`) for hot-reload editing.
- **One-shot regen script** — `docs/regenerate_docs.sh` runs `make clean`, `sphinx-apidoc` over every app, then `make html`. Drives both local development and the CI pipeline.
- **Mermaid for diagrams** — text-based, diffable, lives in the `.md` / `.rst` source. Avoid binary diagram assets.
- **Static images in `source/_static/`** — referenced with relative paths.
- **Hand-written prose in Markdown (MyST)** alongside autogenerated reference docs in RST. The two coexist via `myst_parser` + `source_suffix`.
- **Project root `CLAUDE.md` (or equivalent) names docs as the single source of truth** — discourage parallel READMEs that drift.

---

## Pattern Variant 1: DRF / QuerySet Autodoc Poisoning

**Problem.** Sphinx 9 autodoc renders class attributes by calling `repr()` on the live object. Django's `QuerySet.__repr__` triggers `_fetch_all()`, which opens a database connection and runs a `SELECT`. For DRF viewsets like:

```python
class CurrencyViewSet(viewsets.ReadOnlyModelViewSet):
    queryset = Currency.objects.filter(is_active=True)  # ← autodoc tries to execute this
    serializer_class = CurrencySerializer
```

…the docs build crashes with `psycopg.OperationalError: failed to resolve host 'postgres'` (or whatever DB hostname is configured), even in TESTING mode where the in-memory SQLite has no tables.

**Solution.** Monkey-patch `sphinx.util.inspect.object_description` in `conf.py` to short-circuit QuerySets before `repr()` is called:

```python
from django.db.models.query import QuerySet
import sphinx.util.inspect as _sphinx_inspect

_orig_object_description = _sphinx_inspect.object_description


def _safe_object_description(obj, *args, **kwargs):
    if isinstance(obj, QuerySet):
        return f'<QuerySet [{obj.model.__name__}]>'
    return _orig_object_description(obj, *args, **kwargs)


_sphinx_inspect.object_description = _safe_object_description
```

This must run **after** `django.setup()` (so `QuerySet` can be imported) but **before** Sphinx starts processing documents.

---

## Pattern Variant 2: Settings-Driven TESTING Mode

**Problem.** Docs build needs to import `settings.py` but has no real `.env` in CI. Production-mode `env('SECRET_KEY')` calls (no default) raise `ImproperlyConfigured` and the build crashes before Sphinx even starts.

**Solution.** Read `TESTING` first in `settings.py`, then gate every required `env('X')` behind it:

```python
TESTING = env.bool('TESTING', default=False)
if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
else:
    SECRET_KEY = env('SECRET_KEY')
```

`conf.py` flips the switch:

```python
os.environ['TESTING'] = 'true'
django.setup()
```

**Bonus.** This also fixes a latent bug where `python manage.py test` would fail in any environment without `.env`. The same defaults that unblock the docs build now unblock the test suite — one mechanism, two payoffs.

---

## Pattern Variant 3: Gitea Actions Deploy Workflow

The workflow has four logical phases:

1. **Setup** — checkout, Python, `pip install -e ".[docs]"`, read version from `pyproject.toml`.
2. **Build with failure visibility** — the three-step trio shown above. The `continue-on-error: true` on the build step plus `if: steps.build_html.outcome == 'failure'` on the log-dump and fail steps ensures the Sphinx traceback reaches the Gitea log even when the build crashes.
3. **SSH setup** — write the deploy key to `~/.ssh/id_ed25519`, scan the host into `known_hosts`, verify connectivity.
4. **Deploy** — rsync to `/var/www/docs/<project>/<version>/`, rsync to `…/latest/`, regenerate the versions index page on the remote host via a heredoc Python script.

The deploy host is expected to serve `/var/www/docs/` over HTTPS via nginx or similar. Each pushed version gets its own directory; `latest/` is a copy of the most recent build. The versions index lists every directory alphabetically.

---

## Domain Extension Examples

### Project without DRF / class-level QuerySets

If your project has no `queryset = Model.objects.filter(...)` attributes at module load time, the `_safe_object_description` monkey-patch is unnecessary. You can omit it. The `TESTING=true` switch is still required because settings.py still has required env vars.

### Project with extra required secrets

Add each extra key to the TESTING branch in `settings.py`:

```python
if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
    SLACK_TOKEN = env('SLACK_TOKEN', default='testing-insecure-slack')
    STRIPE_API_KEY = env('STRIPE_API_KEY', default='testing-insecure-stripe')
else:
    SECRET_KEY = env('SECRET_KEY')
    SLACK_TOKEN = env('SLACK_TOKEN')
    STRIPE_API_KEY = env('STRIPE_API_KEY')
```

No changes needed to `conf.py` — the single `TESTING=true` flip covers them all.

### Project on a non-Postgres database (MySQL, MariaDB)

No special handling needed. The `if TESTING:` branch in `settings.py` switches to in-memory SQLite regardless of what production uses. The MySQL driver is never imported during a docs build.

---

## Anti-Patterns

- ❌ **Don't load `.env.example` as a runtime fallback.** It's a documentation file with placeholder values like `DB_HOST=postgres` — those placeholders will poison the docs build by making `settings.py` believe Postgres is available.
- ❌ **Don't override `settings.DATABASES` after `django.setup()`.** Django's `ConnectionHandler.databases` is a `@cached_property` populated during app loading; mutating `settings.DATABASES` afterwards has no effect.
- ❌ **Don't add a separate `settings_docs.py`.** Env-var toggles are the project convention. A separate settings module fragments the config surface and forces every dev to remember which settings file applies in which context.
- ❌ **Don't hand-edit `docs/source/reference/apps/`.** That tree is regenerated by `sphinx-apidoc` on every CI run. Hand-edits get overwritten.
- ❌ **Don't suppress build errors in CI without dumping `/tmp/sphinx-err-*.log` first.** Sphinx writes its full traceback there and nowhere else; without the dump, the Gitea UI shows a one-line `ValueError` with no useful context.
- ❌ **Don't use `os.environ.setdefault('TESTING', 'true')` in `conf.py`.** A user with `TESTING=false` in their local `.env` will see the setdefault skipped and hit production-mode behaviour during docs build. Use plain `os.environ['TESTING'] = 'true'` so it always wins.

---

## Settings

Document the `TESTING` env var contract:

```python
# settings.py
TESTING = env.bool('TESTING', default=False)
# When true, gates safe-default branches for:
#   - Required secrets (SECRET_KEY and any other env('X') with no default)
#   - Required lists (ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS)
#   - DATABASES → in-memory SQLite
#   - CACHES → dummy backend
#   - DRF throttling → disabled
#   - MIGRATION_MODULES → disabled (no DB schema)
#   - PASSWORD_HASHERS → fast hashers
#   - LOGGING → minimal
#
# Set true for: pytest, manage.py test, docs build.
# Set false (or unset) for: production, local dev with real services.
```

---

## Testing

Two verification recipes every project should run before pushing.

### 1. Local build with real `.env`

```bash
cd docs
make clean && make html
```

Expected: `build succeeded.` with zero warnings. Open `_build/html/index.html` to spot-check rendering.

### 2. CI simulation (no `.env`)

```bash
mv .env .env.bak
cd docs && make clean && make html
cd .. && mv .env.bak .env
```

Expected: `build succeeded.` again. Settings.py uses TESTING-mode dummies; the in-memory SQLite has no tables but autodoc never queries it because the monkey-patch short-circuits QuerySet repr().

### 3. Latent test-suite bug check

```bash
mv .env .env.bak
python manage.py test --keepdb 2>&1 | head -5
mv .env.bak .env
```

Expected: tests start running normally (not `ImproperlyConfigured: Set the SECRET_KEY environment variable`). This confirms the TESTING-mode defaults are wired into `settings.py` correctly — the docs build and the test suite share the same fallback mechanism.

### 4. CI dry-run (Gitea Actions)

Push to a feature branch. The workflow's failure-debug trio means any crash surfaces with a full traceback in the Gitea Actions log. Read the trace, fix the cause, push again.