mnemosyne/docs/Pattern_Sphinx-Documentation_V1-00.md

Sphinx Documentation Pattern v1.0.0

Standardizes how Django projects build, configure, and deploy Sphinx documentation under a single settings.py — using the TESTING env-var flag to relax required-secret checks so docs build cleanly in CI without a real .env.

🐾 Red Panda Approval™

This pattern follows Red Panda Approval standards.

---

Why a Pattern, Not a Shared Implementation

Every Django project has its own:

• Required env vars — one project needs MCP_JWT_SECRET, another needs SLACK_TOKEN, a third needs neither.
• App layout — apps/ vs. top-level packages; some projects ship one app, others fifteen.
• Autodoc-poisoning attributes — DRF projects have class-level queryset = Model.objects.filter(...); pure-Django projects may not.
• Deploy target — different hosts, ports, paths, and SSH key names per environment.

A shared library can't paper over those differences. Instead, this pattern defines:

• Required interface — the four files every project must have.
• Recommended behaviours — what most projects should include.
• Extension guidelines — what to add or skip per project.
• Standard Sphinx extension set — for consistency across projects.

---

Required Interface

The non-negotiable minimum every Django project must provide.

1. settings.py — TESTING-gated safe defaults

Every required env var (those without a default=) must have a TESTING-mode fallback. Read TESTING first, then branch every required env('X') call:

# Test mode flag — read first so it can relax required-env-var checks below.
TESTING = env.bool('TESTING', default=False)

DEBUG = env.bool('DEBUG', default=False)

# In TESTING mode (unit tests, docs build) required keys fall back to safe
# dummies so the settings module imports without a real .env. In production
# they remain required — missing values fail loud.
if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS', default=['testserver', 'localhost', '127.0.0.1'])
    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS', default=['http://localhost'])
    # ...any other required secrets get a 'testing-insecure-*' default here
else:
    SECRET_KEY = env('SECRET_KEY')
    ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
    CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS')
    # ...and the production no-default form here

Rule: every required env var read in settings.py (anything that uses env('X') without default=) gets paired branches like above. Production fails loud on missing; TESTING falls back.

2. Database choice gated on TESTING

if TESTING:
    # Test/docs build: in-memory SQLite. No real DB needed.
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': ':memory:',
        }
    }
elif env('APP_DB_NAME', default=None):
    # Production: PostgreSQL (or whatever the project uses)
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': env('APP_DB_NAME'),
            'USER': env('APP_DB_USER'),
            'PASSWORD': env('APP_DB_PASSWORD'),
            'HOST': env('DB_HOST'),
            'PORT': env('DB_PORT'),
        }
    }
else:
    # Local development: SQLite file
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': BASE_DIR / 'db.sqlite3',
        }
    }

3. docs/source/conf.py — boot Django in TESTING mode + neuter QuerySet repr

import os
import sys

import django

# Adjust this path to point at your Django package directory.
sys.path.insert(0, os.path.abspath('../../<project_package>'))

os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_package>.settings')

# Load real .env if present (local dev). In CI there is none and that's fine.
_repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
_env_file = os.path.join(_repo_root, '.env')
if os.path.exists(_env_file):
    with open(_env_file) as _f:
        for _line in _f:
            _line = _line.strip()
            if not _line or _line.startswith('#') or '=' not in _line:
                continue
            _key, _val = _line.split('=', 1)
            os.environ.setdefault(_key.strip(), _val.strip())

# Force TESTING mode so settings.py uses its safe dummy defaults and the
# in-memory SQLite database. The docs build never serves traffic or touches
# real data, so the production "fail loud on missing secret" contract does
# not apply here.
os.environ['TESTING'] = 'true'

django.setup()

# Sphinx 9 autodoc calls repr() on every class attribute it documents.
# Django's QuerySet.__repr__ executes a SELECT against the database — which
# documentation has no business doing. Intercept object_description so
# QuerySet instances render as a static string instead.
from django.db.models.query import QuerySet  # noqa: E402
import sphinx.util.inspect as _sphinx_inspect  # noqa: E402

_orig_object_description = _sphinx_inspect.object_description


def _safe_object_description(obj, *args, **kwargs):
    if isinstance(obj, QuerySet):
        return f'<QuerySet [{obj.model.__name__}]>'
    return _orig_object_description(obj, *args, **kwargs)


_sphinx_inspect.object_description = _safe_object_description

# ── Sphinx configuration below ────────────────────────────────────────────
project = '<Project Name>'
copyright = '<year>, <Project Team>'
author = '<Project Team>'
release = '1.0'

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
    'sphinx.ext.napoleon',
    'sphinx.ext.intersphinx',
    'sphinx_autodoc_typehints',
    'sphinxcontrib.httpdomain',
    'sphinxcontrib.mermaid',
    'myst_parser',
]

source_suffix = {'.rst': 'restructuredtext', '.md': 'markdown'}

myst_enable_extensions = ['colon_fence', 'deflist', 'tasklist', 'attrs_inline']
myst_heading_anchors = 4

autodoc_default_options = {
    'members': True,
    'member-order': 'bysource',
    'special-members': '__init__',
    'undoc-members': True,
    'exclude-members': '__weakref__',
}
autodoc_inherit_docstrings = False
napoleon_use_ivar = True

html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']
html_theme_options = {
    'navigation_depth': 4,
    'collapse_navigation': False,
    'sticky_navigation': True,
    'includehidden': True,
    'titles_only': False,
}

4. .gitea/workflows/docs.yml — build + failure-debug + deploy

The failure-debug trio (continue-on-error + log dump + explicit fail) is required — without it, the Sphinx ValueError traceback in /tmp/sphinx-err-*.log is invisible in the Gitea UI and the build is effectively undiagnosable.

name: Build & Deploy Docs

on:
  push:
    branches: [main]
    paths:
      - '<project_package>/**'
      - 'docs/**'
      - 'pyproject.toml'
      - '.gitea/workflows/docs.yml'

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install package + docs deps
        run: |
          pip install --upgrade pip
          pip install -e ".[docs]"

      - name: Read version from pyproject.toml
        id: version
        run: |
          VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
          echo "version=$VERSION" >> "$GITHUB_OUTPUT"

      # ─── Failure-debug trio (REQUIRED) ─────────────────────────────────
      - name: Build HTML
        id: build_html
        run: |
          cd docs
          ./regenerate_docs.sh
        continue-on-error: true

      - name: Print Sphinx error log on failure
        if: steps.build_html.outcome == 'failure'
        run: |
          echo "=== Sphinx error log ==="
          cat /tmp/sphinx-err-*.log 2>/dev/null || echo "(no sphinx error log found)"

      - name: Fail if build failed
        if: steps.build_html.outcome == 'failure'
        run: exit 1
      # ───────────────────────────────────────────────────────────────────

      - name: Install rsync + openssh
        run: |
          apt-get update
          apt-get install -y --no-install-recommends rsync openssh-client

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh
          printf '%s\n' "${{ secrets.DOCS_DEPLOY_KEY }}" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          ssh-keyscan -p ${{ vars.DOCS_HOST_PORT }} ${{ vars.DOCS_HOST }} >> ~/.ssh/known_hosts

      - name: Test SSH connectivity
        run: |
          ssh -o BatchMode=yes -o ConnectTimeout=10 \
            -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 \
            git@${{ vars.DOCS_HOST }} "id && echo 'SSH OK'"

      - name: Rsync to versioned path
        run: |
          rsync -av --delete \
            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
            docs/_build/html/ \
            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/${{ steps.version.outputs.version }}/

      - name: Rsync to latest
        run: |
          rsync -av --delete \
            -e "ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519" \
            docs/_build/html/ \
            git@${{ vars.DOCS_HOST }}:/var/www/docs/<project_slug>/latest/

      - name: Regenerate versions index
        run: |
          ssh -p ${{ vars.DOCS_HOST_PORT }} -i ~/.ssh/id_ed25519 git@${{ vars.DOCS_HOST }} \
            'python3 - <<PY
          import pathlib
          root = pathlib.Path("/var/www/docs/<project_slug>")
          versions = sorted(
              (p.name for p in root.iterdir() if p.is_dir()),
              reverse=True,
          )
          html = ["<!DOCTYPE html><html><head><title><Project> Docs</title></head><body>",
                  "<h1><Project> Documentation</h1><ul>"]
          for v in versions:
              html.append(f"<li><a href=\"{v}/\">{v}</a></li>")
          html.append("</ul></body></html>")
          (root / "index.html").write_text("\n".join(html))
          PY'

Required Gitea secrets/variables:

• secrets.DOCS_DEPLOY_KEY — SSH private key authorised on the deploy host.
• vars.DOCS_HOST — deploy host (e.g. docs.example.com).
• vars.DOCS_HOST_PORT — SSH port (typically 22).

---

Standard Sphinx Extensions

Use this exact extension set for consistency across projects:

extensions = [
    'sphinx.ext.autodoc',           # Pull docs from Python docstrings
    'sphinx.ext.viewcode',          # "[source]" links to highlighted source
    'sphinx.ext.napoleon',          # Google / NumPy style docstring support
    'sphinx.ext.intersphinx',       # Cross-link to other projects' Sphinx docs
    'sphinx_autodoc_typehints',     # Render PEP 484 type hints in docs
    'sphinxcontrib.httpdomain',     # ".. http:get::" etc. for REST APIs
    'sphinxcontrib.mermaid',        # Mermaid diagrams in Markdown / RST
    'myst_parser',                  # Markdown source files alongside RST
]

And the matching pyproject.toml extras group:

[project.optional-dependencies]
docs = [
    "sphinx",
    "sphinx-rtd-theme",
    "sphinx-autodoc-typehints",
    "sphinx-autobuild",
    "sphinxcontrib-httpdomain",
    "sphinxcontrib-mermaid",
    "myst-parser",
]

---

Recommended Behaviours

Behaviours that most projects should include but are not strictly required:

• Live rebuild during authoring — make livehtml (via sphinx-autobuild) for hot-reload editing.
• One-shot regen script — docs/regenerate_docs.sh runs make clean, sphinx-apidoc over every app, then make html. Drives both local development and the CI pipeline.
• Mermaid for diagrams — text-based, diffable, lives in the .md / .rst source. Avoid binary diagram assets.
• Static images in source/_static/ — referenced with relative paths.
• Hand-written prose in Markdown (MyST) alongside autogenerated reference docs in RST. The two coexist via myst_parser + source_suffix.
• Project root CLAUDE.md (or equivalent) names docs as the single source of truth — discourage parallel READMEs that drift.

---

Pattern Variant 1: DRF / QuerySet Autodoc Poisoning

Problem. Sphinx 9 autodoc renders class attributes by calling repr() on the live object. Django's QuerySet.__repr__ triggers _fetch_all(), which opens a database connection and runs a SELECT. For DRF viewsets like:

class CurrencyViewSet(viewsets.ReadOnlyModelViewSet):
    queryset = Currency.objects.filter(is_active=True)  # ← autodoc tries to execute this
    serializer_class = CurrencySerializer

…the docs build crashes with psycopg.OperationalError: failed to resolve host 'postgres' (or whatever DB hostname is configured), even in TESTING mode where the in-memory SQLite has no tables.

Solution. Monkey-patch sphinx.util.inspect.object_description in conf.py to short-circuit QuerySets before repr() is called:

from django.db.models.query import QuerySet
import sphinx.util.inspect as _sphinx_inspect

_orig_object_description = _sphinx_inspect.object_description


def _safe_object_description(obj, *args, **kwargs):
    if isinstance(obj, QuerySet):
        return f'<QuerySet [{obj.model.__name__}]>'
    return _orig_object_description(obj, *args, **kwargs)


_sphinx_inspect.object_description = _safe_object_description

This must run after django.setup() (so QuerySet can be imported) but before Sphinx starts processing documents.

---

Pattern Variant 2: Settings-Driven TESTING Mode

Problem. Docs build needs to import settings.py but has no real .env in CI. Production-mode env('SECRET_KEY') calls (no default) raise ImproperlyConfigured and the build crashes before Sphinx even starts.

Solution. Read TESTING first in settings.py, then gate every required env('X') behind it:

TESTING = env.bool('TESTING', default=False)
if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
else:
    SECRET_KEY = env('SECRET_KEY')

conf.py flips the switch:

os.environ['TESTING'] = 'true'
django.setup()

Bonus. This also fixes a latent bug where python manage.py test would fail in any environment without .env. The same defaults that unblock the docs build now unblock the test suite — one mechanism, two payoffs.

---

Pattern Variant 3: Gitea Actions Deploy Workflow

The workflow has four logical phases:

1. Setup — checkout, Python, pip install -e ".[docs]", read version from pyproject.toml.
2. Build with failure visibility — the three-step trio shown above. The continue-on-error: true on the build step plus if: steps.build_html.outcome == 'failure' on the log-dump and fail steps ensures the Sphinx traceback reaches the Gitea log even when the build crashes.
3. SSH setup — write the deploy key to ~/.ssh/id_ed25519, scan the host into known_hosts, verify connectivity.
4. Deploy — rsync to /var/www/docs/<project>/<version>/, rsync to …/latest/, regenerate the versions index page on the remote host via a heredoc Python script.

The deploy host is expected to serve /var/www/docs/ over HTTPS via nginx or similar. Each pushed version gets its own directory; latest/ is a copy of the most recent build. The versions index lists every directory alphabetically.

---

Domain Extension Examples

Project without DRF / class-level QuerySets

If your project has no queryset = Model.objects.filter(...) attributes at module load time, the _safe_object_description monkey-patch is unnecessary. You can omit it. The TESTING=true switch is still required because settings.py still has required env vars.

Project with extra required secrets

Add each extra key to the TESTING branch in settings.py:

if TESTING:
    SECRET_KEY = env('SECRET_KEY', default='testing-insecure-key')
    SLACK_TOKEN = env('SLACK_TOKEN', default='testing-insecure-slack')
    STRIPE_API_KEY = env('STRIPE_API_KEY', default='testing-insecure-stripe')
else:
    SECRET_KEY = env('SECRET_KEY')
    SLACK_TOKEN = env('SLACK_TOKEN')
    STRIPE_API_KEY = env('STRIPE_API_KEY')

No changes needed to conf.py — the single TESTING=true flip covers them all.

Project on a non-Postgres database (MySQL, MariaDB)

No special handling needed. The if TESTING: branch in settings.py switches to in-memory SQLite regardless of what production uses. The MySQL driver is never imported during a docs build.

---

Anti-Patterns

• ❌ Don't load .env.example as a runtime fallback. It's a documentation file with placeholder values like DB_HOST=postgres — those placeholders will poison the docs build by making settings.py believe Postgres is available.
• ❌ Don't override settings.DATABASES after django.setup(). Django's ConnectionHandler.databases is a @cached_property populated during app loading; mutating settings.DATABASES afterwards has no effect.
• ❌ Don't add a separate settings_docs.py. Env-var toggles are the project convention. A separate settings module fragments the config surface and forces every dev to remember which settings file applies in which context.
• ❌ Don't hand-edit docs/source/reference/apps/. That tree is regenerated by sphinx-apidoc on every CI run. Hand-edits get overwritten.
• ❌ Don't suppress build errors in CI without dumping /tmp/sphinx-err-*.log first. Sphinx writes its full traceback there and nowhere else; without the dump, the Gitea UI shows a one-line ValueError with no useful context.
• ❌ Don't use os.environ.setdefault('TESTING', 'true') in conf.py. A user with TESTING=false in their local .env will see the setdefault skipped and hit production-mode behaviour during docs build. Use plain os.environ['TESTING'] = 'true' so it always wins.

---

Settings

Document the TESTING env var contract:

# settings.py
TESTING = env.bool('TESTING', default=False)
# When true, gates safe-default branches for:
#   - Required secrets (SECRET_KEY and any other env('X') with no default)
#   - Required lists (ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS)
#   - DATABASES → in-memory SQLite
#   - CACHES → dummy backend
#   - DRF throttling → disabled
#   - MIGRATION_MODULES → disabled (no DB schema)
#   - PASSWORD_HASHERS → fast hashers
#   - LOGGING → minimal
#
# Set true for: pytest, manage.py test, docs build.
# Set false (or unset) for: production, local dev with real services.

---

Testing

Two verification recipes every project should run before pushing.

1. Local build with real .env

cd docs
make clean && make html

Expected: build succeeded. with zero warnings. Open _build/html/index.html to spot-check rendering.

2. CI simulation (no .env)

mv .env .env.bak
cd docs && make clean && make html
cd .. && mv .env.bak .env

Expected: build succeeded. again. Settings.py uses TESTING-mode dummies; the in-memory SQLite has no tables but autodoc never queries it because the monkey-patch short-circuits QuerySet repr().

3. Latent test-suite bug check

mv .env .env.bak
python manage.py test --keepdb 2>&1 | head -5
mv .env.bak .env

Expected: tests start running normally (not ImproperlyConfigured: Set the SECRET_KEY environment variable). This confirms the TESTING-mode defaults are wired into settings.py correctly — the docs build and the test suite share the same fallback mechanism.

4. CI dry-run (Gitea Actions)

Push to a feature branch. The workflow's failure-debug trio means any crash surfaces with a full traceback in the Gitea Actions log. Read the trace, fix the cause, push again.