diff --git a/README.md b/README.md index 8168bc0..be05745 100644 --- a/README.md +++ b/README.md @@ -76,10 +76,23 @@ Hosts in the Ouranos lab: ```bash cd mnemosyne/ python manage.py migrate # Apply Django ORM migrations -python manage.py setup_neo4j_indexes # Create Neo4j vector + full-text indexes python manage.py load_library_types # Load LIBRARY_TYPE_DEFAULTS into Neo4j +# --- seed the system embedding model in /admin/llm_manager/llmmodel/ here --- +python manage.py setup_neo4j_indexes # Create Neo4j vector + full-text indexes ``` +> **Seed the embedding model before running `setup_neo4j_indexes`.** Vector +> index dimensions are read from the row in ``llm_manager_llmmodel`` that +> has ``is_system_embedding_model=True`` and a non-null ``vector_dimensions``. +> There is deliberately no hardcoded fallback: an index built at the wrong +> dimension silently breaks every search. The command will exit non-zero +> with a clear error if no such row exists, which is also why the +> ``docker compose`` ``init`` sidecar treats vector-index creation as +> best-effort on first boot — the stack starts healthy, migrations and +> library-type seed data land, and you run +> ``docker compose exec app python manage.py setup_neo4j_indexes`` once +> the embedding-model row is in place. + ### Start the web app The Django REST API serves `/library/api/*` (libraries, collections, items, search, workspaces, ingest) and Django admin. Use Gunicorn in production; `runserver` for dev. @@ -199,14 +212,16 @@ cp .env.example .env && $EDITOR .env # Pull the image (or build locally with `docker compose build`) docker compose pull -# DB migrations (one-shot) -docker compose run --rm app migrate - -# Neo4j indexes + library_type defaults (one-shot) -docker compose run --rm app setup - -# Bring the stack up +# Bring the stack up — the `init` sidecar runs migrations + library_type +# defaults automatically. Vector indexes are deferred until you seed the +# system embedding model (see below) — the sidecar logs a clear notice +# and exits 0 either way, so the stack comes up healthy on first boot. docker compose up -d + +# Seed the system embedding model at /admin/llm_manager/llmmodel/ +# (mark one row `is_system_embedding_model=True` with `vector_dimensions` +# set to whatever your embedding provider returns), then: +docker compose exec app python manage.py setup_neo4j_indexes ``` ### Day-to-day diff --git a/mnemosyne/docker/entrypoint.sh b/mnemosyne/docker/entrypoint.sh new file mode 100644 index 0000000..c8fc456 --- /dev/null +++ b/mnemosyne/docker/entrypoint.sh @@ -0,0 +1,111 @@ +#!/bin/sh +# Mnemosyne container entrypoint. +# +# The same image runs all three processes — the compose service supplies +# `web`, `mcp`, `worker`, or `migrate` as CMD. + +set -e + +case "$1" in + web) + # Django REST API + admin (gunicorn → wsgi). + exec gunicorn \ + --config /app/docker/gunicorn.conf.py \ + --bind 0.0.0.0:8000 \ + --workers "${GUNICORN_WORKERS:-3}" \ + --access-logfile - \ + --error-logfile - \ + mnemosyne.wsgi:application + ;; + + mcp) + # FastMCP over Streamable HTTP at /mcp/, mounted by mnemosyne.asgi. + exec uvicorn \ + --host 0.0.0.0 \ + --port 8001 \ + --workers "${UVICORN_WORKERS:-1}" \ + mnemosyne.asgi:app + ;; + + worker) + # Celery worker covering embedding + ingest + batch + default queues. + # In production you may want to split these onto separate worker + # services for queue-level isolation; one process is fine to start. + exec celery -A mnemosyne worker \ + --loglevel="${CELERY_LOG_LEVEL:-info}" \ + --queues="${CELERY_QUEUES:-celery,embedding,batch}" \ + --concurrency="${CELERY_CONCURRENCY:-2}" + ;; + + beat) + # Celery scheduled tasks (only needed if/when periodic jobs are wired). + exec celery -A mnemosyne beat \ + --loglevel="${CELERY_LOG_LEVEL:-info}" + ;; + + migrate) + # One-shot DB migration runner — invoke before bringing services up + # for the first time or after a deploy. + exec python manage.py migrate --noinput + ;; + + setup) + # One-shot init — Neo4j indexes + library_type seed data. + python manage.py setup_neo4j_indexes + python manage.py load_library_types + ;; + + init) + # Bundled one-shot init run by the `init` sidecar on every + # `docker compose up`. Idempotent: re-runs are no-ops unless + # migrations or library-type seed data need to change. + # + # Vector-index creation intentionally runs in *best-effort* mode: + # ``setup_neo4j_indexes`` requires a system embedding model with a + # configured ``vector_dimensions`` value, and that model is data an + # operator seeds via the admin UI after the stack comes up for the + # first time. Blocking the whole stack on first boot would force + # every new deployer through a manual dance with the init sidecar's + # entrypoint; instead we log loudly and carry on, and the operator + # runs the command once post-boot: + # + # docker compose exec app python manage.py setup_neo4j_indexes + # + # Full-text and neomodel constraint indexes are created by the same + # command and are *not* dimension-sensitive, but they also only land + # after the operator re-runs it — acceptable because search against + # an empty graph is itself a no-op. + set -e + python manage.py migrate --noinput + python manage.py load_library_types + if ! python manage.py setup_neo4j_indexes; then + echo "" + echo "============================================================" + echo "NOTICE: Neo4j index creation was skipped." + echo "" + echo "This is expected on a fresh deployment — vector indexes" + echo "require a system embedding model with vector_dimensions set." + echo "" + echo "Seed the embedding model in the Django admin" + echo " (/admin/llm_manager/llmmodel/, mark one row as" + echo " is_system_embedding_model=True with vector_dimensions set)," + echo "then run:" + echo "" + echo " docker compose exec app python manage.py setup_neo4j_indexes" + echo "" + echo "Search endpoints will return empty results until this is done." + echo "============================================================" + echo "" + fi + ;; + + shell) + # Drop into the management shell for ad-hoc work. + exec python manage.py shell + ;; + + *) + # Fall through: run whatever was passed (e.g. `manage.py `). + exec "$@" + ;; +esac