docs(readme): clarify embedding model seed order for Neo4j indexes
Document that the system embedding model must be seeded before running `setup_neo4j_indexes`, since vector index dimensions are read from the `llm_manager_llmmodel` row. Update Docker instructions to reflect the `init` sidecar behavior, which now runs migrations and library_type defaults automatically while deferring vector index creation.
This commit is contained in:
31
README.md
31
README.md
@@ -76,10 +76,23 @@ Hosts in the Ouranos lab:
|
|||||||
```bash
|
```bash
|
||||||
cd mnemosyne/
|
cd mnemosyne/
|
||||||
python manage.py migrate # Apply Django ORM migrations
|
python manage.py migrate # Apply Django ORM migrations
|
||||||
python manage.py setup_neo4j_indexes # Create Neo4j vector + full-text indexes
|
|
||||||
python manage.py load_library_types # Load LIBRARY_TYPE_DEFAULTS into Neo4j
|
python manage.py load_library_types # Load LIBRARY_TYPE_DEFAULTS into Neo4j
|
||||||
|
# --- seed the system embedding model in /admin/llm_manager/llmmodel/ here ---
|
||||||
|
python manage.py setup_neo4j_indexes # Create Neo4j vector + full-text indexes
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Seed the embedding model before running `setup_neo4j_indexes`.** Vector
|
||||||
|
> index dimensions are read from the row in ``llm_manager_llmmodel`` that
|
||||||
|
> has ``is_system_embedding_model=True`` and a non-null ``vector_dimensions``.
|
||||||
|
> There is deliberately no hardcoded fallback: an index built at the wrong
|
||||||
|
> dimension silently breaks every search. The command will exit non-zero
|
||||||
|
> with a clear error if no such row exists, which is also why the
|
||||||
|
> ``docker compose`` ``init`` sidecar treats vector-index creation as
|
||||||
|
> best-effort on first boot — the stack starts healthy, migrations and
|
||||||
|
> library-type seed data land, and you run
|
||||||
|
> ``docker compose exec app python manage.py setup_neo4j_indexes`` once
|
||||||
|
> the embedding-model row is in place.
|
||||||
|
|
||||||
### Start the web app
|
### Start the web app
|
||||||
|
|
||||||
The Django REST API serves `/library/api/*` (libraries, collections, items, search, workspaces, ingest) and Django admin. Use Gunicorn in production; `runserver` for dev.
|
The Django REST API serves `/library/api/*` (libraries, collections, items, search, workspaces, ingest) and Django admin. Use Gunicorn in production; `runserver` for dev.
|
||||||
@@ -199,14 +212,16 @@ cp .env.example .env && $EDITOR .env
|
|||||||
# Pull the image (or build locally with `docker compose build`)
|
# Pull the image (or build locally with `docker compose build`)
|
||||||
docker compose pull
|
docker compose pull
|
||||||
|
|
||||||
# DB migrations (one-shot)
|
# Bring the stack up — the `init` sidecar runs migrations + library_type
|
||||||
docker compose run --rm app migrate
|
# defaults automatically. Vector indexes are deferred until you seed the
|
||||||
|
# system embedding model (see below) — the sidecar logs a clear notice
|
||||||
# Neo4j indexes + library_type defaults (one-shot)
|
# and exits 0 either way, so the stack comes up healthy on first boot.
|
||||||
docker compose run --rm app setup
|
|
||||||
|
|
||||||
# Bring the stack up
|
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
|
|
||||||
|
# Seed the system embedding model at /admin/llm_manager/llmmodel/
|
||||||
|
# (mark one row `is_system_embedding_model=True` with `vector_dimensions`
|
||||||
|
# set to whatever your embedding provider returns), then:
|
||||||
|
docker compose exec app python manage.py setup_neo4j_indexes
|
||||||
```
|
```
|
||||||
|
|
||||||
### Day-to-day
|
### Day-to-day
|
||||||
|
|||||||
111
mnemosyne/docker/entrypoint.sh
Normal file
111
mnemosyne/docker/entrypoint.sh
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Mnemosyne container entrypoint.
|
||||||
|
#
|
||||||
|
# The same image runs all three processes — the compose service supplies
|
||||||
|
# `web`, `mcp`, `worker`, or `migrate` as CMD.
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
case "$1" in
|
||||||
|
web)
|
||||||
|
# Django REST API + admin (gunicorn → wsgi).
|
||||||
|
exec gunicorn \
|
||||||
|
--config /app/docker/gunicorn.conf.py \
|
||||||
|
--bind 0.0.0.0:8000 \
|
||||||
|
--workers "${GUNICORN_WORKERS:-3}" \
|
||||||
|
--access-logfile - \
|
||||||
|
--error-logfile - \
|
||||||
|
mnemosyne.wsgi:application
|
||||||
|
;;
|
||||||
|
|
||||||
|
mcp)
|
||||||
|
# FastMCP over Streamable HTTP at /mcp/, mounted by mnemosyne.asgi.
|
||||||
|
exec uvicorn \
|
||||||
|
--host 0.0.0.0 \
|
||||||
|
--port 8001 \
|
||||||
|
--workers "${UVICORN_WORKERS:-1}" \
|
||||||
|
mnemosyne.asgi:app
|
||||||
|
;;
|
||||||
|
|
||||||
|
worker)
|
||||||
|
# Celery worker covering embedding + ingest + batch + default queues.
|
||||||
|
# In production you may want to split these onto separate worker
|
||||||
|
# services for queue-level isolation; one process is fine to start.
|
||||||
|
exec celery -A mnemosyne worker \
|
||||||
|
--loglevel="${CELERY_LOG_LEVEL:-info}" \
|
||||||
|
--queues="${CELERY_QUEUES:-celery,embedding,batch}" \
|
||||||
|
--concurrency="${CELERY_CONCURRENCY:-2}"
|
||||||
|
;;
|
||||||
|
|
||||||
|
beat)
|
||||||
|
# Celery scheduled tasks (only needed if/when periodic jobs are wired).
|
||||||
|
exec celery -A mnemosyne beat \
|
||||||
|
--loglevel="${CELERY_LOG_LEVEL:-info}"
|
||||||
|
;;
|
||||||
|
|
||||||
|
migrate)
|
||||||
|
# One-shot DB migration runner — invoke before bringing services up
|
||||||
|
# for the first time or after a deploy.
|
||||||
|
exec python manage.py migrate --noinput
|
||||||
|
;;
|
||||||
|
|
||||||
|
setup)
|
||||||
|
# One-shot init — Neo4j indexes + library_type seed data.
|
||||||
|
python manage.py setup_neo4j_indexes
|
||||||
|
python manage.py load_library_types
|
||||||
|
;;
|
||||||
|
|
||||||
|
init)
|
||||||
|
# Bundled one-shot init run by the `init` sidecar on every
|
||||||
|
# `docker compose up`. Idempotent: re-runs are no-ops unless
|
||||||
|
# migrations or library-type seed data need to change.
|
||||||
|
#
|
||||||
|
# Vector-index creation intentionally runs in *best-effort* mode:
|
||||||
|
# ``setup_neo4j_indexes`` requires a system embedding model with a
|
||||||
|
# configured ``vector_dimensions`` value, and that model is data an
|
||||||
|
# operator seeds via the admin UI after the stack comes up for the
|
||||||
|
# first time. Blocking the whole stack on first boot would force
|
||||||
|
# every new deployer through a manual dance with the init sidecar's
|
||||||
|
# entrypoint; instead we log loudly and carry on, and the operator
|
||||||
|
# runs the command once post-boot:
|
||||||
|
#
|
||||||
|
# docker compose exec app python manage.py setup_neo4j_indexes
|
||||||
|
#
|
||||||
|
# Full-text and neomodel constraint indexes are created by the same
|
||||||
|
# command and are *not* dimension-sensitive, but they also only land
|
||||||
|
# after the operator re-runs it — acceptable because search against
|
||||||
|
# an empty graph is itself a no-op.
|
||||||
|
set -e
|
||||||
|
python manage.py migrate --noinput
|
||||||
|
python manage.py load_library_types
|
||||||
|
if ! python manage.py setup_neo4j_indexes; then
|
||||||
|
echo ""
|
||||||
|
echo "============================================================"
|
||||||
|
echo "NOTICE: Neo4j index creation was skipped."
|
||||||
|
echo ""
|
||||||
|
echo "This is expected on a fresh deployment — vector indexes"
|
||||||
|
echo "require a system embedding model with vector_dimensions set."
|
||||||
|
echo ""
|
||||||
|
echo "Seed the embedding model in the Django admin"
|
||||||
|
echo " (/admin/llm_manager/llmmodel/, mark one row as"
|
||||||
|
echo " is_system_embedding_model=True with vector_dimensions set),"
|
||||||
|
echo "then run:"
|
||||||
|
echo ""
|
||||||
|
echo " docker compose exec app python manage.py setup_neo4j_indexes"
|
||||||
|
echo ""
|
||||||
|
echo "Search endpoints will return empty results until this is done."
|
||||||
|
echo "============================================================"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
|
||||||
|
shell)
|
||||||
|
# Drop into the management shell for ad-hoc work.
|
||||||
|
exec python manage.py shell
|
||||||
|
;;
|
||||||
|
|
||||||
|
*)
|
||||||
|
# Fall through: run whatever was passed (e.g. `manage.py <cmd>`).
|
||||||
|
exec "$@"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Reference in New Issue
Block a user