feat: add init sidecar for migrations and setup on compose up
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 50s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m30s

Introduces a one-shot `init` service in docker-compose that runs Postgres
migrations, Neo4j index setup, and library-type seeding on every `up`.
Long-running services (`app`, `mcp`, `worker`) now depend on its
successful completion via `service_completed_successfully`, blocking the
stack on configuration errors (missing embedding model, dimension
mismatch, unreachable DB) rather than serving silent zero-result
searches.

Also standardizes reranker test fixtures to use the `/v1` OpenAI-style
base URL convention used across other service clients.
This commit is contained in:
2026-05-10 08:01:58 -04:00
parent 9ceb01f829
commit a945b382e6
15 changed files with 821 additions and 65 deletions

View File

@@ -61,6 +61,22 @@ POST http://pan.helu.ca:8400/v1/rerank
}
```
> **`LLMApi.base_url` convention.** Every Mnemosyne service client
> (`EmbeddingClient`, `RerankerClient`, `vision.py`, `concepts.py`)
> treats `base_url` as the **OpenAI-style `/v1` root** and appends a
> path-only segment: `/embeddings`, `/rerank`, `/chat/completions`.
> So a single `LLMApi` row with `base_url=http://pan.helu.ca:8400/v1`
> serves both the embedding and the reranker endpoints — no per-purpose
> duplication needed.
>
> Get this wrong (e.g. set `base_url=http://pan.helu.ca:8400` with no
> `/v1`, or have a client prepend `/v1` locally) and you get a
> double-prefixed URL like `…/v1/v1/rerank` that 404s silently —
> `SearchService._rerank` catches the exception, the UI shows
> "Re-rank: Skipped", and the search falls back to raw RRF order.
> Check `results.reranker_skip_reason` on the search page for the
> specific error.
## Deliverables
### 1. Search Service (`library/services/search.py`)

View File

@@ -294,31 +294,37 @@ graph LR
</div>
<div class="alert alert-warning border-start border-4 border-warning">
<h4><i class="bi bi-lightning"></i> Neo4j Vector Indexes</h4>
<pre class="bg-light p-3 rounded mb-0"><code>// Chunk text+image embeddings (4096 dimensions, no pgvector limits!)
CREATE VECTOR INDEX chunk_embedding FOR (c:Chunk)
<h4><i class="bi bi-lightning"></i> Neo4j Indexes (managed by <code>setup_neo4j_indexes</code>)</h4>
<p>Created by the <code>init</code> sidecar on every <code>docker compose up</code>. Vector dimensions come from the system embedding model's <code>vector_dimensions</code> field — the command fails if no model is configured. Current production model: <strong>Pan Synesis · qwen3-vl-embedding-2b · 2048d</strong>.</p>
<pre class="bg-light p-3 rounded mb-0"><code>// Chunk text+image embeddings (dimensions read from system embedding model)
CREATE VECTOR INDEX chunk_embedding_index FOR (c:Chunk)
ON (c.embedding) OPTIONS {indexConfig: {
`vector.dimensions`: 4096,
`vector.dimensions`: 2048,
`vector.similarity_function`: 'cosine'
}}
// Concept embeddings for semantic concept search
CREATE VECTOR INDEX concept_embedding FOR (con:Concept)
CREATE VECTOR INDEX concept_embedding_index FOR (con:Concept)
ON (con.embedding) OPTIONS {indexConfig: {
`vector.dimensions`: 4096,
`vector.dimensions`: 2048,
`vector.similarity_function`: 'cosine'
}}
// Image multimodal embeddings
CREATE VECTOR INDEX image_embedding FOR (ie:ImageEmbedding)
CREATE VECTOR INDEX image_embedding_index FOR (ie:ImageEmbedding)
ON (ie.embedding) OPTIONS {indexConfig: {
`vector.dimensions`: 4096,
`vector.dimensions`: 2048,
`vector.similarity_function`: 'cosine'
}}
// Full-text index for keyword/BM25-style search
CREATE FULLTEXT INDEX chunk_fulltext FOR (c:Chunk) ON EACH [c.text_preview]</code></pre>
// Full-text indexes (BM25-style keyword search)
CREATE FULLTEXT INDEX chunk_text_fulltext FOR (c:Chunk) ON EACH [c.text_preview]
CREATE FULLTEXT INDEX concept_name_fulltext FOR (c:Concept) ON EACH [c.name]
CREATE FULLTEXT INDEX item_title_fulltext FOR (i:Item) ON EACH [i.title]
CREATE FULLTEXT INDEX library_name_fulltext FOR (l:Library) ON EACH [l.name]</code></pre>
<p class="mb-0 mt-3"><strong>Changing the embedding model or dimensions is a re-embedding event.</strong> Drop + recreate the vector indexes (<code>setup_neo4j_indexes --drop</code>) and re-queue all content for embedding. Old vectors at the previous dimension remain on the nodes until overwritten but are no longer indexed.</p>
</div>
</section>
<!-- SECTION: CONTENT TYPES -->
@@ -521,10 +527,11 @@ flowchart TD
<div class="card-body">
<p>Cosine similarity via Neo4j vector index on Chunk and ImageEmbedding nodes.</p>
<pre class="bg-light p-2 rounded"><code>CALL db.index.vector.queryNodes(
'chunk_embedding', 30,
'chunk_embedding_index', 30,
$query_vector
) YIELD node, score
WHERE score > $threshold</code></pre>
</div>
</div>
</div>
@@ -548,9 +555,10 @@ RETURN c2, i2</code></pre>
<div class="card-body">
<p>Neo4j native full-text index for keyword matching (BM25-equivalent).</p>
<pre class="bg-light p-2 rounded"><code>CALL db.index.fulltext.queryNodes(
'chunk_fulltext',
'chunk_text_fulltext',
$query_text
) YIELD node, score</code></pre>
</div>
</div>
</div>