r/mnemosyne
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(integration): mark Phases 1+2 as implemented; add Phase 3 stub

Browse Source 
The integration doc was forward-looking spec but most of it now ships:

  Phase 1 (REST workspace + ingest API for Daedalus)        ✅ implemented
  Phase 2 (MCP server: search/get_chunk/list_*/get_health)  ✅ implemented
  Phase 3 (per-turn signed-token access control)            📋 deferred

Updated:
- Tool table reflects actual implementation (search, get_chunk,
  list_libraries, list_collections, list_items, get_health) instead
  of the speculative names (search_knowledge, search_by_category, etc.)
- Project structure matches the as-built layout (tools/discovery.py
  exists; no separate browse.py).
- REST API table covers both workspace lifecycle endpoints and ingest
  endpoints, with correct routes (/library/api/...).
- Ingest request schema includes content_hash and workspace_id
  (the actual idempotency key on the Mnemosyne side).
- Celery task description matches library.tasks.ingest_from_daedalus
  rather than the placeholder embed_item.
- Phase 6 checklist marks Phases 1+2 done; adds Phase 3 (per-turn
  token access control) with a per-Mnemosyne-side TODO list pointing
  at the matching Daedalus-side §9 design.

Internal MCP port stays 22091; public access via nginx on 23090.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Robert Helewka
2026-05-02 21:54:05 -04:00
parent 236d9e2e74
commit e5618973fc
1 changed files with 106 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

178
docs/mnemosyne_integration.md
View File

@@ -1,6 +1,6 @@
# Mnemosyne Integration — Daedalus & Pallas Reference # Mnemosyne Integration — Daedalus & Pallas Reference
This document summarises the Mnemosyne-specific implementation required for integration with the Daedalus & Pallas architecture. The full specification lives in [`daedalus/docs/mnemosyne_integration.md`](../../daedalus/docs/mnemosyne_integration.md). This document describes Mnemosyne's role in the Daedalus + Pallas architecture and what's actually built today. The Daedalus-side spec lives in [`daedalus/docs/mnemosyne_integration.md`](../../daedalus/docs/mnemosyne_integration.md).
--- ---
@@ -8,49 +8,59 @@ This document summarises the Mnemosyne-specific implementation required for inte
Mnemosyne exposes two interfaces for the wider Ouranos ecosystem: Mnemosyne exposes two interfaces for the wider Ouranos ecosystem:
1. **MCP Server** (port 22091) — consumed by Pallas agents for synchronous search, browse, and retrieval operations 1. **REST API** (`/library/api/*`) — consumed by the Daedalus backend (HTTP Basic auth, service account `daedalus-service`) for workspace lifecycle and asynchronous file ingestion. Phase 1, **implemented**.
2. **REST Ingest API** — consumed by the Daedalus backend for asynchronous file ingestion and embedding job lifecycle management 2. **MCP Server** (port 22091 internal, `/mcp/` via nginx on 23090) — exposes search, browse, and retrieval tools. Phase 5 of Mnemosyne's own roadmap, **implemented** with workspace_id scoping. Currently consumed by an internal validator (`validator/`) and ad-hoc clients; planned production consumer is Pallas FastAgents in Daedalus integration Phase 2 (deferred — see [Phase 3 of this doc](#3-phase-3-deferred-per-turn-token-access-control)).
### Phase status
| Phase | What | Status |
|-------|------|--------|
| 1. REST workspace + ingest API for Daedalus | `POST /workspaces/`, `DELETE /workspaces/{id}/`, `POST /ingest/`, `GET /jobs/{id}/` | **Implemented** |
| 2. MCP Server (Mnemosyne roadmap Phase 5) | `search`, `get_chunk`, `list_libraries`, `list_collections`, `list_items`, `get_health` | **Implemented** (workspace_id scoping in place; access-control to follow in Phase 3) |
| 3. Per-turn signed-token access control for Daedalus integration | Daedalus mints tokens carrying `{workspace_id, allowed_libraries}` claims; Mnemosyne validates and scopes search server-side | **Deferred** |
--- ---
## 1. MCP Server (Phase 5) ## 1. MCP Server
### Port & URL ### Port & URL
| Service | Port | URL | | Endpoint | Internal | Public (via nginx) |
|---------|------|-----| |---|---|---|
| Mnemosyne MCP | 22091 | `http://puck.incus:22091/mcp` | | MCP server | `http://mcp:22091/mcp/` | `http://puck.incus:23090/mcp/` |
| Health check | 22091 | `http://puck.incus:22091/mcp/health` | | Health check | `http://mcp:22091/mcp/health` | `http://puck.incus:23090/healthz` |
### Project Structure ### Project structure (as built)
Following the [Django MCP Pattern](Pattern_Django-MCP_V1-00.md): Follows the [Django MCP Pattern](Pattern_Django-MCP_V1-00.md):
``` ```
mnemosyne/mnemosyne/mcp_server/ mnemosyne/mnemosyne/mcp_server/
├── __init__.py ├── __init__.py
├── server.py # FastMCP instance + tool registration ├── server.py # FastMCP instance + tool registration
├── asgi.py # Starlette ASGI mount at /mcp ├── auth.py # MCPAuthMiddleware
├── middleware.py # MCPAuthMiddleware (disabled for internal use)
├── context.py # get_mcp_user(), get_mcp_token() ├── context.py # get_mcp_user(), get_mcp_token()
└── tools/ └── tools/
├── __init__.py ├── __init__.py
├── search.py # register_search_tools(mcp) → search_knowledge, search_by_category ├── search.py # register_search_tools(mcp) → search, get_chunk
├── browse.py # register_browse_tools(mcp) → list_libraries, list_collections, get_item, get_concepts ├── discovery.py # register_discovery_tools(mcp) → list_libraries, list_collections, list_items
└── health.py # register_health_tools(mcp) → get_health └── health.py # register_health_tools(mcp) → get_health
``` ```
### Tools to Implement The ASGI mount lives at `mnemosyne/mnemosyne/asgi.py` (project-level) — it composes the FastMCP app at `/mcp/` with a 307 redirect from bare `/mcp` so MCP clients that omit the trailing slash still land correctly.
### Tools (as implemented)
| Tool | Module | Description | | Tool | Module | Description |
|------|--------|-------------| |------|--------|-------------|
| `search_knowledge` | `search.py` | Hybrid vector + full-text + graph search → re-rank → return chunks with citations | | `search` | `search.py` | Hybrid vector + full-text + concept-graph search → fusion → optional Synesis re-rank. Accepts `library_uid`, `library_type`, `collection_uid`, and (system-injected, undocumented to LLM) `workspace_id` for scoping. |
| `search_by_category` | `search.py` | Same as above, scoped to a specific `library_type` | | `get_chunk` | `search.py` | Fetch full text of a chunk by uid (typically obtained from `search`). Honors workspace_id scoping. |
| `list_libraries` | `browse.py` | List all libraries with type, description, counts | | `list_libraries` | `discovery.py` | List libraries with uid, name, library_type, description. Workspace_id-aware. |
| `list_collections` | `browse.py` | List collections within a library | | `list_collections` | `discovery.py` | List collections, optionally filtered by parent library. Workspace_id-aware. |
| `get_item` | `browse.py` | Retrieve item detail with chunk previews and concept links | | `list_items` | `discovery.py` | List items with chunk_count, image_count, embedding_status. Workspace_id-aware. |
| `get_concepts` | `browse.py` | Traverse concept graph from a starting concept or item | | `get_health` | `health.py` | Check Neo4j, S3, embedding model reachability. Used by Pallas health pollers. |
| `get_health` | `health.py` | Check Neo4j, S3, embedding model reachability |
The `workspace_id` parameter is present on every search/discovery tool but is **deliberately undocumented in the LLM-facing tool description** — it's a system-injected field the calling LLM should never know about. A workspace-scoped query returns ONLY that workspace's content; an unscoped query (workspace_id is NULL) returns ONLY global libraries. There is no mode that mixes the two — see `library/services/search.py`, `_WORKSPACE_SCOPE_CLAUSE`.
### MCP Resources ### MCP Resources
@@ -91,22 +101,30 @@ Auth is disabled (`MCP_REQUIRE_AUTH=False`) since all traffic is internal (10.10
--- ---
## 2. REST Ingest API ## 2. REST API for Daedalus
### New Endpoints All endpoints require HTTP Basic auth as `daedalus-service`. They are consumed by the Daedalus FastAPI backend only — not by any frontend.
### Workspace lifecycle
| Method | Route | Purpose | | Method | Route | Purpose |
|--------|-------|---------| |--------|-------|---------|
| `POST` | `/api/v1/library/ingest` | Accept a file for ingestion + embedding | | `POST` | `/library/api/workspaces/` | Create workspace Library. Body: `{workspace_id, name, library_type, description?}`. Idempotent on `workspace_id`. `library_type` frozen at create. |
| `GET` | `/api/v1/library/jobs/{job_id}` | Poll job status | | `GET` | `/library/api/workspaces/{workspace_id}/` | Workspace status (item_count, chunk_count, library_uid). |
| `POST` | `/api/v1/library/jobs/{job_id}/retry` | Retry a failed job | | `DELETE` | `/library/api/workspaces/{workspace_id}/` | Delete workspace Library + reachable content. Concept-safe: orphan-only Concept GC; concepts referenced by other libraries survive. |
| `GET` | `/api/v1/library/jobs` | List recent jobs (optional `?status=` filter) |
These endpoints are consumed by the **Daedalus FastAPI backend** only. Not by the frontend. ### Ingest
### New Model: `IngestJob` | Method | Route | Purpose |
|--------|-------|---------|
| `POST` | `/library/api/ingest/` | Accept a file (already in S3) for ingestion + embedding |
| `GET` | `/library/api/jobs/{job_id}/` | Poll job status |
| `POST` | `/library/api/jobs/{job_id}/retry/` | Retry a failed job |
| `GET` | `/library/api/jobs/?status=&library_uid=` | List recent jobs |
Add to `library/` app (Django ORM on PostgreSQL, not Neo4j): ### Model: `IngestJob`
Lives in `library/models.py` (Django ORM on PostgreSQL, not Neo4j). Migration: `library/migrations/0001_initial.py`.
```python ```python
class IngestJob(models.Model): class IngestJob(models.Model):
@@ -153,14 +171,16 @@ class IngestJob(models.Model):
### Ingest Request Schema ### Ingest Request Schema
The target Library can be specified by either `workspace_id` (preferred for Daedalus) or `library_uid`. Idempotency key: `(library, source_ref, content_hash)`. Same triple → existing job returned. New `content_hash` for the same `source_ref` → supersedes the prior Item.
```json ```json
{ {
"s3_key": "workspaces/ws_abc/files/f_def/report.pdf", "s3_key": "workspaces/ws_abc/files/f_def/report.pdf",
"title": "Q4 Technical Report", "title": "Q4 Technical Report",
"library_uid": "lib_technical_001", "workspace_id": "ws_abc",
"collection_uid": "col_reports_2026",
"file_type": "application/pdf", "file_type": "application/pdf",
"file_size": 245000, "file_size": 245000,
"content_hash": "<sha256 hex, 64 chars>",
"source": "daedalus", "source": "daedalus",
"source_ref": "ws_abc/f_def" "source_ref": "ws_abc/f_def"
} }
@@ -198,39 +218,34 @@ class IngestJob(models.Model):
## 3. Celery Embedding Pipeline ## 3. Celery Embedding Pipeline
### New Task: `embed_item` ### Task: `ingest_from_daedalus`
Defined in `library/tasks.py`. Routed to the `embedding` queue (per `CELERY_TASK_ROUTES["library.tasks.ingest_*"]`). Wraps the existing `EmbeddingPipeline.process_item`.
```python ```python
@shared_task( @shared_task(
name="library.embed_item", name="library.tasks.ingest_from_daedalus",
bind=True, bind=True,
queue="embedding",
max_retries=3, max_retries=3,
default_retry_delay=60, default_retry_delay=60,
autoretry_for=(S3ConnectionError, EmbeddingModelError),
retry_backoff=True,
retry_backoff_max=600,
acks_late=True, acks_late=True,
queue="embedding",
) )
def embed_item(self, job_id, item_uid): def ingest_from_daedalus(self, job_id: str): ...
...
``` ```
### Task Flow ### Task flow (as built)
1. Update job → `processing` / `fetching` 1. Mark job `processing`, set `started_at`.
2. Fetch file from Daedalus S3 bucket (cross-bucket read) 2. Resolve target Library by `library_uid`.
3. Copy to Mnemosyne's own S3 bucket 3. If a prior Item exists for this Library with the same `source_ref` but a *different* `content_hash`, delete it (chunks + images + embeddings) before continuing.
4. Load library type → chunking config 4. Fetch file bytes from the Daedalus S3 bucket via `library.services.daedalus_s3.fetch_from_daedalus`.
5. Chunk content per strategy 5. Create the `Item` neomodel node with `s3_key=items/{item_uid}/original.{ext}` and copy bytes into Mnemosyne's own bucket.
6. Store chunk text in S3 6. Connect to a default Collection for the Library (auto-created on first ingest).
7. Generate embeddings (Arke/vLLM batch call) 7. Run `EmbeddingPipeline.process_item(item.uid)` — chunk per `library_type`, embed via the configured model, write Chunks + Concepts to Neo4j.
8. Write Chunk nodes + vectors to Neo4j 8. Mark job `completed` with `chunks_created`, `concepts_extracted`, `embedding_model`, `completed_at`.
9. Extract concepts (LLM call)
10. Build graph relationships
11. Update job → `completed`
On failure at any step: update job → `failed` with error message. On any exception with retries remaining: re-raise via `self.retry()` (exponential backoff). On terminal failure: mark job `failed` with the exception text.
### ⚠️ DEBUG LOG Points — Celery Worker (Critical) ### ⚠️ DEBUG LOG Points — Celery Worker (Critical)
@@ -329,23 +344,40 @@ mnemosyne_s3_operations_total{operation,status} counter
## 6. Implementation Phases (Mnemosyne-specific) ## 6. Implementation Phases (Mnemosyne-specific)
### Phase 1 — REST Ingest API ### Phase 1 — REST API for Daedalus (workspace + ingest) ✅ Implemented
- [ ] Create `IngestJob` model + Django migration - [x] `Library.workspace_id` + `library_type` enum (added `business`, `finance`)
- [ ] Implement `POST /api/v1/library/ingest` endpoint - [x] `IngestJob` Django ORM model + migration `0001_initial.py`
- [ ] Implement `GET /api/v1/library/jobs/{job_id}` endpoint - [x] `POST /library/api/workspaces/`, `GET /library/api/workspaces/{id}/`, `DELETE /library/api/workspaces/{id}/` (concept-safe)
- [ ] Implement `POST /api/v1/library/jobs/{job_id}/retry` endpoint - [x] `POST /library/api/ingest/` with `(library, source_ref, content_hash)` idempotency
- [ ] Implement `GET /api/v1/library/jobs` list endpoint - [x] `GET /library/api/jobs/{job_id}/`, `POST .../retry/`, `GET /library/api/jobs/`
- [ ] Implement `embed_item` Celery task with full debug logging - [x] `library.tasks.ingest_from_daedalus` Celery task with content-hash-aware supersede logic
- [ ] Add S3 cross-bucket copy logic - [x] `library.services.daedalus_s3` cross-bucket fetch + copy
- [ ] Add ingest API serializers and URL routing - [x] HTTP Basic auth via `daedalus-service` user
### Phase 2 — MCP Server (Phase 5 of Mnemosyne roadmap) ### Phase 2 — MCP Server (Mnemosyne roadmap Phase 5) ✅ Implemented
- [ ] Create `mcp_server/` module following Django MCP Pattern - [x] `mcp_server/` module following the [Django MCP Pattern](Pattern_Django-MCP_V1-00.md)
- [ ] Implement `search_knowledge` tool (hybrid search + re-rank) - [x] `search` tool (hybrid vector + fulltext + concept-graph + Synesis re-rank)
- [ ] Implement `search_by_category` tool - [x] `get_chunk` tool (full text by chunk_uid)
- [ ] Implement `list_libraries`, `list_collections`, `get_item`, `get_concepts` tools - [x] `list_libraries`, `list_collections`, `list_items` discovery tools
- [ ] Implement `get_health` tool per Pallas health spec - [x] `get_health` tool (Neo4j + S3 + embedding model probes)
- [ ] Register MCP resources (`mnemosyne://library-types`, `mnemosyne://libraries`) - [x] Workspace_id parameter on every search/discovery tool (undocumented to LLM, scoping enforced in Cypher)
- [ ] ASGI mount + Uvicorn deployment on port 22091 - [x] Single-mode rule: workspace-scoped vs global, never both in one query
- [ ] Systemd service for MCP Uvicorn process - [x] ASGI mount + uvicorn deployment on port 22091; nginx proxies via `/mcp/` on 23090
- [ ] Add Prometheus metrics - [x] Prometheus metrics (`mnemosyne_mcp_*`)
### Phase 3 — Per-turn token access control for Daedalus integration 📋 Deferred
The Phase 2 MCP server is search-capable but currently has no token-based library-access scoping beyond `workspace_id` (which is parameter-level, not auth-level). The intended production access-control layer for the Daedalus integration is a per-turn signed token model:
- Daedalus mints short-lived tokens carrying `{sub: agent_id, workspace_id, allowed_libraries, exp}`.
- Pallas forwards the inbound bearer to its outgoing Mnemosyne MCP calls (requires a small upstream patch — see Daedalus-side §9.4).
- Mnemosyne's MCP token validator extracts the claims; search Cypher additionally filters `WHERE lib.uid IN $allowed_libraries`.
- Workspace libraries are auto-included in the per-turn token's allowed list when the agent is being invoked from that workspace.
Mnemosyne-side work for Phase 3:
- [ ] Extend `MCPToken` (or sibling) to carry signed claims `{workspace_id, allowed_libraries, exp}`
- [ ] Token validator reads claims, attaches them to the FastMCP request context
- [ ] `search` / `list_*` tools consult claim-derived allowed-library set in addition to existing parameter filters
- [ ] Document the JWT/signing format Daedalus mints to (likely HS256 with a shared secret in Vault, or RS256 against Daedalus's JWKS — TBD)
See the Daedalus-side spec [§9](../../daedalus/docs/mnemosyne_integration.md#9-phase-2--knowledge-library-access-control-deferred) for the full integration architecture.