"svg" was in both IMAGE_EXTENSIONS and PYMUPDF_EXTENSIONS, and the image
check ran first, so every SVG reached PIL.Image.open — which cannot decode
vector XML. Ingest raised UnidentifiedImageError, logged "Failed to read
image file_type=svg", and counted documents_parsed_total{status="error"}.
SVG ingest has never worked.
Rasterize to PNG instead, via PyMuPDF (already a dependency). This also
unblocks the vision stage downstream: it sends images to a vision LLM as a
data: URI, which cannot carry image/svg+xml either, so the description and
OCR fields were unreachable for SVG regardless of the parse fix.
Route SVG explicitly before both extension sets. Falling through to
PYMUPDF_EXTENSIONS would parse but not split, and a multi-page Write note
rendered as one document is either blank (no root width/height, so MuPDF
falls back to US-Letter and emits the top-left corner) or an illegible tall
strip (root sized across every stacked page). Splitting per write-page
element is correct for both formats, and since Write never rewrites
existing files both persist indefinitely.
svg_raster is a deliberate twin of daedalus's extraction/svg.py — the repos
share no common package, so the duplication is noted in both docstrings and
fixes belong in both.
lxml is a new dependency: recover=True is needed for the real files that
aren't well-formed XML, and stdlib ElementTree has no equivalent.
Kairos is about to provision per-user app-managed (workspace) libraries for
synced mail, and none of the existing types fit correspondence. New entry in
LIBRARY_TYPE_DEFAULTS mirrors journal's entry-level chunking (one message ≈
one entry) with instructions tuned for email: correspondents, dates,
requests/commitments, and quoted-reply context. Registered in the Library
node choices and the API serializer choice list; forms and
load_library_types derive from the registry, and the compose init sidecar
re-runs the seeder on deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a `scope` GET parameter to `library_list` allowing users to filter
libraries by all, global-only, or Daedalus workspace-only. Includes a
filter form in the template, an updated empty state message, and tests
covering each scope branch with mocked Neo4j dependencies.
The backend already allows admin delete of a workspace-scoped Library via
the shared delete_library_cascade (commit 142e967), and the confirm-delete
page carries the Daedalus caution. But the detail template was never updated
to match: the Delete button stayed hardcoded `disabled` for any library with
a workspace_id, and the banner said "Do not delete it manually." So the
working route was unreachable from the UI — admins still could not clear an
orphaned Library blocking workspace re-registration.
Make Delete an unconditional enabled link to library:library-delete (the
confirm page already warns), and rewrite the workspace banner to describe the
escape hatch (deleting here removes embedded content; Daedalus recreates and
re-embeds on the next sync) instead of forbidding it.
Template-only change; no view/model/migration changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A recreate of a workspace whose Mnemosyne Library was orphaned (left behind
by a failed Daedalus delete-propagate) collides on the global Library.name
unique constraint. neomodel raised UniqueProperty unguarded, so workspace_create
500'd and ingest then 404'd forever — the queue froze silently.
Guard lib.save() and return a structured 409 with a machine code so Daedalus
can classify the failure without string-matching:
- name_conflict — the new name-collision case
- owner_conflict, library_type_immutable — codes added to the two existing 409s
Cypher-touching paths stay covered by the manual end-to-end plan, per the
test module's stated convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Several library tests patched symbols at import paths that no longer
expose them, so they errored (AttributeError) instead of testing anything
— giving false confidence. The underlying code is correct; only the test
patch targets were stale after earlier refactors moved imports
function-local.
- test_pipeline: patch source modules (library.models.Item,
llm_manager.models.LLMModel, library.services.parsers.DocumentParser,
.chunker.ContentTypeChunker, .embedding_client.EmbeddingClient,
.vision.VisionAnalyzer, .concepts.ConceptExtractor) since pipeline.py
imports them inside methods. default_storage stays (still module-level).
- test_search_api: patch library.services.search.SearchService (the view
imports it function-local).
- test_tasks: patch library.services.pipeline.EmbeddingPipeline (tasks.py
imports it function-local).
- test_search_views_admin_scope: patch library.utils.neo4j_available; the
guard moved to utils when views._all_library_uids became a thin alias.
- test_concepts: remove SampleIndexSelectionTests — _select_sample_indices
was deleted in the document-level concept-extraction refactor (dead test).
Not addressed here: SearchAPIAuthTest / SearchAPIValidationTest return 302
instead of 401/400. Static analysis ruled out routing, middleware, and DRF
config; reproducing needs a running server (DB-backed). Flagged for sandbox
diagnosis — not a stale-patch issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin/HTML library delete previously hard-blocked workspace-scoped
(Daedalus-managed) libraries, leaving no way to clear an orphaned Library
node — e.g. one left behind when a Daedalus workspace delete failed to
propagate. A recreate of that workspace then collides on the global
Library.name unique constraint and 500s, freezing ingest.
Allow the delete behind the existing confirm warning (low risk: source
content lives in Daedalus and is recreated + re-embedded on next sync),
and route both the API and HTML delete paths through one shared cascade.
- Add library/services/library_delete.delete_library_cascade(lib), keyed on
Library uid so it covers global and workspace-scoped libraries. It removes
Chunks, Images/ImageEmbeddings, Items, Collections, the Library, then GCs
orphan-only Concepts (verbatim from the API view, re-keyed workspace_id->uid).
- workspace_detail_or_delete (API) now calls the shared helper.
- library_delete (HTML) no longer blocks workspace_id libraries; it calls the
cascade instead of a bare lib.delete() (which leaked child nodes — also a
latent bug for global libraries with content).
- Confirm-delete template shows a caution banner for Daedalus-managed libraries.
No migration: Mnemosyne library data is in Neo4j (neomodel); no schema change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Prometheus custom collector that probes the four system-default
models (chat, vision, embedding, reranker) at /metrics scrape time and
emits up/down, configured, and probe-latency gauges. This complements
the ingest-pipeline counters in the Celery worker, which only move
during active ingests and cannot signal model outages on an idle queue.
- New `library/health_collector.py` registers a custom collector with
a 55s in-process cache to avoid hammering GPU endpoints on rapid
scrapes or across multiple gunicorn workers.
- New `library/services/model_health.py` centralises the probe logic,
resolving system-default models via SystemSettings and dispatching
to chat/embedding/rerank endpoints with a short timeout.
- Register the collector only in the web process (gunicorn/runserver)
via `LibraryConfig.ready`, excluding Celery, pytest, and management
commands to prevent duplicate registration and stray probes.
- Add unit tests covering the collector cache, metric shape, and
per-role probe dispatch.
Generalises the Daedalus-only cross-bucket fetch into a registry
(SOURCE_S3_BUCKETS) keyed on the IngestJob `source` field, so new
upstream sources (Spelunker) can ingest from their own buckets. The
ingest task now calls fetch_from_source(job.source, job.s3_key) and
falls back to "daedalus" for blank/unknown sources (backwards compatible).
Adds SPELUNKER_S3_* env vars and worker env scoping. Replaces
daedalus_s3.py with source_s3.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concept extraction was making up to 10 LLM calls per item by sampling
chunks, which produced redundant work (the same concept reappears in
multiple chunks), context-loss bugs (chunk boundaries cut mid-thought),
and on a 35B model dominated per-item wall time (~3 min/item).
Concepts are document-level semantic objects; chunks are retrieval
units. Extract once per item from the first 100KB of parsed document
text, then connect each chunk to the concepts it explicitly mentions
via case-insensitive substring match — no extra LLM calls. Drops the
sample-indices selector that the old per-chunk loop relied on.
Stage 7 is currently dormant in production because the configured
chat model is a reasoning-mode Qwen variant that returns empty content
on every call (output stuck in reasoning_content). Re-enables cleanly
once a non-reasoning instruct model is set as is_system_chat_model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Rename MCPToken to UserToken across models, views, and tests
- Update URL names from mcp-token-* to token-*
- Add Daedalus/Pallas integration design doc (v2)
- Switch docker-compose to build local mnemosyne:local image via shared
build config instead of pulling from git.helu.ca
- Add read-only ModelAdmin for IngestJob with filters, search, and
date hierarchy for operational visibility
- Inject proxy entries into the admin index for Neo4j-backed entities
(Libraries, Concepts, Search, Embedding pipeline) that link to
existing CRUD views in library/views.py
- Makes library content discoverable from /admin/ without pretending
neomodel StructuredNodes are Django ORM models
Increase max_length for source and file_type fields in IngestJob model from 50 to 100.
This prevents data truncation for longer source references or file type strings.
Move probe execution from Django app ready() to gunicorn.conf.py
Remove threading implementation to simplify startup sequence
Ensure probe runs in worker process context with proper error handling
Move the _run_startup_probe logic into a separate daemon thread
within LibraryConfig.ready. This prevents indefinite blocking on
startup while maintaining a 10-second wait for the probe result.
Rework README and docker-compose comments to document the deliberate
chicken-and-egg escape: the `init` sidecar now only runs `migrate` and
`load_library_types`, leaving `setup_neo4j_indexes` as a manual step
after the system embedding model is configured in `/admin/`. This
avoids making `app` unreachable on first boot when no embedding model
row exists yet, while preserving loud failure on dimension mismatch.
Refine the phase-2 integration spec to reflect implementation details:
- Change `resolved_libraries` from `set[str]` to ordered `list[str]`
- Document `MCPToken.allowed_libraries` as JSONField (not M2M) since
Library lives in Neo4j, not Django's ORM
- Clarify that `Library.workspace_id` is a content-routing attribute,
not an authorization axis
- Describe retirement of the three-branch `_WORKSPACE_SCOPE_CLAUSE` in
favor of a single `lib.uid IN $resolved_libraries` check
- Specify team JWT resolution via `TeamWorkspaceAssignment` DB join
- Note admin UI materializes full Library UID list explicitly
Django's `{# #}` syntax only supports single-line comments; multi-line
blocks were rendering as literal text in the search and library detail
templates. Replace them with `{% comment %}...{% endcomment %}` blocks
and add a note explaining the distinction.
Introduces a one-shot `init` service in docker-compose that runs Postgres
migrations, Neo4j index setup, and library-type seeding on every `up`.
Long-running services (`app`, `mcp`, `worker`) now depend on its
successful completion via `service_completed_successfully`, blocking the
stack on configuration errors (missing embedding model, dimension
mismatch, unreachable DB) rather than serving silent zero-result
searches.
Also standardizes reranker test fixtures to use the `/v1` OpenAI-style
base URL convention used across other service clients.
Root cause
----------
SearchService unconditionally appends _WORKSPACE_SCOPE_CLAUSE to every
Cypher query. With both workspace_id and allowed_libraries NULL, the
clause only matches libraries whose workspace_id is also NULL:
AND ( ($workspace_id IS NOT NULL AND lib.workspace_id = $workspace_id)
OR ($allowed_libraries IS NOT NULL AND lib.uid IN $allowed_libraries)
OR ($workspace_id IS NULL AND $allowed_libraries IS NULL
AND lib.workspace_id IS NULL) )
search_page and library_search both built their SearchRequest without
setting either parameter, so the third branch was always the only one
that matched. Every Daedalus-ingested library carries a non-null
workspace_id, so documents ingested via Daedalus were invisible to the
/library/search/ admin UI — the symptom being zero results for terms
that demonstrably exist in indexed chunks.
Fix
---
Both admin-UI views are `@login_required` debug/admin tools for
Django-authenticated operators, not MCP endpoints — they have no
workspace-scoping contract to honour. Added `_all_library_uids()`
helper that returns every Library UID (or [] when Neo4j is down / a
neomodel error bubbles up) and wired it into both views as
`allowed_libraries=`. This flips the scope clause into its second
branch ('lib.uid IN $allowed_libraries'), which matches every library
regardless of workspace_id — reusing the exact mechanism Phase-2 chat
turns use for user-managed libraries.
SearchRequest.__post_init__ collapses an empty list to None, so an
unreachable Neo4j gracefully reverts to the legacy global-only
behaviour rather than 500-ing the page.
Tests
-----
library/tests/test_search_views_admin_scope.py:
* AllLibraryUidsHelperTests — Neo4j unavailable, normal listing,
empty/None-uid filtering, unexpected-exception degradation.
* SearchPageAllowedLibrariesTests — admin POST to /library/search/
reaches SearchService with the captured list; empty list collapses
to None. Stubs SearchService.search to keep the test hermetic.
6 new tests; all 16 tests in library.tests.test_search* are green:
TEST_NEO4J_ENABLED=0 python manage.py test \
library.tests.test_search_views_admin_scope \
library.tests.test_search_scoping \
--testrunner=test_db_manager.django_integration.PostgreSQLTestRunner
- Add guard in `library_delete` view to block deletion of libraries
owned by a Daedalus workspace, redirecting with an error message
- Disable the Delete button in `library_detail.html` for workspace-
scoped libraries and show a warning alert explaining managed ownership
- Add a "Daedalus workspace" badge in both `library_detail.html` and
`library_list.html` to visually identify workspace-owned libraries
Prevents state desync between Mnemosyne and Daedalus by ensuring
workspace-scoped libraries can only be removed via the Daedalus
workspace DELETE API endpoint.
Relocate `search/`, `concepts/`, and `concepts/<str:uid>/` URL patterns
to appear before the `<str:uid>/` catch-all route, preventing Django
from incorrectly matching those static prefixes as library UIDs.
- Add `query_image_ext` field to `SearchRequest` (defaults to "png")
- Embed query from image when supplied and model supports multimodal,
with fallback to text embedding on failure or unsupported model
- Add search form to library detail page with optional image upload,
shown only when multimodal embeddings are available
- Display side-by-side baseline vs re-ranked results with query mode
indicator, timing stats, and score/rank change highlighting
Daedalus may send `file_type` as a MIME type (e.g. `text/markdown`) rather
than a bare extension. Add a `_normalize_file_type` helper with a MIME→ext
lookup table and sensible fallbacks so ingested items are stored with
proper extensions like `md` instead of `text/markdown`.
- Extend library list endpoint with `include_workspace` and
`with_item_count` query params to support Daedalus registry mirroring
- Expand search scope clause to three modes: workspace-only, workspace
plus allowed user libraries, and global
- Add `allowed_libraries` field to SearchRequest for Phase-2 JWT claims
- Introduce JWT-based actor resolution using a synthetic service user
(`MCP_JWT_SERVICE_USERNAME`) for Daedalus-originated requests
Workspace scoping is the integration's security-critical property: an
agent in workspace A must never see content from workspace B or from
any global library, regardless of what the calling LLM tries.
Adds `workspace_id` to SearchRequest with __post_init__ normalization
that converts empty strings to None — so "" cannot slip through as a
truthy filter at the Cypher boundary. Extracts the workspace scope
clause to a single string and appends it to all five search queries
(vector, fulltext-chunk, fulltext-concept, graph, image):
($workspace_id IS NULL AND lib.workspace_id IS NULL
OR lib.workspace_id = $workspace_id)
Either workspace-only or global-only — never both — and the operator
precedence is bracketed so a refactor can't accidentally widen it. A
test verifies the literal clause string for that exact reason.
Adds `workspace_id` as a parameter to every MCP tool (`search`,
`get_chunk`, `list_libraries`, `list_collections`, `list_items`).
Deliberately undocumented in tool docstrings so the calling LLM is never
told the parameter exists — it is system-injected by Daedalus's chat
path and force-overwritten before reaching Mnemosyne. Mnemosyne also
validates the value but the security guarantee is enforced upstream.
Adds the `get_health` MCP tool per the Pallas health spec: returns
ok / degraded / error after probing Neo4j, S3, and the embedding
model registration. Used by Daedalus's existing health poller.
Updates the server INSTRUCTIONS string to advertise the new tool and
the two new library types (business, finance).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the REST API surface that Daedalus calls to manage workspace
lifecycle and dispatch file ingestion. All endpoints under /library/api/:
POST /workspaces/ create workspace (idempotent on
workspace_id; library_type frozen)
GET /workspaces/{workspace_id}/ workspace status with item/chunk
counts
DELETE /workspaces/{workspace_id}/ delete workspace + reachable
content; concept-safe (orphan-only
Concept GC; concepts referenced
elsewhere are preserved)
POST /ingest/ queue a file for ingest. Idempotent
on (library, source_ref, hash):
same triple → return existing job;
new hash → supersede.
GET /jobs/{job_id}/ poll job status
POST /jobs/{job_id}/retry/ re-dispatch a failed job
GET /jobs/?status=&library_uid= list recent jobs
Workspace-Library lookup uses the unique workspace_id index added in the
schema commit. Concept GC runs as a separate transaction after item/chunk
delete so partial failures don't leave the global graph corrupted.
Tests cover serializer validation, IngestJob ORM behavior, the
(library, source_ref, hash) idempotency query pattern, and auth
boundaries on every new endpoint. Cypher correctness is validated by
manual end-to-end testing — no live Neo4j in unit tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds DAEDALUS_S3_* settings (read-only credentials for the Daedalus bucket)
and a small `daedalus_s3.py` helper that fetches a file from Daedalus's
bucket and writes it into Mnemosyne's bucket via default_storage.
Adds the Celery task `library.tasks.ingest_from_daedalus`. Given an
IngestJob row, it:
1. Resolves the target Library (by library_uid).
2. Supersedes a prior Item with the same source_ref but different
content_hash by deleting the old Item + chunks first.
3. Fetches from Daedalus S3, copies into items/{item_uid}/original.{ext}.
4. Creates the Item node, links it to a default Collection.
5. Runs the existing EmbeddingPipeline.process_item.
6. Marks the job completed with chunks/concepts counts.
Failures retry up to 3× with exponential backoff; final failure marks
the job failed with the exception text. Routed to the embedding queue
so single-worker setups must consume it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two new content-type-aware library types — `business` for
proposals/marketing/strategy (used by the work-team agents) and `finance`
for statements/tax/market commentary (used by Garth). Each ships with
chunking config, embedding/reranker instructions, an LLM-context prompt
that forbids fabricating financial figures, and a vision prompt.
Adds a unique-indexed `workspace_id` property to `Library` so a node
can be scoped to a Daedalus workspace. Null means a global library;
non-null means workspace-scoped. Search Cypher (added in a later
commit) enforces the boundary.
Adds an `IngestJob` Django ORM model — separate from neomodel — that
tracks asynchronous ingestion lifecycle (Daedalus → S3 → Celery →
embedding pipeline) with idempotency on (library, source_ref, hash).
Migration 0001_initial creates the table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace OPTIONAL MATCH with MATCH for Library-Collection-Item paths to
ensure results are properly scoped to libraries, and remove per-query
score normalization since RRF fuses results by rank rather than score
magnitude.
Implement hybrid search pipeline combining vector, fulltext, and graph
search across Neo4j, with cross-attention reranking via Synesis
(Qwen3-VL-Reranker-2B) `/v1/rerank` endpoint.
- Add SearchService with vector, fulltext, and graph search strategies
- Add SynesisRerankerClient for multimodal reranking via HTTP API
- Add search API endpoint (POST /search/) with filtering by library,
collection, and library_type
- Add SearchRequest/Response serializers and image search results
- Add "nonfiction" to library_type choices
- Consolidate reranker stack from two models to single Synesis service
- Handle image analysis_status as "skipped" when analysis is unavailable
- Add comprehensive tests for search pipeline and reranker client
- Introduced a new vision analysis service to classify, describe, and extract text from images.
- Enhanced the Image model with fields for OCR text, vision model name, and analysis status.
- Added a new "nonfiction" library type with specific chunking and embedding configurations.
- Updated content types to include vision prompts for various library types.
- Integrated vision analysis into the embedding pipeline, allowing for image analysis during document processing.
- Implemented metrics to track vision analysis performance and usage.
- Updated UI components to display vision analysis results and statuses in item details and the embedding dashboard.
- Added migration for new vision model fields and usage tracking.
- Implemented custom form widgets for date, time, and datetime fields with DaisyUI styling.
- Created utility functions for formatting dates, times, and numbers according to user preferences.
- Developed views for profile settings, API key management, and notifications, including health check endpoints.
- Added URL configurations for Themis tests and main application routes.
- Established test cases for custom widgets to ensure proper functionality and integration.
- Defined project metadata and dependencies in pyproject.toml for package management.
