Libraries created through the API are now stamped with the name of the
UserToken that created them (Library.managed_by), on both the workspace
and plain create endpoints; web-session creates stay null/unmanaged. The
idempotent workspace re-POST lazily backfills null managed_by, and a
one-off backfill_managed_by command labels pre-existing rows (workspace
inference: kairos-mail-* → Kairos, else Daedalus; Spelunker via ingest
job provenance).
UI badges and warnings now render "Managed by <app>" via
managed_by_display (inference fallback keeps legacy rows accurate before
backfill). The list scope filter becomes all/managed/unmanaged with the
old daedalus/global values aliased for bookmarks. The plain create
endpoint also gains an explicit 409 name_conflict (previously a raw
UniqueProperty 500) reporting the existing library's uid and manager.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"svg" was in both IMAGE_EXTENSIONS and PYMUPDF_EXTENSIONS, and the image
check ran first, so every SVG reached PIL.Image.open — which cannot decode
vector XML. Ingest raised UnidentifiedImageError, logged "Failed to read
image file_type=svg", and counted documents_parsed_total{status="error"}.
SVG ingest has never worked.
Rasterize to PNG instead, via PyMuPDF (already a dependency). This also
unblocks the vision stage downstream: it sends images to a vision LLM as a
data: URI, which cannot carry image/svg+xml either, so the description and
OCR fields were unreachable for SVG regardless of the parse fix.
Route SVG explicitly before both extension sets. Falling through to
PYMUPDF_EXTENSIONS would parse but not split, and a multi-page Write note
rendered as one document is either blank (no root width/height, so MuPDF
falls back to US-Letter and emits the top-left corner) or an illegible tall
strip (root sized across every stacked page). Splitting per write-page
element is correct for both formats, and since Write never rewrites
existing files both persist indefinitely.
svg_raster is a deliberate twin of daedalus's extraction/svg.py — the repos
share no common package, so the duplication is noted in both docstrings and
fixes belong in both.
lxml is a new dependency: recover=True is needed for the real files that
aren't well-formed XML, and stdlib ElementTree has no equivalent.
Kairos is about to provision per-user app-managed (workspace) libraries for
synced mail, and none of the existing types fit correspondence. New entry in
LIBRARY_TYPE_DEFAULTS mirrors journal's entry-level chunking (one message ≈
one entry) with instructions tuned for email: correspondents, dates,
requests/commitments, and quoted-reply context. Registered in the Library
node choices and the API serializer choice list; forms and
load_library_types derive from the registry, and the compose init sidecar
re-runs the seeder on deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kairos is about to start announcing rendered mail documents to
POST /library/api/ingest/ with source="kairos-mail". Without a registry
entry, unknown sources fall back to the daedalus bucket and every worker
fetch fails — this adds the KAIROS_S3_* env vars (pattern-copy of the
spelunker block) and the SOURCE_S3_BUCKETS["kairos-mail"] entry, plumbed
through .env.example and the worker service in docker-compose.
Settings-only; no code changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The exec-form CMD has no shell, so $$HOSTNAME was never expanded,
causing "celery@$HOSTNAME" to match no node and every healthcheck to
fail. Remove the -d filter (one worker per container makes it
unnecessary) and add -t 8 to accommodate broker round-trip latency.
Add a `scope` GET parameter to `library_list` allowing users to filter
libraries by all, global-only, or Daedalus workspace-only. Includes a
filter form in the template, an updated empty state message, and tests
covering each scope branch with mocked Neo4j dependencies.
The backend already allows admin delete of a workspace-scoped Library via
the shared delete_library_cascade (commit 142e967), and the confirm-delete
page carries the Daedalus caution. But the detail template was never updated
to match: the Delete button stayed hardcoded `disabled` for any library with
a workspace_id, and the banner said "Do not delete it manually." So the
working route was unreachable from the UI — admins still could not clear an
orphaned Library blocking workspace re-registration.
Make Delete an unconditional enabled link to library:library-delete (the
confirm page already warns), and rewrite the workspace banner to describe the
escape hatch (deleting here removes embedded content; Daedalus recreates and
re-embeds on the next sync) instead of forbidding it.
Template-only change; no view/model/migration changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the pull_request:[main] trigger so the CVE scan + Docker build runs
only when changes land on main, not when a PR is opened against it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A recreate of a workspace whose Mnemosyne Library was orphaned (left behind
by a failed Daedalus delete-propagate) collides on the global Library.name
unique constraint. neomodel raised UniqueProperty unguarded, so workspace_create
500'd and ingest then 404'd forever — the queue froze silently.
Guard lib.save() and return a structured 409 with a machine code so Daedalus
can classify the failure without string-matching:
- name_conflict — the new name-collision case
- owner_conflict, library_type_immutable — codes added to the two existing 409s
Cypher-touching paths stay covered by the manual end-to-end plan, per the
test module's stated convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Several library tests patched symbols at import paths that no longer
expose them, so they errored (AttributeError) instead of testing anything
— giving false confidence. The underlying code is correct; only the test
patch targets were stale after earlier refactors moved imports
function-local.
- test_pipeline: patch source modules (library.models.Item,
llm_manager.models.LLMModel, library.services.parsers.DocumentParser,
.chunker.ContentTypeChunker, .embedding_client.EmbeddingClient,
.vision.VisionAnalyzer, .concepts.ConceptExtractor) since pipeline.py
imports them inside methods. default_storage stays (still module-level).
- test_search_api: patch library.services.search.SearchService (the view
imports it function-local).
- test_tasks: patch library.services.pipeline.EmbeddingPipeline (tasks.py
imports it function-local).
- test_search_views_admin_scope: patch library.utils.neo4j_available; the
guard moved to utils when views._all_library_uids became a thin alias.
- test_concepts: remove SampleIndexSelectionTests — _select_sample_indices
was deleted in the document-level concept-extraction refactor (dead test).
Not addressed here: SearchAPIAuthTest / SearchAPIValidationTest return 302
instead of 401/400. Static analysis ruled out routing, middleware, and DRF
config; reproducing needs a running server (DB-backed). Flagged for sandbox
diagnosis — not a stale-patch issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin/HTML library delete previously hard-blocked workspace-scoped
(Daedalus-managed) libraries, leaving no way to clear an orphaned Library
node — e.g. one left behind when a Daedalus workspace delete failed to
propagate. A recreate of that workspace then collides on the global
Library.name unique constraint and 500s, freezing ingest.
Allow the delete behind the existing confirm warning (low risk: source
content lives in Daedalus and is recreated + re-embedded on next sync),
and route both the API and HTML delete paths through one shared cascade.
- Add library/services/library_delete.delete_library_cascade(lib), keyed on
Library uid so it covers global and workspace-scoped libraries. It removes
Chunks, Images/ImageEmbeddings, Items, Collections, the Library, then GCs
orphan-only Concepts (verbatim from the API view, re-keyed workspace_id->uid).
- workspace_detail_or_delete (API) now calls the shared helper.
- library_delete (HTML) no longer blocks workspace_id libraries; it calls the
cascade instead of a bare lib.delete() (which leaked child nodes — also a
latent bug for global libraries with content).
- Confirm-delete template shows a caution banner for Daedalus-managed libraries.
No migration: Mnemosyne library data is in Neo4j (neomodel); no schema change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Prometheus custom collector that probes the four system-default
models (chat, vision, embedding, reranker) at /metrics scrape time and
emits up/down, configured, and probe-latency gauges. This complements
the ingest-pipeline counters in the Celery worker, which only move
during active ingests and cannot signal model outages on an idle queue.
- New `library/health_collector.py` registers a custom collector with
a 55s in-process cache to avoid hammering GPU endpoints on rapid
scrapes or across multiple gunicorn workers.
- New `library/services/model_health.py` centralises the probe logic,
resolving system-default models via SystemSettings and dispatching
to chat/embedding/rerank endpoints with a short timeout.
- Register the collector only in the web process (gunicorn/runserver)
via `LibraryConfig.ready`, excluding Celery, pytest, and management
commands to prevent duplicate registration and stray probes.
- Add unit tests covering the collector cache, metric shape, and
per-role probe dispatch.
- Configure nginx `set_real_ip_from` for RFC1918 ranges and enable
`real_ip_recursive` so allowlists evaluate the true client IP
instead of Docker's NAT gateway, preventing public exposure of
`/metrics` and `/nginx_status`
- Update published port from 23181 to 23081 in docker-compose
Generalises the Daedalus-only cross-bucket fetch into a registry
(SOURCE_S3_BUCKETS) keyed on the IngestJob `source` field, so new
upstream sources (Spelunker) can ingest from their own buckets. The
ingest task now calls fetch_from_source(job.source, job.s3_key) and
falls back to "daedalus" for blank/unknown sources (backwards compatible).
Adds SPELUNKER_S3_* env vars and worker env scoping. Replaces
daedalus_s3.py with source_s3.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concept extraction was making up to 10 LLM calls per item by sampling
chunks, which produced redundant work (the same concept reappears in
multiple chunks), context-loss bugs (chunk boundaries cut mid-thought),
and on a 35B model dominated per-item wall time (~3 min/item).
Concepts are document-level semantic objects; chunks are retrieval
units. Extract once per item from the first 100KB of parsed document
text, then connect each chunk to the concepts it explicitly mentions
via case-insensitive substring match — no extra LLM calls. Drops the
sample-indices selector that the old per-chunk loop relied on.
Stage 7 is currently dormant in production because the configured
chat model is a reasoning-mode Qwen variant that returns empty content
on every call (output stuck in reasoning_content). Re-enables cleanly
once a non-reasoning instruct model is set as is_system_chat_model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The OpenAI SDK used by _discover_openai_models tolerates a base_url
without /v1 (it auto-adds it for the probe), but every runtime client
(embedding_client, vision, concepts, reranker) treats base_url as the
/v1 root and appends path-only segments. A non-conforming base_url
silently passed Test & Discover and then 404'd at embed/chat/rerank
time.
Add _check_openai_v1_convention() which probes {base_url}/v1/models
when the URL doesn't end in /v1; on 200, fail the test with an
explicit "set base_url to .../v1 and re-test" message that points at
the exact bare-vs-/v1 mismatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Rename MCPToken to UserToken across models, views, and tests
- Update URL names from mcp-token-* to token-*
- Add Daedalus/Pallas integration design doc (v2)
- Switch docker-compose to build local mnemosyne:local image via shared
build config instead of pulling from git.helu.ca
- Add Gitea Actions workflow to build and deploy docs on push to main
- Generate Sphinx reference documentation for all apps and modules
- Deploy versioned and latest docs via rsync over SSH
- Add read-only ModelAdmin for IngestJob with filters, search, and
date hierarchy for operational visibility
- Inject proxy entries into the admin index for Neo4j-backed entities
(Libraries, Concepts, Search, Embedding pipeline) that link to
existing CRUD views in library/views.py
- Makes library content discoverable from /admin/ without pretending
neomodel StructuredNodes are Django ORM models
Increase max_length for source and file_type fields in IngestJob model from 50 to 100.
This prevents data truncation for longer source references or file type strings.
Update STATIC_ROOT and MEDIA_ROOT in settings.py to read from
environment variables with default fallbacks to BASE_DIR paths.
This allows flexible deployment configurations without modifying
source code for different environments.
The static volume is now Docker-managed, removing the need for Ansible to create the host path. Media volume comments updated to reflect S3 storage usage (USE_LOCAL_STORAGE=False) and that the volume is effectively unused in production.
Add /mcp/health to suppress paths in log_filters.py to demote health
probe logs to DEBUG level. Configure uvicorn.access logger in settings.py
to manage access logs directly instead of relying on mcp_server internal
filters. Update comments to reflect that uvicorn access is now managed
in project settings.
Move probe execution from Django app ready() to gunicorn.conf.py
Remove threading implementation to simplify startup sequence
Ensure probe runs in worker process context with proper error handling
Move the _run_startup_probe logic into a separate daemon thread
within LibraryConfig.ready. This prevents indefinite blocking on
startup while maintaining a 10-second wait for the probe result.
Remove dedicated static-init service and run collectstatic in the init sidecar instead.
Static files baked into the image are copied to /mnt/static for nginx serving on each
deployment. Also update MCP and nginx ports and refresh external service hostnames
in comments.
Update all authentication-related template URLs from Django's default auth
URL names ('login', 'password_reset') to django-allauth's URL names
('account_login', 'account_reset_password') for consistency with the
authentication backend migration.
