18 Commits

Author SHA1 Message Date
0a14cf00c5 🐾 feat(library): per-app managed_by replaces hardcoded Daedalus badge
Libraries created through the API are now stamped with the name of the
UserToken that created them (Library.managed_by), on both the workspace
and plain create endpoints; web-session creates stay null/unmanaged. The
idempotent workspace re-POST lazily backfills null managed_by, and a
one-off backfill_managed_by command labels pre-existing rows (workspace
inference: kairos-mail-* → Kairos, else Daedalus; Spelunker via ingest
job provenance).

UI badges and warnings now render "Managed by <app>" via
managed_by_display (inference fallback keeps legacy rows accurate before
backfill). The list scope filter becomes all/managed/unmanaged with the
old daedalus/global values aliased for bookmarks. The plain create
endpoint also gains an explicit 409 name_conflict (previously a raw
UniqueProperty 500) reporting the existing library's uid and manager.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-02 12:39:10 -04:00
9f5df20d2b Merge pull request '🐾 fix(parsers): rasterize SVG instead of failing ingest' (#6) from fix/svg-ingest-rasterize into main
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 3m16s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m10s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m56s
Reviewed-on: #6
2026-07-26 12:03:42 +00:00
224541a4ce 🐾 fix(parsers): rasterize SVG instead of failing ingest
"svg" was in both IMAGE_EXTENSIONS and PYMUPDF_EXTENSIONS, and the image
check ran first, so every SVG reached PIL.Image.open — which cannot decode
vector XML. Ingest raised UnidentifiedImageError, logged "Failed to read
image file_type=svg", and counted documents_parsed_total{status="error"}.
SVG ingest has never worked.

Rasterize to PNG instead, via PyMuPDF (already a dependency). This also
unblocks the vision stage downstream: it sends images to a vision LLM as a
data: URI, which cannot carry image/svg+xml either, so the description and
OCR fields were unreachable for SVG regardless of the parse fix.

Route SVG explicitly before both extension sets. Falling through to
PYMUPDF_EXTENSIONS would parse but not split, and a multi-page Write note
rendered as one document is either blank (no root width/height, so MuPDF
falls back to US-Letter and emits the top-left corner) or an illegible tall
strip (root sized across every stacked page). Splitting per write-page
element is correct for both formats, and since Write never rewrites
existing files both persist indefinitely.

svg_raster is a deliberate twin of daedalus's extraction/svg.py — the repos
share no common package, so the duplication is noted in both docstrings and
fixes belong in both.

lxml is a new dependency: recover=True is needed for the real files that
aren't well-formed XML, and stdlib ElementTree has no equivalent.
2026-07-26 07:41:20 -04:00
3ae5adebed Merge pull request '🐾 feat(library): add 'email' library type' (#4) from feat/email-library-type into main
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 4m27s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m27s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m30s
Reviewed-on: #4
2026-07-12 12:20:48 +00:00
9f20110f56 🐾 feat(library): add 'email' library type
Kairos is about to provision per-user app-managed (workspace) libraries for
synced mail, and none of the existing types fit correspondence. New entry in
LIBRARY_TYPE_DEFAULTS mirrors journal's entry-level chunking (one message ≈
one entry) with instructions tuned for email: correspondents, dates,
requests/commitments, and quoted-reply context. Registered in the Library
node choices and the API serializer choice list; forms and
load_library_types derive from the registry, and the compose init sidecar
re-runs the seeder on deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 08:03:28 -04:00
d6b541636a Merge pull request '🐾 feat(settings): register kairos-mail source bucket for ingest' (#3) from feat/kairos-mail-bucket into main
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 4m22s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m26s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m53s
Reviewed-on: #3
2026-07-12 11:50:53 +00:00
d01afd6203 🐾 feat(settings): register kairos-mail source bucket for ingest
Kairos is about to start announcing rendered mail documents to
POST /library/api/ingest/ with source="kairos-mail". Without a registry
entry, unknown sources fall back to the daedalus bucket and every worker
fetch fails — this adds the KAIROS_S3_* env vars (pattern-copy of the
spelunker block) and the SOURCE_S3_BUCKETS["kairos-mail"] entry, plumbed
through .env.example and the worker service in docker-compose.

Settings-only; no code changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 07:36:12 -04:00
840b9435a3 ci: remove branch and PR tags from docker metadata
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 3m25s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m34s
Drop type=ref tags for branch and PR events, keeping only SHA-based
and latest tags to simplify the image tagging scheme.
2026-06-25 08:14:56 -04:00
6120e9cd1f fix(mnemosyne): correct celery healthcheck command
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 4m14s
CVE Scan & Docker Build / build-and-push (push) Successful in 47s
The exec-form CMD has no shell, so $$HOSTNAME was never expanded,
causing "celery@$HOSTNAME" to match no node and every healthcheck to
fail. Remove the -d filter (one worker per container makes it
unnecessary) and add -t 8 to accommodate broker round-trip latency.
2026-06-19 11:39:38 -04:00
31a98b4f3a feat(library): add workspace scope filter to library list view
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 3m52s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m15s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m43s
Add a `scope` GET parameter to `library_list` allowing users to filter
libraries by all, global-only, or Daedalus workspace-only. Includes a
filter form in the template, an updated empty state message, and tests
covering each scope branch with mocked Neo4j dependencies.
2026-06-19 06:45:19 -04:00
3394726ca1 Merge branch 'main' of ssh://git.helu.ca:22022/r/mnemosyne
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 4m5s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m26s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m35s
2026-06-19 06:30:21 -04:00
03e3155bd6 fix(settings): load .env from repo root instead of project dir 2026-06-19 06:30:07 -04:00
929a3c8c3c fix(library): expose Delete control for Daedalus-managed libraries in UI
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 3m37s
Build & Deploy Docs / build-and-deploy (push) Successful in 1m9s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m56s
The backend already allows admin delete of a workspace-scoped Library via
the shared delete_library_cascade (commit 142e967), and the confirm-delete
page carries the Daedalus caution. But the detail template was never updated
to match: the Delete button stayed hardcoded `disabled` for any library with
a workspace_id, and the banner said "Do not delete it manually." So the
working route was unreachable from the UI — admins still could not clear an
orphaned Library blocking workspace re-registration.

Make Delete an unconditional enabled link to library:library-delete (the
confirm page already warns), and rewrite the workspace banner to describe the
escape hatch (deleting here removes embedded content; Daedalus recreates and
re-embeds on the next sync) instead of forbidding it.

Template-only change; no view/model/migration changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 23:03:47 -04:00
2af72d6e82 ci: build only on push to main, not on pull_request
All checks were successful
CVE Scan & Docker Build / security-scan (push) Successful in 3m30s
CVE Scan & Docker Build / build-and-push (push) Successful in 2m33s
Drop the pull_request:[main] trigger so the CVE scan + Docker build runs
only when changes land on main, not when a PR is opened against it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 06:14:52 -04:00
70b1fc510b Merge pull request 'fix(tests): repair stale mock.patch targets after service refactors' (#2) from fix/stale-test-patch-targets into main
Some checks failed
CVE Scan & Docker Build / security-scan (push) Has been cancelled
CVE Scan & Docker Build / build-and-push (push) Has been cancelled
Build & Deploy Docs / build-and-deploy (push) Successful in 1m11s
Reviewed-on: #2
2026-06-18 02:01:25 +00:00
46ca2a934d Merge pull request 'feat/workspace-name-conflict-409' (#1) from feat/workspace-name-conflict-409 into main
Some checks failed
CVE Scan & Docker Build / security-scan (push) Has been cancelled
CVE Scan & Docker Build / build-and-push (push) Has been cancelled
Build & Deploy Docs / build-and-deploy (push) Has been cancelled
Reviewed-on: #1
2026-06-18 02:00:55 +00:00
dd06f923cd feat(workspaces): return 409 name_conflict instead of 500 on Library name clash
Some checks failed
CVE Scan & Docker Build / security-scan (pull_request) Successful in 3m49s
CVE Scan & Docker Build / build-and-push (pull_request) Has been cancelled
A recreate of a workspace whose Mnemosyne Library was orphaned (left behind
by a failed Daedalus delete-propagate) collides on the global Library.name
unique constraint. neomodel raised UniqueProperty unguarded, so workspace_create
500'd and ingest then 404'd forever — the queue froze silently.

Guard lib.save() and return a structured 409 with a machine code so Daedalus
can classify the failure without string-matching:
- name_conflict   — the new name-collision case
- owner_conflict, library_type_immutable — codes added to the two existing 409s

Cypher-touching paths stay covered by the manual end-to-end plan, per the
test module's stated convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 20:26:43 -04:00
142e9675b5 feat(library): allow admin delete of Daedalus-managed library via shared cascade
Admin/HTML library delete previously hard-blocked workspace-scoped
(Daedalus-managed) libraries, leaving no way to clear an orphaned Library
node — e.g. one left behind when a Daedalus workspace delete failed to
propagate. A recreate of that workspace then collides on the global
Library.name unique constraint and 500s, freezing ingest.

Allow the delete behind the existing confirm warning (low risk: source
content lives in Daedalus and is recreated + re-embedded on next sync),
and route both the API and HTML delete paths through one shared cascade.

- Add library/services/library_delete.delete_library_cascade(lib), keyed on
  Library uid so it covers global and workspace-scoped libraries. It removes
  Chunks, Images/ImageEmbeddings, Items, Collections, the Library, then GCs
  orphan-only Concepts (verbatim from the API view, re-keyed workspace_id->uid).
- workspace_detail_or_delete (API) now calls the shared helper.
- library_delete (HTML) no longer blocks workspace_id libraries; it calls the
  cascade instead of a bare lib.delete() (which leaked child nodes — also a
  latent bug for global libraries with content).
- Confirm-delete template shows a caution banner for Daedalus-managed libraries.

No migration: Mnemosyne library data is in Neo4j (neomodel); no schema change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 19:37:58 -04:00
26 changed files with 1367 additions and 111 deletions

View File

@@ -91,6 +91,19 @@ SPELUNKER_S3_REGION_NAME=us-east-1
SPELUNKER_S3_USE_SSL=True
SPELUNKER_S3_VERIFY=True
# --- Kairos S3 (cross-bucket reads for ingest, source="kairos-mail") ---
# Consumed by: worker only
# Kairos renders synced mail to text documents in its own bucket and posts
# ingest requests with source="kairos-mail". These creds should be scoped
# read-only to the Kairos bucket in your secret manager.
KAIROS_S3_ENDPOINT_URL=https://nyx.helu.ca:8555
KAIROS_S3_ACCESS_KEY_ID=
KAIROS_S3_SECRET_ACCESS_KEY=
KAIROS_S3_BUCKET_NAME=kairos
KAIROS_S3_REGION_NAME=us-east-1
KAIROS_S3_USE_SSL=True
KAIROS_S3_VERIFY=True
# --- Celery / RabbitMQ (Oberon) ---------------------------------------------
# Consumed by: app (producer), worker (consumer). NOT mcp.
# Remember to percent-encode any password characters that have meaning in a

View File

@@ -3,8 +3,6 @@ name: CVE Scan & Docker Build
on:
push:
branches: [main]
pull_request:
branches: [main]
env:
REGISTRY: git.helu.ca
@@ -75,8 +73,6 @@ jobs:
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha,prefix=
type=raw,value=latest,enable=${{ gitea.ref == 'refs/heads/main' }}

View File

@@ -346,6 +346,13 @@ services:
- SPELUNKER_S3_REGION_NAME=${SPELUNKER_S3_REGION_NAME}
- SPELUNKER_S3_USE_SSL=${SPELUNKER_S3_USE_SSL}
- SPELUNKER_S3_VERIFY=${SPELUNKER_S3_VERIFY}
- KAIROS_S3_ENDPOINT_URL=${KAIROS_S3_ENDPOINT_URL}
- KAIROS_S3_ACCESS_KEY_ID=${KAIROS_S3_ACCESS_KEY_ID}
- KAIROS_S3_SECRET_ACCESS_KEY=${KAIROS_S3_SECRET_ACCESS_KEY}
- KAIROS_S3_BUCKET_NAME=${KAIROS_S3_BUCKET_NAME}
- KAIROS_S3_REGION_NAME=${KAIROS_S3_REGION_NAME}
- KAIROS_S3_USE_SSL=${KAIROS_S3_USE_SSL}
- KAIROS_S3_VERIFY=${KAIROS_S3_VERIFY}
# Celery / RabbitMQ
- CELERY_BROKER_URL=${CELERY_BROKER_URL}
- CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
@@ -371,7 +378,12 @@ services:
volumes:
- media:/mnt/media
healthcheck:
test: ["CMD", "celery", "-A", "mnemosyne", "inspect", "ping", "-d", "celery@$$HOSTNAME"]
# No -d destination: exec-form CMD has no shell, so $$HOSTNAME never
# expanded and the literal "celery@$HOSTNAME" matched no node → every
# check failed. There's one worker per container, so an unfiltered ping
# (any node replies = healthy) is correct. -t gives the reply room to
# round-trip through the broker on Oberon (~450ms observed) under jitter.
test: ["CMD", "celery", "-A", "mnemosyne", "inspect", "ping", "-t", "8"]
interval: 60s
timeout: 10s
retries: 3

View File

@@ -15,6 +15,7 @@ LIBRARY_TYPE_CHOICES = [
"film",
"art",
"journal",
"email",
"business",
"finance",
]
@@ -36,6 +37,7 @@ class LibrarySerializer(serializers.Serializer):
required=False, allow_blank=True, default=""
)
workspace_id = serializers.CharField(read_only=True)
managed_by = serializers.CharField(read_only=True)
created_at = serializers.DateTimeField(read_only=True)
@@ -192,6 +194,7 @@ class WorkspaceStatusSerializer(serializers.Serializer):
name = serializers.CharField()
library_type = serializers.CharField()
description = serializers.CharField(allow_blank=True)
managed_by = serializers.CharField(allow_null=True, required=False)
item_count = serializers.IntegerField()
chunk_count = serializers.IntegerField()
created_at = serializers.DateTimeField()

View File

@@ -17,6 +17,7 @@ from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from library.content_types import get_library_type_config
from mcp_server.drf_auth import request_token_label
from .serializers import (
CollectionSerializer,
@@ -84,6 +85,28 @@ def library_list_create(request):
serializer.is_valid(raise_exception=True)
data = serializer.validated_data
# Library.name is globally unique; reject collisions with a clean 409
# (uid + managed_by included so the caller can say who owns the name)
# instead of letting the unique-index save raise a 500.
try:
existing = Library.nodes.get(name=data["name"])
except Library.DoesNotExist:
existing = None
if existing is not None:
logger.warning(
"library_create name_conflict name=%s existing_uid=%s caller=%s",
data["name"], existing.uid, request.user.username,
)
return Response(
{
"detail": f"A library named '{data['name']}' already exists.",
"code": "name_conflict",
"uid": existing.uid,
"managed_by": existing.managed_by_display or None,
},
status=status.HTTP_409_CONFLICT,
)
# Populate defaults from content-type config if not provided
library_type = data["library_type"]
defaults = get_library_type_config(library_type)
@@ -92,6 +115,7 @@ def library_list_create(request):
name=data["name"],
library_type=library_type,
description=data.get("description", ""),
managed_by=request_token_label(request),
chunking_config=data.get("chunking_config") or defaults["chunking_config"],
embedding_instruction=(
data.get("embedding_instruction") or defaults["embedding_instruction"]

View File

@@ -17,12 +17,15 @@ across users.
import logging
from neomodel import db
from neomodel.exceptions import UniqueProperty
from rest_framework import status
from rest_framework.decorators import api_view, permission_classes
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from library.content_types import get_library_type_config
from library.services.library_delete import delete_library_cascade
from mcp_server.drf_auth import request_token_label
from .serializers import WorkspaceCreateSerializer, WorkspaceStatusSerializer
@@ -47,6 +50,7 @@ def _serialize_workspace(lib):
"name": lib.name,
"library_type": lib.library_type,
"description": lib.description or "",
"managed_by": lib.managed_by,
"item_count": item_count,
"chunk_count": chunk_count,
"created_at": lib.created_at,
@@ -84,7 +88,10 @@ def workspace_create(request):
data["workspace_id"], request.user.username,
)
return Response(
{"detail": "Workspace id is already in use."},
{
"detail": "Workspace id is already in use.",
"code": "owner_conflict",
},
status=status.HTTP_409_CONFLICT,
)
if existing.library_type != data["library_type"]:
@@ -94,10 +101,23 @@ def workspace_create(request):
"library_type is immutable for an existing workspace "
f"(have '{existing.library_type}', "
f"got '{data['library_type']}')."
)
),
"code": "library_type_immutable",
},
status=status.HTTP_409_CONFLICT,
)
# Lazy backfill: pre-managed_by libraries pick up the label from
# the first idempotent re-POST. Null-only — an already-stamped
# library never changes manager.
if not existing.managed_by:
label = request_token_label(request)
if label:
existing.managed_by = label
existing.save()
logger.info(
"Backfilled managed_by=%s workspace_id=%s library_uid=%s",
label, existing.workspace_id, existing.uid,
)
logger.info(
"Workspace already exists workspace_id=%s library_uid=%s",
data["workspace_id"], existing.uid,
@@ -114,12 +134,35 @@ def workspace_create(request):
description=data.get("description", ""),
workspace_id=data["workspace_id"],
owner_username=request.user.username,
managed_by=request_token_label(request),
chunking_config=defaults["chunking_config"],
embedding_instruction=defaults["embedding_instruction"],
reranker_instruction=defaults["reranker_instruction"],
llm_context_prompt=defaults["llm_context_prompt"],
)
lib.save()
try:
lib.save()
except UniqueProperty:
# Library.name is globally unique. A name collision here almost always
# means an orphaned Library survived a failed Daedalus workspace delete
# (the old node kept the name), and the recreate under a new
# workspace_id now clashes. Surface a clean 409 instead of a 500 so
# Daedalus can record + report it; the operator clears the orphan
# (admin delete) or renames the workspace.
logger.warning(
"workspace_create name_conflict workspace_id=%s name=%s",
data["workspace_id"], data["name"],
)
return Response(
{
"detail": (
f"A library named '{data['name']}' already exists in "
"Mnemosyne."
),
"code": "name_conflict",
},
status=status.HTTP_409_CONFLICT,
)
logger.info(
"Workspace created workspace_id=%s library_uid=%s library_type=%s",
data["workspace_id"], lib.uid, lib.library_type,
@@ -165,74 +208,15 @@ def workspace_detail_or_delete(request, workspace_id):
if lib is None:
return Response(status=status.HTTP_204_NO_CONTENT)
library_uid = lib.uid
library_name = lib.name
# Step 1-4: delete chunks, items, collections, then the library itself.
# We collect Item s3_keys first so the caller can clean up S3
# asynchronously (a future enhancement — for now, the keys are logged).
s3_rows, _ = db.cypher_query(
"MATCH (l:Library {workspace_id: $wsid})-[:CONTAINS]->(:Collection)"
"-[:CONTAINS]->(i:Item) RETURN i.uid, i.s3_key",
{"wsid": workspace_id},
)
item_s3_keys = [(r[0], r[1]) for r in s3_rows if r[1]]
db.cypher_query(
"""
MATCH (l:Library {workspace_id: $wsid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)-[:HAS_CHUNK]->(c:Chunk)
DETACH DELETE c
""",
{"wsid": workspace_id},
)
db.cypher_query(
"""
MATCH (l:Library {workspace_id: $wsid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)-[:HAS_IMAGE]->(img:Image)
OPTIONAL MATCH (img)-[:HAS_EMBEDDING]->(emb:ImageEmbedding)
DETACH DELETE img, emb
""",
{"wsid": workspace_id},
)
db.cypher_query(
"""
MATCH (l:Library {workspace_id: $wsid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)
DETACH DELETE i
""",
{"wsid": workspace_id},
)
db.cypher_query(
"""
MATCH (l:Library {workspace_id: $wsid})-[:CONTAINS]->(col:Collection)
DETACH DELETE col
""",
{"wsid": workspace_id},
)
db.cypher_query(
"MATCH (l:Library {workspace_id: $wsid}) DETACH DELETE l",
{"wsid": workspace_id},
)
# Step 5: orphan Concept garbage collection.
orphan_result, _ = db.cypher_query(
"""
MATCH (con:Concept)
WHERE NOT (con)<-[:REFERENCES]-() AND NOT (con)<-[:MENTIONS]-()
AND NOT (con)<-[:DEPICTS]-()
WITH con
DETACH DELETE con
RETURN count(con) AS deleted
"""
)
orphans_deleted = orphan_result[0][0] if orphan_result else 0
# Delete the Library and everything reachable + unique to it, plus
# orphan-Concept GC. Shared with the admin/HTML delete path.
result = delete_library_cascade(lib)
logger.info(
"Workspace deleted workspace_id=%s library_uid=%s name=%s "
"items=%d orphans_deleted=%d",
workspace_id, library_uid, library_name,
len(item_s3_keys), orphans_deleted,
workspace_id, result["library_uid"], result["name"],
result["item_count"], result["orphans_deleted"],
)
return Response(status=status.HTTP_204_NO_CONTENT)

View File

@@ -241,6 +241,38 @@ LIBRARY_TYPE_DEFAULTS = {
"4) The commercial purpose — positioning, pricing, capability demonstration."
),
},
"email": {
"chunking_config": {
"strategy": "entry_level",
"chunk_size": 512,
"chunk_overlap": 32,
"respect_boundaries": ["message", "quote", "paragraph"],
},
"embedding_instruction": (
"Represent this email message for retrieval. "
"Focus on the sender, recipients, subject, dates, requests and "
"commitments made, and the people, organizations, and events discussed."
),
"reranker_instruction": (
"Re-rank email messages based on relevance to the query. "
"Prioritize messages matching the correspondents, subject matter, "
"time period, and any specific commitments or requests mentioned."
),
"llm_context_prompt": (
"The following excerpts are from personal email correspondence. "
"This is private content — answer with discretion. Attribute "
"statements to their senders, note dates, and distinguish what was "
"asked from what was agreed. Quoted text below a reply is earlier "
"context, not the sender's own words."
),
"vision_prompt": (
"Analyze this image from an email message. Identify:\n"
"1) Image type (photograph, screenshot, scanned document, chart, signature graphic).\n"
"2) What it depicts — people, places, documents, data.\n"
"3) Any visible text, dates, or figures.\n"
"4) Its role in the message — attachment content, inline illustration, or boilerplate."
),
},
"finance": {
"chunking_config": {
"strategy": "section_aware",
@@ -282,7 +314,7 @@ def get_library_type_config(library_type):
Args:
library_type: One of 'fiction', 'nonfiction', 'technical', 'music',
'film', 'art', 'journal', 'business', 'finance'
'film', 'art', 'journal', 'email', 'business', 'finance'
Returns:
dict with keys: chunking_config, embedding_instruction,

View File

@@ -0,0 +1,111 @@
"""One-off backfill of ``Library.managed_by`` for pre-existing libraries.
``managed_by`` is stamped from the creating API token's name, so
libraries created before the property existed have it null. This
command labels them:
* Workspace-scoped libraries get :func:`infer_legacy_manager`'s answer —
``Kairos`` for ``kairos-mail-*`` workspace ids, ``Daedalus`` otherwise.
* Global libraries that Spelunker ingested into (any ``IngestJob`` with
``source="spelunker"``) get the Spelunker label.
* Everything else stays null (hand-made in the web UI).
Label flags let the operator match the *actual* production token names
so backfilled rows render identically to newly stamped ones.
Idempotent: only null ``managed_by`` is ever written, and the lazy fill
in ``workspace_create`` is also null-only, so re-runs and later API
traffic never overwrite these labels.
"""
from __future__ import annotations
import logging
from django.core.management.base import BaseCommand, CommandError
from library.models import IngestJob
logger = logging.getLogger(__name__)
class Command(BaseCommand):
help = (
"Backfill Library.managed_by for libraries created before the "
"property existed. Workspace libraries are labelled by inference "
"(kairos-mail-* → Kairos, else Daedalus); global libraries with "
"Spelunker ingest jobs get the Spelunker label."
)
def add_arguments(self, parser):
parser.add_argument(
"--daedalus-label", default="Daedalus",
help="Label for non-Kairos workspace libraries (default: Daedalus).",
)
parser.add_argument(
"--kairos-label", default="Kairos",
help="Label for kairos-mail-* workspace libraries (default: Kairos).",
)
parser.add_argument(
"--spelunker-label", default="Spelunker",
help="Label for global libraries with Spelunker ingest jobs "
"(default: Spelunker).",
)
parser.add_argument(
"--dry-run", action="store_true",
help="Report what would be labelled, don't persist.",
)
def handle(self, *args, **options):
try:
from library.models import Library, infer_legacy_manager
except Exception as exc: # pragma: no cover
raise CommandError(
f"Could not import library.models.Library (Neo4j unreachable?): {exc}"
) from exc
overrides = {
"Daedalus": options["daedalus_label"],
"Kairos": options["kairos_label"],
}
spelunker_uids = set(
IngestJob.objects
.filter(source="spelunker")
.values_list("library_uid", flat=True)
.distinct()
)
candidates = list(Library.nodes.filter(managed_by__isnull=True))
to_label = []
for lib in candidates:
label = infer_legacy_manager(lib.workspace_id)
if label:
label = overrides[label]
elif lib.uid in spelunker_uids:
label = options["spelunker_label"]
if label:
to_label.append((lib, label))
self.stdout.write(f"Libraries with null managed_by: {len(candidates)}")
self.stdout.write(
self.style.SUCCESS(f"Will label: {len(to_label)} "
f"(unmanaged, left null: {len(candidates) - len(to_label)})")
)
for lib, label in to_label:
self.stdout.write(f" {lib.uid} {lib.name!r}{label}")
if options["dry_run"]:
self.stdout.write(self.style.WARNING("--dry-run: nothing written."))
return
for lib, label in to_label:
lib.managed_by = label
lib.save()
logger.info(
"backfill_managed_by uid=%s name=%s label=%s",
lib.uid, lib.name, label,
)
self.stdout.write(self.style.SUCCESS("Done."))

View File

@@ -51,6 +51,18 @@ class NearbyImageRel(StructuredRel):
# --- Node models ---
def infer_legacy_manager(workspace_id):
"""Manager label for pre-``managed_by`` workspace libraries, else None.
Kairos mail workspaces are recognisable by their deterministic
``kairos-mail-`` id prefix; every other workspace id is a Daedalus
workspace UUID.
"""
if not workspace_id:
return None
return "Kairos" if workspace_id.startswith("kairos-mail-") else "Daedalus"
class Library(StructuredNode):
"""
Top-level container representing a content library.
@@ -63,6 +75,11 @@ class Library(StructuredNode):
across the whole instance) or *workspace-scoped* (workspace_id set —
visible only to agents inside that Daedalus workspace). Scoping is
enforced structurally by every search query.
Independently of scoping, a library may be *app-managed*
(``managed_by`` set — created through the API by an external app such
as Daedalus, Kairos, or Spelunker, which owns its content lifecycle)
or unmanaged (created by hand in the web UI).
"""
uid = UniqueIdProperty()
@@ -77,6 +94,7 @@ class Library(StructuredNode):
"film": "Film",
"art": "Art",
"journal": "Journal",
"email": "Email",
"business": "Business",
"finance": "Finance",
},
@@ -92,6 +110,12 @@ class Library(StructuredNode):
# this user. Null for global libraries.
owner_username = StringProperty(required=False, index=True)
# Name of the API token that created this library ("Daedalus",
# "Kairos", "Spelunker", ...). Null for libraries created in the
# web UI. Stamped at create time only — token rotation or edits by
# another token never change it.
managed_by = StringProperty(required=False, index=True)
# Content-type configuration
chunking_config = JSONProperty(default={})
embedding_instruction = StringProperty(default="")
@@ -103,6 +127,16 @@ class Library(StructuredNode):
# Relationships
collections = RelationshipTo("Collection", "CONTAINS")
@property
def managed_by_display(self):
"""Managing-app label for display; empty string when unmanaged.
Falls back to inference for workspace libraries created before
``managed_by`` existed, so rendering is identical before and
after the backfill command runs.
"""
return self.managed_by or infer_legacy_manager(self.workspace_id) or ""
def __str__(self):
return f"{self.name} ({self.library_type})"

View File

@@ -0,0 +1,108 @@
"""
Shared Library deletion cascade.
Deletes a Library node and everything reachable AND unique to it
(Collections, Items, Chunks, Images + ImageEmbeddings), then garbage-collects
Concepts that are no longer referenced by any other Library.
Keyed on the Library ``uid`` so it works for *both* global libraries
(``workspace_id`` is null) and workspace-scoped libraries. This is the single
source of truth used by:
* the Daedalus integration API (``DELETE /library/api/workspaces/{id}/``), and
* the admin/HTML delete view (``library_delete``).
Concept-safe: orphan-only Concept GC happens at the end. Concepts still
referenced by another library (workspace or global) are preserved.
"""
import logging
from neomodel import db
logger = logging.getLogger(__name__)
def delete_library_cascade(lib) -> dict:
"""Delete ``lib`` and all content reachable and unique to it.
:param lib: A ``library.models.Library`` node instance.
:returns: Dict with ``library_uid``, ``name``, ``item_count``,
``item_s3_keys`` (list of ``(uid, s3_key)`` for async S3 cleanup),
and ``orphans_deleted`` (Concept GC count).
"""
library_uid = lib.uid
library_name = lib.name
# Collect Item s3_keys first so the caller can clean up S3 asynchronously
# (a future enhancement — for now, the keys are returned/logged).
s3_rows, _ = db.cypher_query(
"MATCH (l:Library {uid: $uid})-[:CONTAINS]->(:Collection)"
"-[:CONTAINS]->(i:Item) RETURN i.uid, i.s3_key",
{"uid": library_uid},
)
item_s3_keys = [(r[0], r[1]) for r in s3_rows if r[1]]
db.cypher_query(
"""
MATCH (l:Library {uid: $uid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)-[:HAS_CHUNK]->(c:Chunk)
DETACH DELETE c
""",
{"uid": library_uid},
)
db.cypher_query(
"""
MATCH (l:Library {uid: $uid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)-[:HAS_IMAGE]->(img:Image)
OPTIONAL MATCH (img)-[:HAS_EMBEDDING]->(emb:ImageEmbedding)
DETACH DELETE img, emb
""",
{"uid": library_uid},
)
db.cypher_query(
"""
MATCH (l:Library {uid: $uid})-[:CONTAINS]->(:Collection)
-[:CONTAINS]->(i:Item)
DETACH DELETE i
""",
{"uid": library_uid},
)
db.cypher_query(
"""
MATCH (l:Library {uid: $uid})-[:CONTAINS]->(col:Collection)
DETACH DELETE col
""",
{"uid": library_uid},
)
db.cypher_query(
"MATCH (l:Library {uid: $uid}) DETACH DELETE l",
{"uid": library_uid},
)
# Orphan Concept garbage collection: drop Concepts no longer referenced
# by any Item (REFERENCES/MENTIONS) or Image (DEPICTS).
orphan_result, _ = db.cypher_query(
"""
MATCH (con:Concept)
WHERE NOT (con)<-[:REFERENCES]-() AND NOT (con)<-[:MENTIONS]-()
AND NOT (con)<-[:DEPICTS]-()
WITH con
DETACH DELETE con
RETURN count(con) AS deleted
"""
)
orphans_deleted = orphan_result[0][0] if orphan_result else 0
logger.info(
"Library cascade-deleted library_uid=%s name=%s items=%d orphans_deleted=%d",
library_uid, library_name, len(item_s3_keys), orphans_deleted,
)
return {
"library_uid": library_uid,
"name": library_name,
"item_count": len(item_s3_keys),
"item_s3_keys": item_s3_keys,
"orphans_deleted": orphans_deleted,
}

View File

@@ -31,8 +31,11 @@ PYMUPDF_EXTENSIONS = {
# Plain text extensions — read directly, no PyMuPDF needed
PLAINTEXT_EXTENSIONS = {"txt", "md", "csv", "tsv", "log", "json", "yaml", "yml", "xml"}
# Image extensions — store as Image nodes directly
IMAGE_EXTENSIONS = {"jpg", "jpeg", "png", "gif", "bmp", "tiff", "tif", "webp", "svg"}
# Image extensions — store as Image nodes directly.
# SVG is deliberately absent: it is vector XML that Pillow cannot decode and
# that the vision stage cannot send as a data URI, so it gets rasterized by
# _parse_svg_file instead.
IMAGE_EXTENSIONS = {"jpg", "jpeg", "png", "gif", "bmp", "tiff", "tif", "webp"}
# Minimum image dimensions to extract (skip tiny icons/bullets)
MIN_IMAGE_WIDTH = 50
@@ -98,6 +101,12 @@ class DocumentParser:
if file_type in PLAINTEXT_EXTENSIONS:
return self._parse_plaintext(file_path, file_type)
# Checked before PYMUPDF_EXTENSIONS: PyMuPDF can open an SVG, but
# rendering a multi-page Write note as one document yields a blank or
# illegible image (see svg_raster), so it needs page-aware handling.
if file_type == "svg":
return self._parse_svg_file(file_path, file_type)
if file_type in IMAGE_EXTENSIONS:
return self._parse_image_file(file_path, file_type)
@@ -309,6 +318,61 @@ class DocumentParser:
file_type=file_type,
)
def _parse_svg_file(self, file_path: str, file_type: str) -> ParseResult:
"""
Rasterize an SVG into one ExtractedImage per page.
SVG is vector XML: Pillow cannot decode it and the vision stage cannot
put it in a data URI, so it is rendered to PNG here. Multi-page Write
notes become one image per page so each page reaches vision/OCR at a
legible size.
:param file_path: Path to the SVG file.
:param file_type: Normalized file extension ("svg").
:returns: ParseResult with one image per rendered page.
"""
with DOCUMENT_PARSE_DURATION.labels(file_type=file_type).time():
try:
from library.services.svg_raster import render_svg_pages
with open(file_path, "rb") as f:
data = f.read()
pages = render_svg_pages(data)
except Exception as exc:
DOCUMENTS_PARSED_TOTAL.labels(file_type=file_type, status="error").inc()
logger.error("Failed to rasterize SVG file_type=%s: %s", file_type, exc)
raise
images = [
ExtractedImage(
data=png,
ext="png",
width=width,
height=height,
source_page=index,
source_index=0,
)
for index, (png, width, height) in enumerate(pages)
]
DOCUMENTS_PARSED_TOTAL.labels(file_type=file_type, status="success").inc()
IMAGES_EXTRACTED_TOTAL.labels(file_type=file_type).inc(len(images))
logger.info(
"Parsed SVG file_type=%s pages=%d bytes=%d",
file_type,
len(images),
len(data),
)
return ParseResult(
text_blocks=[],
images=images,
metadata={"page_count": len(images)},
file_type=file_type,
)
def _parse_image_file(self, file_path: str, file_type: str) -> ParseResult:
"""
Handle a standalone image file — store as a single ExtractedImage.

View File

@@ -0,0 +1,263 @@
"""SVG rasterization for the ingest pipeline.
The vision stage sends each extracted image to a vision LLM as a ``data:`` URI,
which cannot carry ``image/svg+xml`` — so an SVG has to become raster before it
can be described, OCR'd, or embedded. PyMuPDF (already a dependency for PDF
parsing) renders SVG natively, so this needs no cairo/rsvg system libraries.
The non-obvious part is page splitting. Handwritten notes from the Write
(Stylus Labs) app are a single SVG document holding one or more
``<svg class="write-page">`` children stacked vertically via x/y offsets. Two
root formats exist in the wild and *both* must be split per page:
- Older files carry no width/height on the root ``<svg>``. Rendering the
document as-is makes PyMuPDF fall back to US-Letter and emit the top-left
corner only — ruled lines and no handwriting, in a perfectly valid PNG.
- Newer files (Write commit eeab021) do carry root width/height, spanning the
full stacked extent. Rendering those as-is yields one very tall strip; capped
to a sane longest edge, a 5-page note squashes to ~209px wide, well past
illegible.
Write never rewrites existing files, so the old format is permanent rather than
a migration window. Page geometry lives on the ``write-page`` element and is
identical across both formats, so splitting ignores the root dimensions
entirely and needs no format detection.
.. note::
Daedalus carries a twin of this module at
``backend/daedalus/extraction/svg.py``, which returns base64 for direct
chat attachment. The two are deliberately duplicated rather than shared —
the repos ship no common package — so fixes belong in both.
"""
from __future__ import annotations
import copy
import logging
import re
from lxml import etree
logger = logging.getLogger(__name__)
SVG_NS = "http://www.w3.org/2000/svg"
XLINK_NS = "http://www.w3.org/1999/xlink"
_SVG = f"{{{SVG_NS}}}"
_PAGE_CLASS = "write-page"
# Write documents render on a grey backdrop; pages themselves are transparent,
# so each rendered page needs an explicit white underlay or strokes land on
# black in the flattened PNG.
_PAGE_BACKGROUND = "#ffffff"
#: Longest edge of a rendered page, in pixels. Enough for a vision model to
#: read handwriting without spending tokens on unusable resolution.
DEFAULT_TARGET_PX = 1568
#: Cap on pages rendered from one document.
DEFAULT_MAX_PAGES = 20
# Unquoted attribute value in the root tag, e.g. Write's `width=auto`. Matches
# only bare alphabetic values so numeric or already-quoted attributes are left
# alone.
_UNQUOTED_ATTR = re.compile(rb"(\s[-\w:]+)=([A-Za-z][-\w]*)(?=[\s>])")
_ROOT_TAG = re.compile(rb"<svg[^>]*>")
class SvgRenderError(Exception):
"""Raised when an SVG cannot be parsed or contains no renderable page."""
def _repair_root_tag(data: bytes) -> bytes:
"""Quote unquoted attribute values in the root ``<svg>`` tag.
Some Write files emit ``width=auto height=auto``, which is not valid XML.
A malformation *inside* the root tag defeats ``recover=True`` differently
from one in the body: rather than dropping a subtree, libxml2 abandons the
whole document and yields a bare root, so every page becomes invisible.
Quoting the values first recovers the full tree.
"""
match = _ROOT_TAG.search(data)
if not match:
return data
repaired = _UNQUOTED_ATTR.sub(rb'\1="\2"', match.group(0))
if repaired == match.group(0):
return data
return data[: match.start()] + repaired + data[match.end() :]
def _parser() -> etree.XMLParser:
"""Build the hardened parser used for all untrusted SVG input.
``resolve_entities=False`` blocks XXE — ingest content is untrusted.
``huge_tree`` is required because handwriting path data runs to megabytes.
``recover`` salvages the handful of Write files that emit unescaped
attribute content and are not well-formed XML.
"""
return etree.XMLParser(
huge_tree=True,
resolve_entities=False,
no_network=True,
recover=True,
)
def _dimension(element: etree._Element, name: str) -> float | None:
"""Read a CSS-pixel dimension attribute, tolerating a ``px`` suffix."""
raw = element.get(name)
if not raw:
return None
try:
return float(raw.strip().removesuffix("px"))
except ValueError:
return None
def _viewbox_size(element: etree._Element) -> tuple[float, float] | None:
"""Derive width/height from a viewBox extent."""
raw = element.get("viewBox")
if not raw:
return None
parts = raw.replace(",", " ").split()
if len(parts) != 4:
return None
try:
width, height = float(parts[2]), float(parts[3])
except ValueError:
return None
return (width, height) if width > 0 and height > 0 else None
def _element_size(element: etree._Element) -> tuple[float, float] | None:
"""Resolve an element's rendered size from width/height, else viewBox."""
width = _dimension(element, "width")
height = _dimension(element, "height")
if width and height:
return width, height
return _viewbox_size(element)
def _standalone_page(
page: etree._Element,
defs: etree._Element | None,
width: float,
height: float,
) -> bytes:
"""Wrap one ``write-page`` element as its own renderable SVG document.
The page is repositioned to the origin (its x/y place it within the stacked
parent) and given an explicit viewBox so the renderer has an unambiguous
size. ``defs`` is copied in because pen and ruling definitions live on the
root and are referenced by page content.
"""
root = etree.Element(f"{_SVG}svg", nsmap={None: SVG_NS, "xlink": XLINK_NS})
root.set("width", str(width))
root.set("height", str(height))
root.set("viewBox", f"0 0 {width} {height}")
background = etree.SubElement(root, f"{_SVG}rect")
background.set("width", "100%")
background.set("height", "100%")
background.set("fill", _PAGE_BACKGROUND)
if defs is not None:
root.append(copy.deepcopy(defs))
element = copy.deepcopy(page)
for positional in ("x", "y"):
element.attrib.pop(positional, None)
element.set("viewBox", f"0 0 {width} {height}")
root.append(element)
return etree.tostring(root)
def split_svg_pages(data: bytes) -> list[bytes]:
"""Split an SVG into one standalone document per renderable page.
Write multi-page notes yield one document per ``write-page`` child. Any
other SVG (a diagram, a logo) yields a single document — the input itself,
which the renderer sizes from its own width/height or viewBox.
:param data: Raw SVG bytes.
:returns: One or more standalone SVG documents, in page order.
:raises SvgRenderError: If the SVG cannot be parsed or has no usable size.
"""
try:
root = etree.fromstring(_repair_root_tag(data), _parser())
except etree.XMLSyntaxError as exc:
raise SvgRenderError(f"Could not parse SVG: {exc}") from exc
if root is None:
raise SvgRenderError("Could not parse SVG: no root element.")
defs = root.find(f"{_SVG}defs")
pages: list[bytes] = []
for element in root.iter(f"{_SVG}svg"):
if element is root:
continue
if _PAGE_CLASS not in (element.get("class") or "").split():
continue
size = _element_size(element)
if not size:
continue
pages.append(_standalone_page(element, defs, *size))
if pages:
return pages
# Generic SVG: render as-is. It must still be sizeable, or the renderer
# would silently substitute a default page box.
if not _element_size(root):
raise SvgRenderError(
"SVG has no width/height or viewBox, so its size is undefined."
)
return [data]
def render_svg_pages(
data: bytes,
max_pages: int = DEFAULT_MAX_PAGES,
target_px: int = DEFAULT_TARGET_PX,
) -> list[tuple[bytes, int, int]]:
"""Rasterize an SVG to one PNG per page.
:param data: Raw SVG bytes.
:param max_pages: Cap on pages rendered.
:param target_px: Longest edge of each rendered page, in pixels.
:returns: One ``(png_bytes, width, height)`` tuple per page, in page order.
:raises SvgRenderError: If nothing could be rendered.
"""
import fitz
pages = split_svg_pages(data)
rendered: list[tuple[bytes, int, int]] = []
for index, page_svg in enumerate(pages[:max_pages]):
try:
with fitz.open(stream=page_svg, filetype="svg") as document:
page = document[0]
longest = max(page.rect.width, page.rect.height)
zoom = target_px / longest if longest else 1.0
pixmap = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom))
rendered.append(
(pixmap.tobytes("png"), pixmap.width, pixmap.height)
)
except Exception as exc:
raise SvgRenderError(
f"Could not render SVG page {index + 1}: {exc}"
) from exc
if not rendered:
raise SvgRenderError("No renderable pages in SVG.")
logger.info(
"Rasterized SVG pages_rendered=%d total_pages=%d target_px=%d",
len(rendered),
len(pages),
target_px,
)
return rendered

View File

@@ -12,6 +12,25 @@
<div class="alert alert-warning mb-6">
<span>Are you sure you want to delete <strong>{{ library.name }}</strong>? This action cannot be undone.</span>
</div>
{% if library.managed_by_display %}
<div class="alert alert-error mb-6">
<span>
<strong>This Library is managed by {{ library.managed_by_display }}</strong>{% if library.workspace_id %}
(workspace <code>{{ library.workspace_id }}</code>){% endif %}.
{% if library.workspace_id %}
Deleting it here removes its embedded content from Mnemosyne, but the
source files still live in {{ library.managed_by_display }} — it will
be <strong>recreated and re-embedded on the next sync</strong>. Use
this to clear an orphaned Library that is blocking workspace
re-registration.
{% else %}
Deleting it here removes its embedded content from Mnemosyne;
{{ library.managed_by_display }} may recreate and re-embed it on its
next sync.
{% endif %}
</span>
</div>
{% endif %}
<form method="post">
{% csrf_token %}
<div class="flex gap-2">

View File

@@ -12,10 +12,10 @@
<h1 class="text-3xl font-bold">{{ library.name }}</h1>
<div class="flex flex-wrap gap-2 mt-2">
<div class="badge badge-primary">{{ library.library_type }}</div>
{% if library.workspace_id %}
{% if library.managed_by_display %}
<div class="badge badge-warning gap-1"
title="Workspace {{ library.workspace_id }}">
Daedalus workspace
{% if library.workspace_id %}title="Workspace {{ library.workspace_id }}"{% endif %}>
Managed by {{ library.managed_by_display }}
</div>
{% endif %}
</div>
@@ -25,26 +25,29 @@
</div>
<div class="flex gap-2">
<a href="{% url 'library:library-edit' uid=library.uid %}" class="btn btn-sm btn-outline">Edit</a>
{% if library.workspace_id %}
<button type="button" class="btn btn-sm btn-error btn-outline" disabled
title="This library is managed by Daedalus. Delete it from the Daedalus workspace, not here.">
Delete
</button>
{% else %}
<a href="{% url 'library:library-delete' uid=library.uid %}" class="btn btn-sm btn-error btn-outline">Delete</a>
{% endif %}
</div>
</div>
{% if library.workspace_id %}
{% if library.managed_by_display %}
<div class="alert alert-warning mb-6">
<div>
<div class="font-semibold">Managed by Daedalus</div>
<div class="font-semibold">Managed by {{ library.managed_by_display }}</div>
<div class="text-sm opacity-80">
This library was created for Daedalus workspace
{% if library.workspace_id %}
This library was created for workspace
<code class="font-mono">{{ library.workspace_id }}</code>.
Items here are owned by the workspace; deleting the workspace in
Daedalus will remove this library. Do not delete it manually.
Normally you manage it from {{ library.managed_by_display }}.
Deleting it here removes its embedded content from Mnemosyne, but
the source files still live in {{ library.managed_by_display }} —
it will be recreated and re-embedded on the next sync.
Use Delete to clear an orphaned library that is blocking workspace
re-registration.
{% else %}
Content in this library is pushed by
{{ library.managed_by_display }}. Edits made here may be
overwritten or re-created on its next sync.
{% endif %}
</div>
</div>
</div>

View File

@@ -15,6 +15,18 @@
</div>
</div>
<form method="get" class="mb-4 flex flex-wrap gap-3 items-end">
<div class="form-control">
<label class="label"><span class="label-text">Scope</span></label>
<select name="scope" class="select select-bordered select-sm">
<option value="all" {% if scope == "all" %}selected{% endif %}>All libraries</option>
<option value="unmanaged" {% if scope == "unmanaged" %}selected{% endif %}>Unmanaged only</option>
<option value="managed" {% if scope == "managed" %}selected{% endif %}>App-managed only</option>
</select>
</div>
<button type="submit" class="btn btn-sm btn-outline">Filter</button>
</form>
{% if error %}
<div class="alert alert-warning mb-4">
<span>{{ error }}</span>
@@ -33,9 +45,9 @@
</h2>
<div class="flex flex-wrap gap-1">
<div class="badge badge-outline">{{ lib.library_type }}</div>
{% if lib.workspace_id %}
<div class="badge badge-warning gap-1" title="Managed by Daedalus workspace {{ lib.workspace_id }} — do not delete from Mnemosyne.">
Daedalus workspace
{% if lib.managed_by_display %}
<div class="badge badge-warning gap-1" title="Managed by {{ lib.managed_by_display }}{% if lib.workspace_id %} (workspace {{ lib.workspace_id }}){% endif %} — do not delete from Mnemosyne.">
Managed by {{ lib.managed_by_display }}
</div>
{% endif %}
</div>
@@ -53,8 +65,12 @@
{% else %}
{% if not error %}
<div class="text-center py-12 opacity-60">
{% if scope == "all" %}
<p class="text-lg">No libraries yet.</p>
<p class="mt-2">Create your first library to get started.</p>
{% else %}
<p class="text-lg">No libraries match this filter.</p>
{% endif %}
</div>
{% endif %}
{% endif %}

View File

@@ -21,6 +21,7 @@ class LibraryTypeDefaultsTests(TestCase):
"film",
"art",
"journal",
"email",
"business",
"finance",
}

View File

@@ -0,0 +1,179 @@
"""Tests for the per-app ``managed_by`` concept.
Covers the pure helpers (``infer_legacy_manager``,
``Library.managed_by_display``), the token-derived stamping and
duplicate-name rejection on the plain create endpoint (Neo4j stubbed
via ``sys.modules``, same style as ``test_views.py``), and the
``WorkspaceStatusSerializer`` surface. Cypher-touching paths are
covered by the manual end-to-end plan, not these unit tests.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import patch
from django.contrib.auth import get_user_model
from django.test import TestCase
from rest_framework.test import APIClient
from library.api.serializers import WorkspaceStatusSerializer
from library.models import Library, infer_legacy_manager
from mcp_server.models import UserToken
User = get_user_model()
class InferLegacyManagerTests(TestCase):
"""Truth table for the pre-``managed_by`` inference."""
def test_null_workspace_is_unmanaged(self):
self.assertIsNone(infer_legacy_manager(None))
self.assertIsNone(infer_legacy_manager(""))
def test_kairos_mail_prefix_is_kairos(self):
self.assertEqual(
infer_legacy_manager("kairos-mail-abc123-7"), "Kairos"
)
def test_other_workspace_is_daedalus(self):
self.assertEqual(
infer_legacy_manager("2f9c4a1e-uuid-ish"), "Daedalus"
)
class ManagedByDisplayTests(TestCase):
"""``managed_by_display`` on in-memory (unsaved) Library nodes."""
def test_stamped_label_wins(self):
lib = Library(name="x", managed_by="Spelunker")
self.assertEqual(lib.managed_by_display, "Spelunker")
def test_stamped_label_wins_over_inference(self):
lib = Library(
name="x", managed_by="My Token", workspace_id="kairos-mail-a-1"
)
self.assertEqual(lib.managed_by_display, "My Token")
def test_legacy_workspace_falls_back_to_inference(self):
lib = Library(name="x", workspace_id="ws-uuid")
self.assertEqual(lib.managed_by_display, "Daedalus")
def test_unmanaged_is_empty_string(self):
lib = Library(name="x")
self.assertEqual(lib.managed_by_display, "")
class _FakeLibrary:
"""Stand-in for the neomodel Library on the plain create endpoint."""
class DoesNotExist(Exception):
pass
existing = None # what nodes.get(name=...) returns
instances = [] # constructor kwargs, in order
def __init__(self, **kwargs):
type(self).instances.append(kwargs)
self.__dict__.update(kwargs)
self.uid = "lib-new"
self.workspace_id = None
self.created_at = None
def save(self):
return self
class _Nodes:
@staticmethod
def get(**kwargs):
if _FakeLibrary.existing is None:
raise _FakeLibrary.DoesNotExist()
return _FakeLibrary.existing
nodes = _Nodes()
class LibraryCreateStampingTests(TestCase):
"""POST /library/api/libraries/ stamps ``managed_by`` and rejects dupes."""
def setUp(self):
self.user = User.objects.create_user(username="op", password="pw")
self.client = APIClient()
_FakeLibrary.existing = None
_FakeLibrary.instances = []
def _post(self, token=None):
self.client.force_authenticate(user=self.user, token=token)
with patch.dict(
"sys.modules",
{"library.models": SimpleNamespace(Library=_FakeLibrary)},
):
return self.client.post(
"/library/api/libraries/",
{"name": "Docs", "library_type": "technical"},
format="json",
)
def test_token_create_stamps_token_name(self):
response = self._post(token=UserToken(name="Spelunker"))
self.assertEqual(response.status_code, 201)
self.assertEqual(_FakeLibrary.instances[0]["managed_by"], "Spelunker")
self.assertEqual(response.json()["managed_by"], "Spelunker")
def test_session_create_leaves_managed_by_null(self):
response = self._post(token=None)
self.assertEqual(response.status_code, 201)
self.assertIsNone(_FakeLibrary.instances[0]["managed_by"])
def test_duplicate_name_returns_409_name_conflict(self):
_FakeLibrary.existing = SimpleNamespace(
uid="lib-old", managed_by_display="Daedalus"
)
response = self._post(token=UserToken(name="Spelunker"))
self.assertEqual(response.status_code, 409)
body = response.json()
self.assertEqual(body["code"], "name_conflict")
self.assertIn("Docs", body["detail"])
self.assertEqual(body["uid"], "lib-old")
self.assertEqual(body["managed_by"], "Daedalus")
self.assertEqual(_FakeLibrary.instances, [])
def test_duplicate_of_unmanaged_reports_null_manager(self):
_FakeLibrary.existing = SimpleNamespace(
uid="lib-old", managed_by_display=""
)
response = self._post(token=None)
self.assertEqual(response.status_code, 409)
self.assertIsNone(response.json()["managed_by"])
class WorkspaceStatusSerializerManagedByTests(TestCase):
"""The workspace status payload carries ``managed_by`` (nullable)."""
BASE = {
"workspace_id": "ws_a",
"library_uid": "lib_1",
"name": "W",
"library_type": "technical",
"description": "",
"item_count": 0,
"chunk_count": 0,
"created_at": "2026-01-01T00:00:00Z",
}
def test_managed_by_value_round_trips(self):
s = WorkspaceStatusSerializer(data={**self.BASE, "managed_by": "Daedalus"})
self.assertTrue(s.is_valid(), s.errors)
self.assertEqual(s.validated_data["managed_by"], "Daedalus")
def test_managed_by_null_accepted(self):
s = WorkspaceStatusSerializer(data={**self.BASE, "managed_by": None})
self.assertTrue(s.is_valid(), s.errors)
def test_managed_by_absent_accepted(self):
s = WorkspaceStatusSerializer(data=self.BASE)
self.assertTrue(s.is_valid(), s.errors)

View File

@@ -0,0 +1,195 @@
"""
Tests for SVG rasterization in the ingest pipeline.
SVG is vector XML: Pillow cannot decode it and the vision stage cannot send it
as a data URI, so it is rendered to PNG at parse time. The cases that matter
are the ones that fail *silently* — Write notes stack their pages inside one
document, and rendering that document whole yields a plausible-looking PNG that
is either blank or an illegible tall strip.
"""
import io
import os
import tempfile
from django.test import TestCase
from PIL import Image
from library.services.parsers import (
IMAGE_EXTENSIONS,
DocumentParser,
)
from library.services.svg_raster import (
SvgRenderError,
render_svg_pages,
split_svg_pages,
)
# Geometry copied from a real Write note; the page element is identical across
# both root formats, which is what makes splitting format-agnostic.
PAGE_WIDTH = 1094
PAGE_HEIGHT = 1654
PAGE_PITCH = 1674 # page height + inter-page gap
def write_document(pages: int, root_size: bool = False) -> bytes:
"""
Build a Write-style document.
:param pages: Number of stacked pages.
:param root_size: Emit width/height on the root <svg>. False models
pre-eeab021 files (the permanent majority), True models newer saves.
"""
root_attrs = ""
if root_size:
total = 10 + pages * PAGE_PITCH
root_attrs = f' width="{PAGE_WIDTH + 20}" height="{total}"'
body = "".join(
f'<svg class="write-page" x="10" y="{10 + i * PAGE_PITCH}" '
f'width="{PAGE_WIDTH}px" height="{PAGE_HEIGHT}px" '
f'xmlns="http://www.w3.org/2000/svg">'
f'<path d="M 100 {100 + i * 40} L {600 + i * 120} {700 + i * 40}" '
f'stroke="#000000" stroke-width="12" fill="none"/></svg>'
for i in range(pages)
)
return (
f'<svg id="write-document"{root_attrs} '
f'xmlns="http://www.w3.org/2000/svg" '
f'xmlns:xlink="http://www.w3.org/1999/xlink">'
f'<rect id="write-doc-background" width="100%" height="100%" fill="#808080"/>'
f'<defs id="write-defs"><style>.write-flat-pen{{fill:none}}</style></defs>'
f"{body}</svg>"
).encode()
GENERIC_SVG = (
b'<svg xmlns="http://www.w3.org/2000/svg" width="400" height="200" '
b'viewBox="0 0 400 200">'
b'<path d="M 20 20 L 380 180" stroke="#000" stroke-width="10"/></svg>'
)
def ink_fraction(png: bytes) -> float:
"""Fraction of non-white pixels — the blank-render canary."""
grey = Image.open(io.BytesIO(png)).convert("L")
histogram = grey.histogram()
return sum(histogram[:200]) / sum(histogram)
class SvgSplitTests(TestCase):
"""Splitting a Write document into per-page SVGs."""
def test_splits_one_document_per_page(self):
for pages in (1, 2, 5):
for root_size in (False, True):
with self.subTest(pages=pages, root_size=root_size):
document = write_document(pages, root_size=root_size)
self.assertEqual(len(split_svg_pages(document)), pages)
def test_both_root_formats_split_identically(self):
"""Root width/height (Write eeab021) must not change the outcome.
Existing notes are never rewritten, so both formats persist
indefinitely and have to render the same.
"""
without = split_svg_pages(write_document(3, root_size=False))
with_size = split_svg_pages(write_document(3, root_size=True))
self.assertEqual(len(without), len(with_size))
self.assertEqual(len(without), 3)
def test_generic_svg_is_a_single_page(self):
self.assertEqual(len(split_svg_pages(GENERIC_SVG)), 1)
def test_unsized_svg_is_rejected_rather_than_guessed(self):
unsized = b'<svg xmlns="http://www.w3.org/2000/svg"><rect/></svg>'
with self.assertRaises(SvgRenderError):
split_svg_pages(unsized)
def test_script_inside_defs_does_not_desync_the_split(self):
"""Regression: a non-greedy <defs>...</defs> regex mis-parses these."""
document = write_document(2).replace(
b'<defs id="write-defs">',
b'<defs id="write-defs"><script><float value="770" /></script>',
)
self.assertEqual(len(split_svg_pages(document)), 2)
def test_unquoted_root_attributes_are_repaired(self):
"""Write can emit ``width=auto``, which is not valid XML.
A malformation inside the root tag defeats recovery differently from
one in the body: libxml2 abandons the whole document and returns a
bare root, so every page silently disappears.
"""
document = write_document(2).replace(
b'<svg id="write-document"',
b'<svg width=auto height=auto id="write-document"',
)
self.assertEqual(len(split_svg_pages(document)), 2)
class SvgRenderTests(TestCase):
"""Rasterizing pages to PNG."""
def test_renders_one_png_per_page(self):
pages = render_svg_pages(write_document(3))
self.assertEqual(len(pages), 3)
for png, width, height in pages:
self.assertEqual(Image.open(io.BytesIO(png)).format, "PNG")
self.assertEqual(max(width, height), 1568)
def test_rendered_pages_are_not_blank(self):
"""The pre-patch trap: a whole-document render emits ruling, no ink."""
for root_size in (False, True):
with self.subTest(root_size=root_size):
for png, _, _ in render_svg_pages(
write_document(3, root_size=root_size)
):
self.assertGreater(ink_fraction(png), 0.001)
def test_pages_are_page_shaped_not_a_stacked_strip(self):
"""The post-patch trap: root dimensions span every stacked page.
Rendering that whole gives one tall strip which, once capped, squashes
a 5-page note to ~209px wide.
"""
expected = PAGE_HEIGHT / PAGE_WIDTH
for root_size in (False, True):
with self.subTest(root_size=root_size):
for _, width, height in render_svg_pages(
write_document(5, root_size=root_size)
):
self.assertAlmostEqual(height / width, expected, delta=0.05)
def test_page_cap_is_respected(self):
self.assertEqual(len(render_svg_pages(write_document(8), max_pages=5)), 5)
class SvgParserIntegrationTests(TestCase):
"""The parser dispatch — SVG must not reach the Pillow image path."""
def setUp(self):
self.parser = DocumentParser()
def test_svg_is_not_in_image_extensions(self):
# It was, and PIL.Image.open cannot decode SVG, so ingest always failed.
self.assertNotIn("svg", IMAGE_EXTENSIONS)
def test_parse_multipage_svg_yields_one_image_per_page(self):
with tempfile.NamedTemporaryFile(suffix=".svg", delete=False) as f:
f.write(write_document(3))
f.flush()
path = f.name
try:
result = self.parser.parse(path, "svg")
finally:
os.unlink(path)
self.assertEqual(len(result.images), 3)
self.assertEqual(result.metadata["page_count"], 3)
self.assertEqual(result.text_blocks, [])
for index, image in enumerate(result.images):
# PNG, not svg — the vision stage needs raster for its data URI.
self.assertEqual(image.ext, "png")
self.assertEqual(image.source_page, index)
self.assertGreater(ink_fraction(image.data), 0.001)

View File

@@ -0,0 +1,110 @@
"""Tests for the library CRUD HTML views.
Currently covers ``library_list``'s app-managed scope filter. The view
loads every ``Library`` node from Neo4j and narrows it in Python by a
``scope`` GET param (``all`` / ``unmanaged`` / ``managed``, with legacy
``global`` / ``daedalus`` aliases). These tests stub out Neo4j entirely —
patching ``neo4j_available`` and injecting a fake ``Library`` class via
``sys.modules`` — so they assert on the filtering the view does and the
context it renders, not on real graph behaviour. Mirrors the mocking
style in ``test_search_views_admin_scope.py``.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
from django.contrib.auth import get_user_model
from django.test import TestCase
from django.urls import reverse
User = get_user_model()
def _lib(name, managed_by_display):
return SimpleNamespace(
uid=f"uid-{name}",
name=name,
library_type="technical",
description="",
workspace_id=None,
managed_by_display=managed_by_display,
)
class LibraryListScopeFilterTests(TestCase):
"""Cover the ``scope`` filter branches of ``library_list``."""
def setUp(self):
self.user = User.objects.create_user(
username="op", email="op@example.com", password="pw"
)
self.client.force_login(self.user)
self.url = reverse("library:library-list")
self.managed = _lib("Docs", "Spelunker")
self.unmanaged = _lib("Notes", "")
def _fake_library_cls(self):
"""Return a Library stub whose ``nodes.order_by`` yields two libraries."""
fake_nodes = MagicMock()
fake_nodes.order_by.return_value = [self.managed, self.unmanaged]
return SimpleNamespace(nodes=fake_nodes)
def _get(self, **params):
with patch("library.views.neo4j_available", return_value=True), \
patch.dict(
"sys.modules",
{"library.models": SimpleNamespace(Library=self._fake_library_cls())},
):
return self.client.get(self.url, params)
def test_default_scope_is_all_and_returns_everything(self):
response = self._get()
self.assertEqual(response.status_code, 200)
self.assertEqual(response.context["scope"], "all")
self.assertEqual(
list(response.context["libraries"]), [self.managed, self.unmanaged]
)
def test_managed_scope_keeps_only_managed(self):
response = self._get(scope="managed")
self.assertEqual(response.context["scope"], "managed")
self.assertEqual(list(response.context["libraries"]), [self.managed])
def test_unmanaged_scope_keeps_only_unmanaged(self):
response = self._get(scope="unmanaged")
self.assertEqual(response.context["scope"], "unmanaged")
self.assertEqual(list(response.context["libraries"]), [self.unmanaged])
def test_legacy_daedalus_scope_aliases_to_managed(self):
response = self._get(scope="daedalus")
self.assertEqual(response.context["scope"], "managed")
self.assertEqual(list(response.context["libraries"]), [self.managed])
def test_legacy_global_scope_aliases_to_unmanaged(self):
response = self._get(scope="global")
self.assertEqual(response.context["scope"], "unmanaged")
self.assertEqual(list(response.context["libraries"]), [self.unmanaged])
def test_unknown_scope_does_not_filter(self):
"""An unexpected scope value degrades to the unfiltered list."""
response = self._get(scope="bogus")
self.assertEqual(response.context["scope"], "bogus")
self.assertEqual(
list(response.context["libraries"]), [self.managed, self.unmanaged]
)
def test_neo4j_unavailable_sets_error_and_empty_list(self):
with patch("library.views.neo4j_available", return_value=False):
response = self.client.get(self.url)
self.assertEqual(response.status_code, 200)
self.assertEqual(list(response.context["libraries"]), [])
self.assertEqual(response.context["error"], "Neo4j is not available.")

View File

@@ -31,14 +31,23 @@ logger = logging.getLogger(__name__)
@login_required
def library_list(request):
"""List all libraries."""
"""List libraries, optionally filtered by app-managed scope."""
scope = request.GET.get("scope", "all")
# Legacy bookmark values from before managed_by existed.
scope = {"daedalus": "managed", "global": "unmanaged"}.get(scope, scope)
libraries = []
error = None
if neo4j_available():
try:
from .models import Library
libraries = Library.nodes.order_by("name")
libraries = list(Library.nodes.order_by("name"))
# managed_by_display covers legacy workspace libraries that
# predate the managed_by property, so filter in Python.
if scope == "managed":
libraries = [l for l in libraries if l.managed_by_display]
elif scope == "unmanaged":
libraries = [l for l in libraries if not l.managed_by_display]
except Exception as e:
error = f"Could not connect to Neo4j: {e}"
logger.error(error)
@@ -47,7 +56,7 @@ def library_list(request):
return render(
request,
"library/library_list.html",
{"libraries": libraries, "error": error},
{"libraries": libraries, "error": error, "scope": scope},
)
@@ -319,20 +328,20 @@ def library_delete(request, uid):
messages.error(request, f"Library not found: {e}")
return redirect("library:library-list")
# Daedalus owns the lifecycle of workspace-scoped libraries — they can
# only be deleted via DELETE /library/api/workspaces/{workspace_id}/.
# Block the human delete path so a stray click can't desync state.
if lib.workspace_id:
messages.error(
request,
f'"{lib.name}" is managed by Daedalus workspace '
f"{lib.workspace_id}. Delete it from Daedalus, not here.",
)
return redirect("library:library-detail", uid=uid)
# Daedalus owns the lifecycle of workspace-scoped libraries. Deleting one
# here is allowed but discouraged: the confirm page warns that Daedalus
# still holds the source content and will recreate + re-embed it on the
# next sync. The risk is low (no data loss — only re-embedding cost), and
# this is the supported escape hatch for clearing an orphaned Library that
# blocks workspace re-registration.
if request.method == "POST":
name = lib.name
lib.delete()
# Use the shared cascade so child nodes (Collections/Items/Chunks/
# Images) and orphan Concepts are removed too — a bare lib.delete()
# would leak them.
from .services.library_delete import delete_library_cascade
delete_library_cascade(lib)
messages.success(request, f'Library "{name}" deleted.')
return redirect("library:library-list")
return render(request, "library/library_confirm_delete.html", {"library": lib})

View File

@@ -26,6 +26,20 @@ from rest_framework import authentication, exceptions
from .auth import MCPAuthError, resolve_mcp_user
def request_token_label(request):
"""Name of the ``UserToken`` authenticating this request, or None.
Session-authenticated requests (``request.auth`` is None) and blank
token names return None — the caller treats both as "no managing app".
"""
from .models import UserToken
token = getattr(request, "auth", None)
if isinstance(token, UserToken):
return token.name.strip() or None
return None
class UserTokenAuthentication(authentication.BaseAuthentication):
"""Authenticate DRF requests with a ``UserToken`` bearer."""

View File

@@ -57,6 +57,12 @@ class UserTokenCreateForm(forms.Form):
"class": "input input-bordered w-full",
"placeholder": "e.g. Claude Desktop, CI script",
}),
help_text=(
"A friendly label so you can identify this token later. It also "
"labels any library the token creates (shown as “Managed by "
"<name>”) — for an app integration, use the app's name, e.g. "
"Daedalus, Kairos, Spelunker."
),
)
expires_at = forms.DateTimeField(
required=False,

View File

@@ -21,7 +21,7 @@
</label>
{{ form.name }}
<label class="label">
<span class="label-text-alt opacity-60">A friendly label so you can identify this token later (e.g. “Claude Desktop”).</span>
<span class="label-text-alt opacity-60">{{ form.name.help_text }}</span>
</label>
</div>
<div class="form-control mt-4">

View File

@@ -105,6 +105,44 @@ class UserTokenAuthenticationTest(TestCase):
resp = self._get(f"Bearer {self.plaintext} extra")
self.assertEqual(resp.status_code, status.HTTP_401_UNAUTHORIZED)
def test_request_token_label_reads_token_name(self):
from types import SimpleNamespace
from mcp_server.drf_auth import request_token_label
token = UserToken(name=" Spelunker ")
self.assertEqual(
request_token_label(SimpleNamespace(auth=token)), "Spelunker"
)
def test_request_token_label_none_for_session(self):
from types import SimpleNamespace
from mcp_server.drf_auth import request_token_label
self.assertIsNone(request_token_label(SimpleNamespace(auth=None)))
# A request object with no auth attribute at all (plain Django).
self.assertIsNone(request_token_label(SimpleNamespace()))
def test_request_token_label_none_for_blank_name(self):
from types import SimpleNamespace
from mcp_server.drf_auth import request_token_label
self.assertIsNone(
request_token_label(SimpleNamespace(auth=UserToken(name=" ")))
)
def test_request_token_label_none_for_foreign_auth_object(self):
from types import SimpleNamespace
from mcp_server.drf_auth import request_token_label
# e.g. a JWT dict from another auth class — not a UserToken.
self.assertIsNone(
request_token_label(SimpleNamespace(auth={"iss": "daedalus"}))
)
def test_request_auth_stashes_token(self):
# The auth class returns (user, token); DRF places the token on
# request.auth. Re-use a UserToken-aware endpoint to verify.

View File

@@ -23,8 +23,9 @@ env = environ.Env(
DEBUG=(bool, True),
)
# Read .env file if it exists
environ.Env.read_env(BASE_DIR / ".env")
# Read .env file if it exists. Lives at the repo root (one level above the
# Django project package), shared with the docker-compose stack.
environ.Env.read_env(BASE_DIR.parent / ".env")
# --- Security ---
SECRET_KEY = env("SECRET_KEY", default="django-insecure-change-me-in-production")
@@ -267,6 +268,15 @@ SPELUNKER_S3_REGION_NAME = env("SPELUNKER_S3_REGION_NAME", default="us-east-1")
SPELUNKER_S3_USE_SSL = env.bool("SPELUNKER_S3_USE_SSL", default=False)
SPELUNKER_S3_VERIFY = env.bool("SPELUNKER_S3_VERIFY", default=True)
# Kairos renders synced mail to text documents in its own bucket.
KAIROS_S3_ENDPOINT_URL = env("KAIROS_S3_ENDPOINT_URL", default="")
KAIROS_S3_ACCESS_KEY_ID = env("KAIROS_S3_ACCESS_KEY_ID", default="")
KAIROS_S3_SECRET_ACCESS_KEY = env("KAIROS_S3_SECRET_ACCESS_KEY", default="")
KAIROS_S3_BUCKET_NAME = env("KAIROS_S3_BUCKET_NAME", default="kairos")
KAIROS_S3_REGION_NAME = env("KAIROS_S3_REGION_NAME", default="us-east-1")
KAIROS_S3_USE_SSL = env.bool("KAIROS_S3_USE_SSL", default=False)
KAIROS_S3_VERIFY = env.bool("KAIROS_S3_VERIFY", default=True)
# Registry keyed by the ingest `source` field. Unknown/blank sources fall
# back to "daedalus" for backwards compatibility.
SOURCE_S3_BUCKETS = {
@@ -288,6 +298,15 @@ SOURCE_S3_BUCKETS = {
"use_ssl": SPELUNKER_S3_USE_SSL,
"verify": SPELUNKER_S3_VERIFY,
},
"kairos-mail": {
"endpoint_url": KAIROS_S3_ENDPOINT_URL,
"access_key_id": KAIROS_S3_ACCESS_KEY_ID,
"secret_access_key": KAIROS_S3_SECRET_ACCESS_KEY,
"bucket_name": KAIROS_S3_BUCKET_NAME,
"region_name": KAIROS_S3_REGION_NAME,
"use_ssl": KAIROS_S3_USE_SSL,
"verify": KAIROS_S3_VERIFY,
},
}
# --- Celery / RabbitMQ ---

View File

@@ -30,6 +30,9 @@ dependencies = [
"semantic-text-splitter>=0.20,<1.0",
"tokenizers>=0.20,<1.0",
"Pillow>=10.0,<12.0",
# SVG page splitting — needs recover=True for Write notes that emit
# unescaped attribute content and aren't well-formed XML
"lxml>=5.3,<7",
"requests>=2.31,<3.0",
# Phase 5: MCP Server
"fastmcp>=2.0,<3.0",