mnemosyne/docs/DAEDALUS_PALLAS_INTEGRATION_v2.md

Daedalus ↔ Pallas ↔ Mnemosyne Integration — v2

Status: Approved design — supersedes DAEDALUS_PALLAS_INTEGRATION_v1.md. Authoritative home: mnemosyne/docs/DAEDALUS_PALLAS_INTEGRATION_v2.md Versioning: subsequent major revisions ship as ..._v3.md etc. alongside this file. Cross-service docs (Daedalus, Pallas) link here.

---

1. Summary

This document describes the end-state authentication / authorization model connecting three services:

• Mnemosyne — knowledge platform. Owns Libraries, users, and the MCP surface third-party clients query.
• Daedalus — workspace + file-lifecycle UI. Registers Pallas instances, syncs file content to Mnemosyne, drives chat. Acts on behalf of one Mnemosyne user per Daedalus instance.
• Pallas — FastAgent-backed MCP host that exposes agent teams (Kottos, Mentor, Iolaus, …) as HTTP MCP servers.

What changed from v1:

• Single token model. The two-token split in v1 (DRF authtoken for REST, MCPToken for /mcp/) is gone. One model — UserToken — authenticates both surfaces, managed from one UI at /profile/tokens/. The DRF authtoken app has been removed from INSTALLED_APPS.
• Per-user authorization on the REST surface. The Daedalus-facing endpoints (/library/api/*, /mcp_server/api/teams/*) are no longer open to any authenticated account. Each Team has an owner FK and each workspace-scoped Library has an owner_username property; the endpoints scope by these and return 404 for non-owners. The daedalus-service shared account has been retired.
• Per-turn JWT path retired. The legacy iss=daedalus JWT flow (v1 §5.1, §6.2) is gone. Mnemosyne now only validates one JWT shape: typ=team, iss=mnemosyne. The replay cache and the _resolve_jwt_actor service-user fallback are also gone.
• Authorization headers normalised to Bearer. DRF TokenAuthentication (and its Token keyword) is replaced by UserTokenAuthentication, which accepts Authorization: Bearer <plaintext>. Anonymous requests get 401 + WWW-Authenticate: Bearer (RFC 7235).

Everything else in v1 — the resolved-library abstraction, team JWT shape, Pallas's static-bearer configuration, the workspace ↔ Team attachment model in Daedalus, agent picker UX, signing-key model — is unchanged.

---

2. Motivation

v1 closed the per-turn JWT forwarding hairball by introducing static team JWTs. v2 finishes the cleanup pass: it deletes the per-turn JWT path entirely (now that Daedalus has migrated off it), collapses the remaining two-token muddle into a single UserToken system, and tightens the REST surface so authentication-as-user is sufficient for access control without a shared service account.

---

3. Architecture

3.1 Services and responsibilities

Service	Role in auth model
Mnemosyne	Owns Libraries, Library memberships, UserTokens, Teams, TeamWorkspaceAssignments, signing keys. Validates bearers. Resolves every authenticated request to a Library set.
Daedalus	Control plane. Registers Pallas instances as Teams in Mnemosyne. Manages workspace ↔ team attachments. Stores team JWTs for copying into Pallas deployment configs. Acts as a single Mnemosyne user via a UserToken.
Pallas	Stateless MCP host. Holds a static team JWT in fastagent.secrets.yaml. No custom auth-forwarding code.

3.2 Two credential types

Every authenticated request to Mnemosyne presents a Bearer token of exactly one of these shapes:

#	Credential	iss	Issuer	Lifetime	Used on	Library scope source
1	Opaque UserToken	n/a	The Mnemosyne user, via /profile/tokens/	Until revoked / expiry	/mcp/ and DRF REST	MCP: allowed_libraries. REST: ignored (owner-scoped).
2	Team JWT	mnemosyne	Mnemosyne (/mcp_server/api/teams/)	10 years	/mcp/ only	Live DB lookup via TeamWorkspaceAssignment → Library

The v1 per-turn JWT (category 2 in v1) has been retired and is no longer accepted by resolve_mcp_jwt.

3.3 Scope split by surface

A UserToken carries optional allowed_libraries / allowed_tools fields. These are honoured only on the MCP surface (/mcp/):

• /mcp/ — MCPAuthMiddleware enforces allowed_libraries (fail-closed: empty list = zero libraries) and allowed_tools (empty list = any tool). This is the surface third-party clients (Claude Desktop, Cline) use.
• /library/api/*, /mcp_server/api/teams/* — The DRF auth class resolves who is calling. Access is gated by Team.owner (mcp_server) and Library.owner_username (library workspaces). The scope claims are ignored. Daedalus tokens are therefore unrestricted; the user identity plus owner-scope is the access model.

The rationale: enforcing allowed_libraries on the REST endpoints would force Daedalus to mint an effectively-unrestricted token (since it manages the whole workspace lifecycle), which would defeat the field. Owner-scope already encodes the right access pattern there.

3.4 Resolved-library abstraction (MCP)

Mnemosyne's MCP auth middleware populates a single resolved_libraries: list[str] per request. Downstream code (search, get_chunk, …) only reads that list.

Bearer → classify → dispatch
                     ├─ Opaque UserToken    → token.allowed_libraries (JSON list of UIDs)
                     └─ team JWT (typ=team) → live DB join:
                                                TeamWorkspaceAssignment.workspace_id
                                                → Library.workspace_id → Library.uid
                                 ↓
                   resolved_libraries: list[str]
                                 ↓
                         downstream tools

Fail-closed: empty resolution → no libraries visible.

---

4. Data model

4.1 Mnemosyne

UserToken (renamed from MCPToken)

mnemosyne/mcp_server/models.py. Per-user opaque bearer. Hashed at rest (SHA-256, 64-char hex).

class UserToken(models.Model):
    user              = FK(User, related_name="api_tokens")
    token_hash        = CharField(64, unique=True, db_index=True)
    name              = CharField(100)
    is_active         = BooleanField(default=True)
    expires_at        = DateTimeField(null=True, blank=True)
    last_used_at      = DateTimeField(null=True, blank=True)
    allowed_tools     = JSONField(default=list, blank=True)
    allowed_libraries = JSONField(default=list, blank=True)
    created_at, updated_at = …

• Plaintext shown once at mint via UserTokenManager.create_token; never persisted.
• Display masking via get_masked_token() returns tok_…<hash[:8]>.
• allowed_* fields apply only on /mcp/ — see §3.3.

LibraryMembership

Unchanged from v1. Roles owner / manager / reader over Neo4j Libraries (joined by uid string since Library is a neomodel node).

Team

v1 + new non-null owner FK:

class Team(models.Model):
    id          = UUIDField(primary_key=True, editable=False)
    name        = CharField(200)
    owner       = FK(User, on_delete=PROTECT, related_name="teams")
    active      = BooleanField(default=True)
    active_jti  = UUIDField(null=True)
    created_at, updated_at = …

Team.owner is set on creation in team_create from request.user. All other team endpoints filter by (pk, owner=request.user); non-owners receive 404, never 403, so a team's existence isn't disclosed across users.

Soft-delete via Team.active = False is unchanged.

TeamWorkspaceAssignment

Unchanged from v1. Live-queried per request; PUT /workspaces/ replaces the assignment set.

MCPSigningKey

Unchanged. Signs team JWTs.

Library.owner_username (new neomodel property)

mnemosyne/library/models.py. For workspace-scoped libraries (i.e. those with workspace_id set), the Mnemosyne username of the creating user. Null for global libraries. Indexed.

owner_username = StringProperty(required=False, index=True)

The workspace endpoints (/library/api/workspaces/…) set this on create and require lib.owner_username == request.user.username for all mutations and reads; non-owners get 404 on GET/PUT and 204 on DELETE (idempotent).

4.2 Daedalus (informational — managed in the Daedalus repo)

Unchanged from v1 except:

• vault_mnemosyne_daedalus_service_password is gone. Daedalus authenticates to Mnemosyne with a UserToken plaintext minted at /profile/tokens/, stored in whatever secret the operator wires (suggestion: vault_mnemosyne_user_token).
• Daedalus's HTTP client sends Authorization: Bearer <plaintext> to every Mnemosyne endpoint (/library/api/*, /mcp_server/api/teams/*, /mcp/). The Token <key> keyword is no longer accepted anywhere.

4.3 Pallas

Unchanged from v1. Static Authorization: Bearer <team-jwt> in fastagent.secrets.yaml.

---

5. JWT claim shapes

Only one JWT shape remains — the team JWT from v1 §5.2:

{
  "iss":  "mnemosyne",
  "aud":  "mnemosyne",
  "sub":  "team:<pallas_instance_uuid>",
  "typ":  "team",
  "iat":  1715000000,
  "exp":  1976000000,
  "jti":  "uuid4"
}

mnemosyne/mcp_server/teams.py:mint_team_jwt.

5.1 Validator changes vs v1

mnemosyne/mcp_server/auth.py:

• resolve_mcp_jwt no longer accepts iss=daedalus. The _JTI_CACHE replay cache still exists but is exercised by no live code path — scheduled for removal in a follow-up cleanup commit.
• _resolve_jwt_actor resolves to team.owner (the Mnemosyne user that created the team) rather than a synthetic service user. Audit log / usage accounting now correctly attribute each turn to the acting user.

def _resolve_jwt_actor(claims: dict):
    if claims.get("typ") != "team":
        raise MCPAuthError("Per-turn JWTs are no longer accepted; mint a team JWT.")
    team = Team.objects.select_related("owner").get(pk=claims["team_id"])
    if not team.active:
        raise MCPAuthError("Team JWT references an inactive team.")
    if not team.owner.is_active:
        raise MCPAuthError("Team owner is disabled.")
    return team.owner

---

6. Auth flow

6.1 Third-party MCP client with UserToken

1. Client sends Authorization: Bearer <plaintext> to /mcp/.
2. MCPAuthMiddleware hashes → looks up UserToken → validates active/expired/user-active.
3. resolved_libraries = list(token.allowed_libraries or []).
4. Fails closed if empty.

6.2 Agent team (Kottos / Mentor / Iolaus / Daedalus-chat-team)

1. Pallas sends Authorization: Bearer <team-jwt> to /mcp/.
2. Middleware validates signature, iss=mnemosyne, typ=team.
3. Loads Team by UUID from sub. Verifies active=True and jti == active_jti.
4. Expands to resolved_libraries via TeamWorkspaceAssignment → Library.workspace_id.
5. The acting user (for audit, usage accounting) is team.owner.

6.3 Daedalus REST control / ingest

1. Daedalus sends Authorization: Bearer <user-token-plaintext> to /library/api/* or /mcp_server/api/teams/*.
2. DRF UserTokenAuthentication (first in the auth stack) resolves the token to its user.
3. Endpoint scopes by Team.owner (mcp_server) or Library.owner_username (library). Non-owner ⇒ 404.

6.4 Browser / web session

SessionAuthentication runs second; cookie-authenticated users hit the DRF browsable API as themselves with no special handling.

6.5 Failure modes

Condition	Response
No Authorization header	401 + WWW-Authenticate: Bearer
Authorization: Token … (legacy DRF keyword)	401 (not consumed by any auth class)
Invalid bearer plaintext	401 + WWW-Authenticate: Bearer
Inactive / expired token	401
Disabled user	401
JWT signature invalid	401 + WWW-Authenticate: Bearer
JWT exp past (+30s leeway)	401
JWT iss not mnemosyne	401
JWT typ not team (legacy per-turn)	401 ("per-turn JWTs no longer accepted")
Team inactive / unknown / jti stale	401
Team endpoint, non-owner caller	404
Workspace endpoint, non-owner caller (GET/PUT)	404
Workspace endpoint, non-owner caller (DELETE)	204 (idempotent)

---

7. REST API — Mnemosyne team lifecycle

Endpoints under /mcp_server/api/teams/ are authenticated as the Mnemosyne user the team belongs to via a per-user UserToken (Authorization: Bearer <plaintext>, minted at /profile/tokens/). Each team has an owner FK; non-owners receive 404 (never 403) so a team's existence isn't disclosed across users.

7.1 POST /mcp_server/api/teams/

Create a team. Team.owner is set to request.user.

Request

{ "id": "a3f1…", "name": "Kottos" }

Response 201 — fresh id

{ "id": "a3f1…", "name": "Kottos", "jwt": "eyJhbGci…" }

Response 200 — same id, same owner (idempotent; no new JWT issued). Response 409 — same id, different owner ("Team id is already in use.").

7.2 DELETE /mcp_server/api/teams/{id}/

Soft-delete (active=False, clear active_jti). Old JWT invalid on next call. Non-owner ⇒ 404.

7.3 PUT /mcp_server/api/teams/{id}/workspaces/

Replace the team's workspace assignment set. Idempotent.

{ "workspace_ids": ["ws_abc", "ws_def"] }

7.4 POST /mcp_server/api/teams/{id}/rotate/

Generate a fresh jti and JWT, replace active_jti. Old JWT invalid immediately.

Upsert-on-missing. If no Team exists for id, rotate creates one owned by the caller (with name = str(id)) and mints its first JWT — the operator clicks "Rotate JWT" in Daedalus settings and things just work even if Daedalus's provision_teams workflow never ran for this PallasInstance. The placeholder name can be edited via admin.

Response	Condition
200 + jwt	Same-owner id (rotates) or fresh id (upserts + mints)
409	id exists under a different owner ("Team id is already in use.")
409	Team is inactive (soft-deleted) — explicit recreate required

The upsert path logs team_rotate upserted_missing team_id=… owner=… at INFO. Surfacing this in metrics is a useful drift signal: Daedalus and Mnemosyne fell out of sync on team provisioning.

7.5 GET /mcp_server/api/teams/{id}/

Read-only detail (no JWT). Used by the Daedalus reconciler.

7.6 /library/api/ingest/ and /library/api/jobs/…

Same owner-scope model as the workspace endpoints: every ingest write, job read, retry, and list filter against Library.owner_username == request.user.username (global libraries with null owner_username remain shared). Cross-user calls get 404 with the same "not registered" wording as a genuinely missing workspace — existence is not disclosed across users. The list endpoint silently filters; a library_uid the caller has no access to returns an empty list rather than 404.

---

8. Daedalus lifecycle hooks

Unchanged from v1 §8 except the HTTP client now sends Authorization: Bearer <UserToken-plaintext> and Daedalus's config exposes one UserToken plaintext (one per Mnemosyne user the Daedalus instance acts on behalf of, in deployments that multiplex).

---

9. Operator workflows

9.1 Register a new Pallas deployment

Unchanged from v1 §9.1.

9.2 Attach a Pallas team to a workspace

Unchanged from v1 §9.2.

9.3 Retire a Pallas deployment

Unchanged from v1 §9.3.

9.4 Rotate a compromised team JWT

Unchanged from v1 §9.4.

9.5 Provision Mnemosyne integration on a fresh Daedalus instance

Replaces v1 §9.5 (provision_teams) and the deleted ensure_service_user flow:

1. Mint a UserToken for the Mnemosyne user Daedalus will act as: /profile/tokens/add/ (UI) or python manage.py create_user_token --user <username> --name "Daedalus". Copy the plaintext (shown once).
2. Stage the plaintext in Daedalus's config as the bearer for all Mnemosyne calls.
3. Run Daedalus's provision_teams to materialize a Team row in Mnemosyne for every existing PallasInstance.
4. Distribute team JWTs to each Pallas deployment as v1 §9.5 describes.

9.6 Issue a UserToken for a third-party MCP client

1. User logs in to Mnemosyne, navigates to /profile/tokens/, clicks "Generate API Token".
2. (Optional) opens the "Restrictions (optional)" section to set allowed_tools / allowed_libraries — these apply only on /mcp/; for purely REST use they can stay empty.
3. Plaintext is shown once on the response page.
4. User pastes plaintext into the third-party client's config (Claude Desktop, Cline, etc.) with Authorization: Bearer ….

The same UI and command (create_user_token) mint tokens for any purpose — Daedalus, MCP clients, scripts, CI. There is no separate "DRF token" category.

---

10. UX changes in Daedalus

Unchanged from v1 §10.

---

11. Migration

11.1 State at the start of v2

• Mnemosyne is not in a production deployment; migrations are reset on schema changes and the project assumes a clean DB on the next release.
• Daedalus has already migrated to Authorization: Bearer <plaintext> and is configured to use a per-user token; the v1 DRF-token shim is no longer used at runtime.
• No live Pallas deployments authenticate via per-turn JWT (the path is removed).

11.2 Order of operations

1. Mnemosyne v2 deploys. New UserTokenAuthentication, owner-scoped REST endpoints, retired per-turn JWT validation, removed authtoken app. Operator mints a UserToken for Daedalus's Mnemosyne account before deploy.
2. Daedalus's config swap. Operator points Daedalus at the new UserToken plaintext. (If Daedalus was still sending Authorization: Token …, switch to Authorization: Bearer … at the same time.)
3. Existing Teams. None expected at the v2 cutover (migrations are reset). If any existed, Team.owner would need backfill; not in scope.

11.3 Rollback

Mnemosyne v2 is a coordinated cutover with Daedalus's bearer-header swap. Rolling Mnemosyne back to v1 without rolling Daedalus back too means Daedalus's Authorization: Bearer … won't be recognised on /library/api/* (v1 only accepted Token). Plan the deploy as a single window.

---

12. Deprecated / removed in v2

Mnemosyne

• rest_framework.authtoken (removed from INSTALLED_APPS). Generated migration drops the authtoken_token table on next migrate; on a reset schema there's nothing to drop.
• rest_framework.authentication.TokenAuthentication and BasicAuthentication (removed from REST_FRAMEWORK["DEFAULT_AUTHENTICATION_CLASSES"]).
• "API Token" card on /profile/settings/ (removed). The whole api_token_regenerate view + URL are gone.
• mcp_server.management.commands.ensure_service_user (deleted).
• daedalus-service user (no longer provisioned by Mnemosyne; no longer assumed by any endpoint).
• MCP_JWT_SERVICE_USERNAME setting (no longer read by _resolve_jwt_actor).
• Per-turn JWT path in mcp_server/auth.py — accepted shapes shrink to typ=team only. _JTI_CACHE is now exercised by no live path; scheduled for cleanup.
• MCPToken (renamed to UserToken); MCPTokenManager, MCPTokenAdmin, MCPTokenCreateForm, MCPTokenEditForm (renamed in lockstep). The mcp_… masked-token prefix becomes tok_….
• create_mcp_token management command (renamed create_user_token).
• /profile/mcp-tokens/ URL prefix (renamed /profile/tokens/); URL names mcp-token-* (renamed token-*).

Daedalus

• vault_mnemosyne_daedalus_service_password (no longer needed; the service user is gone).
• Any code path that distinguished DRF-Token from MCP-Bearer — one bearer header for everything now.

Pallas

No changes from v1.

---

13. Security

13.1 Token lifetimes

• UserToken: until revoked (user) or expires_at. Rotation is manual via the /profile/tokens/ dashboard.
• Team JWT: 10 years. Revocation via Team.active, Team.active_jti, or key rotation.

13.2 Revocation levers

1. PUT /teams/{id}/workspaces/ with [] — team sees nothing, JWT still validates. Useful for pausing without redistributing tokens.
2. DELETE /teams/{id}/ — team inactive, all its JWTs rejected.
3. POST /teams/{id}/rotate/ — active_jti changes; leaked JWT stops working.
4. Revoke a UserToken — /profile/tokens/{id}/revoke/ flips is_active=False; immediate effect for both /mcp/ and REST.
5. MCPSigningKey.retire() — nuclear option for team JWTs.

13.3 At-rest protection

• UserToken.token_hash: SHA-256 of plaintext; plaintext never stored.
• MCPSigningKey.secret_hex: 256-bit hex secret stored in Mnemosyne DB only.
• PallasInstance.team_jwt_encrypted: Fernet-encrypted by Daedalus.

13.4 Audit attribution

Every authenticated request resolves to a real Mnemosyne user:

• Opaque UserToken → token.user.
• Team JWT → team.owner.

Both flow through to usage accounting (LLMUsage, search metrics) and the audit log. The synthetic daedalus-service actor is gone; nothing in the audit trail is attributed to a non-user account.

Notable audit events:

• team_create created team_id=… name=… — fresh team registered.
• team_create idempotent_hit team_id=… — same-owner re-POST.
• team_create owner_conflict team_id=… caller=… — id collision.
• team_rotate team_id=… new_jti=… — explicit rotation.
• team_rotate upserted_missing team_id=… owner=… — rotate created a missing team on the fly. Useful drift signal: Daedalus and Mnemosyne fell out of sync on team provisioning.
• team_delete team_id=… — soft-delete.

13.5 Isolation model

Unchanged from v1 §13.5.

---

14. Testing

14.1 Mnemosyne test surface (relevant to v2)

• resolve_mcp_jwt rejects iss=daedalus / non-team payloads.
• _resolve_jwt_actor resolves to team.owner; rejects per-turn JWTs and inactive owners. See test_auth.py::ResolveJWTActorTest.
• UserTokenAuthentication issues 401 + WWW-Authenticate: Bearer for anonymous and rejected-token cases; 200 for valid bearer; stashes the UserToken on request.auth. See test_drf_auth.py.
• Team endpoints scope by owner; cross-user GET/DELETE/PUT return 404; same-id different-owner POST/rotate returns 409. rotate upserts a missing team owned by the caller. See test_teams_api.py.
• Ingest endpoints (POST /library/api/ingest/, GET/POST /library/api/jobs/…) scope by Library.owner_username. Cross-user writes/reads return 404; list silently filters. The Cypher-touching paths require Neo4j, so the scoping is exercised by the manual e2e plan in §14.3 rather than unit tests.
• UserToken model: hash-at-rest, tok_… masked prefix, allowed_libraries round-trip. See test_token.py, test_models.py.

14.2 Daedalus test surface

Unchanged from v1 §14.2 except:

• HTTP client uses Authorization: Bearer … against every Mnemosyne endpoint.
• Provisioning command depends on a configured UserToken, not the retired daedalus-service Basic-auth credential.

14.3 Integration

• End-to-end: MCP client with UserToken → search scoped to token.allowed_libraries.
• End-to-end: Pallas with team JWT → search scoped to team's attached workspaces.
• End-to-end: Daedalus REST call with UserToken → workspace mutation succeeds only for the owning user; cross-user attempts get 404.
• End-to-end: ingest as one user, then a different user attempts POST /library/api/ingest/, GET /jobs/{id}/, POST /jobs/{id}/retry/ and GET /jobs/?library_uid=<theirs> — first three return 404, the list returns an empty array.
• End-to-end: anonymous REST call → 401 + WWW-Authenticate: Bearer.
• End-to-end: POST /mcp_server/api/teams/{fresh-uuid}/rotate/ on a team Mnemosyne has never seen → 200 + JWT, Team row created with owner=request.user. Second rotate on the same id → 200 with a fresh active_jti. Rotate on an id owned by a different user → 409.

---

15. Phased delivery

#	Phase	Surface	Status
1	Design v1	DAEDALUS_PALLAS_INTEGRATION_v1.md	Superseded
2	Mnemosyne core	LibraryMembership, MCPToken, Team, TeamWorkspaceAssignment, /mcp_server/api/teams/, team JWT mint	Implemented (v1)
3	Pallas cleanup	Remove _fastagent_patch.py internals	Implemented (v1)
4	Daedalus integration	Lifecycle hooks, reconciler, provision_teams, attached-teams UI	Implemented (v1)
5	Per-user REST authorization	Team.owner, Library.owner_username, owner-scope on all Daedalus-facing endpoints, _resolve_jwt_actor → team.owner	Implemented (v2)
6	Token consolidation	Rename MCPToken → UserToken, UserTokenAuthentication DRF class, drop authtoken + DRF Token UI, retire per-turn JWT, Bearer-first auth stack	Implemented (v2)
7	Documentation	This file; updates to mnemosyne_integration.md and deploy.md	Implemented (v2)

---

16. Open items (v2)

• _JTI_CACHE in auth.py is dead code (the per-turn replay path is gone). Cleanup commit pending; not blocking.
• BasicAuthentication is removed from the DRF default stack. If any internal tooling relied on it, that path is now broken and will need an explicit re-add to the relevant viewset's authentication_classes rather than the global default.

---

17. Cross-references

• Mnemosyne MCP auth: mnemosyne/mcp_server/auth.py.
• Mnemosyne DRF auth class: mnemosyne/mcp_server/drf_auth.py.
• Mnemosyne token model: mnemosyne/mcp_server/models.py (UserToken).
• Mnemosyne team REST: mnemosyne/mcp_server/api/teams.py.
• Mnemosyne workspace REST: mnemosyne/library/api/workspaces.py.
• Token self-service dashboard: mnemosyne/mcp_server/views.py, urls.py.
• create_user_token management command: mnemosyne/mcp_server/management/commands/create_user_token.py.
• v1 design (superseded but kept for history): DAEDALUS_PALLAS_INTEGRATION_v1.md.