Ingest: unsupported file type is retried 3× and (mis)counted as a 5xx — should be a terminal client-data failure #5

New Issue

r · 2026-07-20T10:54:26Z

r commented

2026-07-20 10:54:26 +00:00

Summary

When Daedalus sends a file whose type Mnemosyne can't parse (e.g. .drawio → vnd.jgraph.mxfile, macro-enabled .xlsm → vnd.ms-excel.sheet.macroenabled.12), the ingest_from_daedalus Celery task treats the resulting ValueError as a transient error and retries it 3× at 60s intervals before finally marking the job failed. An unsupported file type is deterministic — re-parsing identical bytes can never succeed — so all three retries are guaranteed waste.

This surfaced via a production alert (MnemosyneApp5xxErrors, WARNING) that had been firing continuously since 2026-07-13. Investigation showed the app is healthy: no HTTP 5xx are served to any client (all uvicorn.access lines are 200). The only real errors are these worker-side parse failures. The django_http_responses_total_by_status_total{job="mnemosyne",status="503"} counter increments in lockstep with the retry cycles (flat ~36 retries/6h for the whole week), which is what keeps the alert lit.

An unsupported file type is a client/data condition (the input isn't parseable), not a server fault. It should be recorded as a terminal failed job with a clear reason and should not be retried, nor classified as a 5xx.

Confirmed in code

1. Deterministic failure is on the retry path — library/tasks.py:468-476:

except Exception as exc:                        # catches ValueError from the parser
    logger.error("Task ingest_from_daedalus failed job_id=%s: %s", job_id, exc, exc_info=True)
    if self.request.retries < self.max_retries: # max_retries=3, default_retry_delay=60
        job.retry_count = self.request.retries + 1
        job.save(update_fields=["retry_count"])
        raise self.retry(exc=exc)               # <-- retries a permanent failure
    job.status = "failed"
    job.error = str(exc)
    ...

The bare except Exception doesn't distinguish "transient" (S3 hiccup, Neo4j blip, embedding-server timeout — worth retrying) from "permanent" (unsupported file type, corrupt file — never worth retrying).

2. The raise it's catching is deterministic — library/services/parsers.py:111-114:

raise ValueError(
    f"Unsupported file type '{file_type}'. "
    f"Supported: {sorted(PYMUPDF_EXTENSIONS | PLAINTEXT_EXTENSIONS | IMAGE_EXTENSIONS)}"
)

Proposed fix

Primary — don't retry a permanent parse failure. Introduce a non-retriable exception (e.g. UnsupportedFileTypeError(ValueError) raised from parsers.py), and in ingest_from_daedalus treat it as terminal:

except UnsupportedFileTypeError as exc:
    # Deterministic, input-driven — never retry.
    job.status = "failed"
    job.error = str(exc)
    job.completed_at = datetime.now(timezone.utc)
    job.save(update_fields=["status", "error", "completed_at"])
    return {"success": False, "job_id": job_id, "error": str(exc), "reason": "unsupported_file_type"}
except Exception as exc:
    # Genuinely transient — keep the existing retry behaviour.
    ...

This alone eliminates the retry loop and the steady error signal. (Celery's Reject/Ignore or raise self.retry being skipped for this class achieves the same; the exact mechanism is the dev team's call.)

Secondary — classification. Wherever the failure is reflected as an HTTP status, an unsupported/unparseable input should be 4xx (422 Unprocessable Entity is the precise fit) or simply a normal failed job record — not a 5xx. Note: the ingest REST endpoints in library/api/views.py (ingest_create, ingest_job_detail, etc.) do not emit 503 — they use 200/201/202/204/400/404/409/500. I was not able to pin down the exact code path emitting the status="503" counter increment (it isn't in the ingest views, and no 503 appears in access logs — so it may be the readiness endpoint, DRF throttling, or an nginx upstream blip coincident with retries). Flagging for triage rather than asserting a location. The metrics correlation with the retry cadence is strong, but the origin is unconfirmed.

Optional — normalization gap. _normalize_file_type (tasks.py:41-53) doesn't map vnd.ms-excel.sheet.macroenabled.12 → xlsx. Since PyMuPDF can open .xlsm, adding this (and .drawio handling/rejection) to _MIME_TO_EXT would let some currently-rejected files ingest. Lower priority than stopping the retry loop.

Impact

Low user impact (bad files were never going to ingest), but continuous log/metric noise and a permanently-firing production alert.
~36 wasted retry cycles per 6h, each re-fetching the file from S3 and re-running the parser, since 2026-07-13.

Repro

Ingest any file whose type isn't in PYMUPDF_EXTENSIONS | PLAINTEXT_EXTENSIONS | IMAGE_EXTENSIONS (e.g. a .drawio or .xlsm) via the Daedalus ingest path. Observe in the worker log: Retry in 60s: ValueError("Unsupported file type ...") repeating 3× before the job settles to failed.

Sample log line (production, altair, 2026-07-20)

level=ERROR logger=library.tasks funcName=ingest_from_daedalus lineno=469
Task ingest_from_daedalus failed job_id=job_6b6ef3d3538e4a298eb50c0a:
Unsupported file type 'vnd.jgraph.mxfile'. Supported: [...]
  → Task library.tasks.ingest_from_daedalus[706194e5-...] retry: Retry in 60s: ValueError(...)

Filed from an infra investigation of the MnemosyneApp5xxErrors alert. The alert rule itself (Taurus pplg/alert_rules.yml.j2) is also being hardened separately so it keys on genuine 500/502/504 per-instance rather than any 5xx; that's tracked on the infra side and is independent of this code fix.

## Summary When Daedalus sends a file whose type Mnemosyne can't parse (e.g. `.drawio` → `vnd.jgraph.mxfile`, macro-enabled `.xlsm` → `vnd.ms-excel.sheet.macroenabled.12`), the `ingest_from_daedalus` Celery task treats the resulting `ValueError` as a **transient** error and retries it 3× at 60s intervals before finally marking the job `failed`. An unsupported file type is deterministic — re-parsing identical bytes can never succeed — so all three retries are guaranteed waste. This surfaced via a production alert (`MnemosyneApp5xxErrors`, WARNING) that had been firing continuously since **2026-07-13**. Investigation showed the app is healthy: **no HTTP 5xx are served to any client** (all `uvicorn.access` lines are 200). The only real errors are these worker-side parse failures. The `django_http_responses_total_by_status_total{job="mnemosyne",status="503"}` counter increments in lockstep with the retry cycles (flat ~36 retries/6h for the whole week), which is what keeps the alert lit. An unsupported file type is a **client/data condition** (the input isn't parseable), not a server fault. It should be recorded as a terminal `failed` job with a clear reason and should **not** be retried, nor classified as a 5xx. ## Confirmed in code **1. Deterministic failure is on the retry path** — [`library/tasks.py:468-476`](https://git.helu.ca/r/mnemosyne/src/branch/main/mnemosyne/library/tasks.py#L468-L476): ```python except Exception as exc: # catches ValueError from the parser logger.error("Task ingest_from_daedalus failed job_id=%s: %s", job_id, exc, exc_info=True) if self.request.retries < self.max_retries: # max_retries=3, default_retry_delay=60 job.retry_count = self.request.retries + 1 job.save(update_fields=["retry_count"]) raise self.retry(exc=exc) # <-- retries a permanent failure job.status = "failed" job.error = str(exc) ... ``` The bare `except Exception` doesn't distinguish "transient" (S3 hiccup, Neo4j blip, embedding-server timeout — worth retrying) from "permanent" (unsupported file type, corrupt file — never worth retrying). **2. The raise it's catching is deterministic** — [`library/services/parsers.py:111-114`](https://git.helu.ca/r/mnemosyne/src/branch/main/mnemosyne/library/services/parsers.py#L111-L114): ```python raise ValueError( f"Unsupported file type '{file_type}'. " f"Supported: {sorted(PYMUPDF_EXTENSIONS | PLAINTEXT_EXTENSIONS | IMAGE_EXTENSIONS)}" ) ``` ## Proposed fix **Primary — don't retry a permanent parse failure.** Introduce a non-retriable exception (e.g. `UnsupportedFileTypeError(ValueError)` raised from `parsers.py`), and in `ingest_from_daedalus` treat it as terminal: ```python except UnsupportedFileTypeError as exc: # Deterministic, input-driven — never retry. job.status = "failed" job.error = str(exc) job.completed_at = datetime.now(timezone.utc) job.save(update_fields=["status", "error", "completed_at"]) return {"success": False, "job_id": job_id, "error": str(exc), "reason": "unsupported_file_type"} except Exception as exc: # Genuinely transient — keep the existing retry behaviour. ... ``` This alone eliminates the retry loop and the steady error signal. (Celery's `Reject`/`Ignore` or `raise self.retry` being skipped for this class achieves the same; the exact mechanism is the dev team's call.) **Secondary — classification.** Wherever the failure is reflected as an HTTP status, an unsupported/unparseable input should be **4xx (422 Unprocessable Entity is the precise fit)** or simply a normal `failed` job record — not a 5xx. Note: the ingest REST endpoints in `library/api/views.py` (`ingest_create`, `ingest_job_detail`, etc.) do **not** emit 503 — they use 200/201/202/204/400/404/409/500. **I was not able to pin down the exact code path emitting the `status="503"` counter increment** (it isn't in the ingest views, and no 503 appears in access logs — so it may be the readiness endpoint, DRF throttling, or an nginx upstream blip coincident with retries). Flagging for triage rather than asserting a location. The metrics correlation with the retry cadence is strong, but the origin is unconfirmed. **Optional — normalization gap.** `_normalize_file_type` ([`tasks.py:41-53`](https://git.helu.ca/r/mnemosyne/src/branch/main/mnemosyne/library/tasks.py#L41-L53)) doesn't map `vnd.ms-excel.sheet.macroenabled.12` → `xlsx`. Since PyMuPDF can open `.xlsm`, adding this (and `.drawio` handling/rejection) to `_MIME_TO_EXT` would let some currently-rejected files ingest. Lower priority than stopping the retry loop. ## Impact - Low user impact (bad files were never going to ingest), but continuous log/metric noise and a permanently-firing production alert. - ~36 wasted retry cycles per 6h, each re-fetching the file from S3 and re-running the parser, since 2026-07-13. ## Repro Ingest any file whose type isn't in `PYMUPDF_EXTENSIONS | PLAINTEXT_EXTENSIONS | IMAGE_EXTENSIONS` (e.g. a `.drawio` or `.xlsm`) via the Daedalus ingest path. Observe in the worker log: `Retry in 60s: ValueError("Unsupported file type ...")` repeating 3× before the job settles to `failed`. ## Sample log line (production, altair, 2026-07-20) ``` level=ERROR logger=library.tasks funcName=ingest_from_daedalus lineno=469 Task ingest_from_daedalus failed job_id=job_6b6ef3d3538e4a298eb50c0a: Unsupported file type 'vnd.jgraph.mxfile'. Supported: [...] → Task library.tasks.ingest_from_daedalus[706194e5-...] retry: Retry in 60s: ValueError(...) ``` --- Filed from an infra investigation of the `MnemosyneApp5xxErrors` alert. The alert rule itself (Taurus `pplg/alert_rules.yml.j2`) is also being hardened separately so it keys on genuine `500/502/504` per-instance rather than any 5xx; that's tracked on the infra side and is independent of this code fix.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: r/mnemosyne#5