🐾 fix(parsers): rasterize SVG instead of failing ingest #6
Reference in New Issue
Block a user
Delete Branch "fix/svg-ingest-rasterize"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The bug
"svg"was in bothIMAGE_EXTENSIONSandPYMUPDF_EXTENSIONS, and the image check ran first — so every SVG reachedPIL.Image.open, which cannot decode vector XML. Ingest raisedUnidentifiedImageError, logged"Failed to read image file_type=svg", and counteddocuments_parsed_total{status="error"}.SVG ingest has never worked. Confirmed empirically, not inferred:
PIL.Image.openon a real note raisesUnidentifiedImageError.There were no SVG tests, which is why it went unnoticed.
The fix
Rasterize to PNG via PyMuPDF (already a dependency — no new system libraries).
This also unblocks the vision stage downstream:
services/vision.pysends each image to a vision LLM as adata:URI, which cannot carryimage/svg+xmleither. Sodescription,ocr_text, and theDEPICTSconcept graph were unreachable for SVG regardless of the parse fix.Why SVG is routed explicitly, before both extension sets
Simply removing
svgfromIMAGE_EXTENSIONSwould let it fall through toPYMUPDF_EXTENSIONS— which parses but does not split pages. Handwritten notes from the Write app store a multi-page document as one SVG holding several<svg class="write-page">children stacked viax/yoffsets, and rendering that whole fails two different ways:width/height(the majority of real files): MuPDF falls back to US-Letter and emits only the top-left corner — ruled lines, zero handwriting, in a perfectly valid PNG. This is the dangerous one: it looks like it worked.Splitting per
write-pageelement is correct for both, and since Write never rewrites existing files, both formats persist indefinitely — this isn't a migration window.Each page becomes its own
ExtractedImagewithext="png"andsource_page=<index>, so every page reaches vision/OCR at a legible size.New dependency
lxml—recover=Trueis required for the real files that aren't well-formed XML, and stdlibElementTreehas no equivalent. (It does find the pages, so it's viable only if you're willing to lose those files.)Verification
test_svg_raster.py+ existingtest_parsers.py), plus 84 across the DB-free service suite — no regressions.The Django test runner couldn't create a test database in my environment (
permission denied to create database), so these were run throughunittestdirectly — none of them touch the DB.Notes for review
services/svg_raster.pyis a deliberate twin of daedalus'sbackend/daedalus/extraction/svg.py(see daedalus PR #11). The repos share no common package and a themis-style shared wheel is disproportionate for ~80 lines, so the duplication is noted in both docstrings — fixes belong in both.UnsupportedFileTypeErrorinparsers.py,tasks.py,test_tasks.py,docs/deploy.md). SinceUnsupportedFileTypeErrorlives in a file I also edited, I staged only my own hunks and verified the staged tree passes standalone in a scratch worktree — this branch does not contain or depend on that work, and it remains uncommitted locally.🤖 Generated with Claude Code
"svg" was in both IMAGE_EXTENSIONS and PYMUPDF_EXTENSIONS, and the image check ran first, so every SVG reached PIL.Image.open — which cannot decode vector XML. Ingest raised UnidentifiedImageError, logged "Failed to read image file_type=svg", and counted documents_parsed_total{status="error"}. SVG ingest has never worked. Rasterize to PNG instead, via PyMuPDF (already a dependency). This also unblocks the vision stage downstream: it sends images to a vision LLM as a data: URI, which cannot carry image/svg+xml either, so the description and OCR fields were unreachable for SVG regardless of the parse fix. Route SVG explicitly before both extension sets. Falling through to PYMUPDF_EXTENSIONS would parse but not split, and a multi-page Write note rendered as one document is either blank (no root width/height, so MuPDF falls back to US-Letter and emits the top-left corner) or an illegible tall strip (root sized across every stacked page). Splitting per write-page element is correct for both formats, and since Write never rewrites existing files both persist indefinitely. svg_raster is a deliberate twin of daedalus's extraction/svg.py — the repos share no common package, so the duplication is noted in both docstrings and fixes belong in both. lxml is a new dependency: recover=True is needed for the real files that aren't well-formed XML, and stdlib ElementTree has no equivalent.