feat(ingest): source-bucket registry keyed on ingest source
Generalises the Daedalus-only cross-bucket fetch into a registry (SOURCE_S3_BUCKETS) keyed on the IngestJob `source` field, so new upstream sources (Spelunker) can ingest from their own buckets. The ingest task now calls fetch_from_source(job.source, job.s3_key) and falls back to "daedalus" for blank/unknown sources (backwards compatible). Adds SPELUNKER_S3_* env vars and worker env scoping. Replaces daedalus_s3.py with source_s3.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -339,6 +339,13 @@ services:
|
||||
- DAEDALUS_S3_REGION_NAME=${DAEDALUS_S3_REGION_NAME}
|
||||
- DAEDALUS_S3_USE_SSL=${DAEDALUS_S3_USE_SSL}
|
||||
- DAEDALUS_S3_VERIFY=${DAEDALUS_S3_VERIFY}
|
||||
- SPELUNKER_S3_ENDPOINT_URL=${SPELUNKER_S3_ENDPOINT_URL}
|
||||
- SPELUNKER_S3_ACCESS_KEY_ID=${SPELUNKER_S3_ACCESS_KEY_ID}
|
||||
- SPELUNKER_S3_SECRET_ACCESS_KEY=${SPELUNKER_S3_SECRET_ACCESS_KEY}
|
||||
- SPELUNKER_S3_BUCKET_NAME=${SPELUNKER_S3_BUCKET_NAME}
|
||||
- SPELUNKER_S3_REGION_NAME=${SPELUNKER_S3_REGION_NAME}
|
||||
- SPELUNKER_S3_USE_SSL=${SPELUNKER_S3_USE_SSL}
|
||||
- SPELUNKER_S3_VERIFY=${SPELUNKER_S3_VERIFY}
|
||||
# Celery / RabbitMQ
|
||||
- CELERY_BROKER_URL=${CELERY_BROKER_URL}
|
||||
- CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
|
||||
|
||||
Reference in New Issue
Block a user