| Metadata | Value |
|---|---|
| Status | Active |
| Version | 1.2.0 |
| Last Updated | 2026-09-10 |
| Author | Sangeetha Grantha Team |
| Document Type | Current guide |
The ingestion pipeline turns supported source documents into reviewable canonical composition data. Kotlin owns orchestration, reference resolution, editorial acceptance, and canonical persistence. Python owns extraction and optional enrichment. They communicate through the PostgreSQL extraction queue.
sequenceDiagram
participant Curator
participant API as Kotlin API
participant DB as PostgreSQL
participant Worker as Python worker
participant Processor as Kotlin result processor
Curator->>API: Submit URL, manifest, or extraction request
API->>DB: Record import/task and extraction request
Worker->>DB: Claim pending extraction
Worker->>Worker: Fetch/parse source and validate canonical payload
Worker->>DB: Store result payload or failure
Processor->>DB: Read completed extraction
Processor->>Processor: Match composition and resolve reference identities
Processor->>DB: Persist staging/variants/evidence through services
Curator->>API: Review, approve/map, or reingest
API->>DB: Accepted canon, revision attribution, and audit
There are several entry points and acceptance paths. Do not infer a universal lifecycle from a single worker status: extraction, import review, canonical persistence, and publication are separate states.
| Area | Responsibility | Source |
|---|---|---|
| Manifest/batch orchestration | Upload, batch/job/task/event state and controls | BulkImportRoutes, bulkimport services |
| Individual imports | Source requests, review, mapping, reingestion | ImportService |
| Extraction worker | Queue claim, HTML/PDF strategies, canonical output | worker README |
| Canonical contract | Python validation and Kotlin serialization | schema.py, CanonicalExtractionDto |
| Canonical creation | Composition identity, metadata and creation | KrithiCreationFromExtractionService |
| Lyric variants | Variant/section persistence and provenance | LyricVariantPersistenceService |
| Reference resolution | Composer/raga/tala/deity/temple candidates | EntityResolutionService |
Use Imports for a single source request, Bulk Import for a CSV manifest, or Sources and Processing for extraction operations. Batch controls include pause, resume, cancel, retry, approve/reject, finalize, and report export where supported.
The worker registers HTML and PDF extraction strategies. PDF processing includes text extraction, page segmentation, structural parsing, and OCR fallback. HTML processing is source-aware. Source formats listed in schemas or older plans may extend beyond implemented strategies.
For a local PDF, the path must be readable by the worker process. Compose mounts repository data/pdfs at /app/pdfs; a host-only absolute path cannot be assumed to work inside the container.
Canonical extraction carries composition metadata, ordered ragas, sections, language/script variants, and source context. Python and Kotlin must agree on field names, enums, nullability, and nesting. New producers use canonical payloads; older ScrapedKrithiMetadata compatibility remains in the backend until TRACK-096 convergence is complete.
Parser behavior includes source-specific Indic-script handling, diacritic normalization, section-label cleanup, per-script variations, and repairs for repeated Pallavi/compound labels. Preserve source distinctions while normalizing presentation noise. Never discard a section or invent an Anupallavi simply to force a common template.
Unclassified output uses UNESTABLISHED. Ragamalika must retain ordered membership in krithi_ragas; primary-raga fields alone are insufficient. See the domain model.
Composition matching must distinguish candidates before creating or enriching canon. Raga identity requires mela-qualified match keys and aliases; unknown/ambiguous names go to the curator queue. Raga identity explains attach-alias, confirm-new, and disambiguation.
Inspect source evidence, extracted text, candidate identity, section count/order, and language/script before accepting a result. Structural votes aggregate source assertions; they do not replace review of dissenting evidence. Similarity, extraction confidence, and source authority are different signals.
Acceptance can write current-state projection, versioned section snapshots, source evidence, and audit events through the appropriate service path. See versioned canon. Verify actual workflow state before assuming public publication.
A row claim using FOR UPDATE SKIP LOCKED prevents workers from claiming the same pending row concurrently. It is not a guarantee of exactly-once end-to-end delivery. Retries, worker restarts, duplicate URLs, and result processing still need idempotent handling and verification.
For a parser correction:
POST /v1/admin/imports/{id}/reingest or the appropriate review path.Do not implement composition corrections as new Flyway data-fix migrations. Earlier corpus migrations were retired under TRACK-139; current migration numbers may have been reused for legitimate schema work.
| Symptom | Evidence to inspect |
|---|---|
| Pending request never starts | Worker process/logs, database connectivity, queue status |
| Failed extraction | Source accessibility, container file path, source format, parser/OCR/provider error |
| DONE extraction but no usable composition | Result processor logs, canonical decoding, import state, candidate matching |
| Missing or merged variant | Language/script/source identity, variant match decision, persistence joins |
| Missing Ragamalika sequence | Canonical raga order and krithi_ragas, not just primary_raga_id |
| Unexpected public result | Publication state, V1/V2 form visibility, reader DTO and selected variant |
| Search misses after reingest | Embedding documents/hashes/profile coverage; indexing is separate |
Monitoring and configuration cover operational inspection. Post-import verification gives the final checks.
Run the matching worker/parser tests, backend import/service tests, and affected browser journeys. Use real PostgreSQL/Flyway Testcontainers for database behavior; external source/provider calls should be controlled in deterministic tests. Keep live corpus counts and provider evaluations as dated evidence, separate from code test results.
Use the bulk-import overview for related strategy/history documents. ADR-012 records the ownership decision.