Sangeetha-Grantha

Metadata	Value
Status	Final
Version	1.2.1
Last Updated	2026-09-10
Author	Sangita Grantha Architect
Document Type	Evidence record

Multi-Source Import Remediation Report

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current feature map.

1. Executive Summary

Following the initial E2E validation of TRACK-063, several critical failures were identified in the multi-source ingestion pipeline. Specifically, the system failed to correctly ingest Sanskrit variants from mdskt.pdf, produced garbled text in English variants, and created duplicate records due to composer identity mismatches.

Through a multi-stage remediation process involving Python extractor patches, Kotlin backend refactoring, and database-level fixes, the pipeline has been hardened to support a clean, unified “System of Record.”

2. Issues & Root Cause Analysis

2.1. Garbled Body Text (English PDF)

Problem: Lyrics contained raw PDF combining marks (e.g., p¯ujite) instead of proper IAST (pūjite).
Cause: The normalize_garbled_diacritics utility was correctly applied to titles, but the body text extractor in the Python worker was skipping this step.
Resolution: Patched tools/krithi-extract-enrich-worker/src/worker.py to normalize the entire segment text block.
Post-Fix Data: Cleaned 13 existing garbled variants in the database using tools/fix_db_garbled.py.

2.2. Sanskrit Ingestion Failures (`mdskt.pdf`)

Problem 1 (Broken Encoding): PyMuPDF extraction returned replacement characters () for the Sanskrit font because the PDF lacked a /ToUnicode map.
Problem 2 (Missing Persistence): The VariantMatchingService correctly identified matches but never wrote the resulting lyrics to the database.
Resolution:
- Garbage Detection: Updated extractor.py to automatically trigger OCR fallback when replacement characters exceed 10% of the page text.
- Match Persistence: Implemented persistMatch() in VariantMatchingService.kt. Approved matches (including auto-approved ones with >85% confidence) now trigger the creation of krithi_lyric_variant records.

2.3. Deduplication & Architectural Alignment

Problem: CSV imports created duplicates because they couldn’t find the existing PDF records.
Root Cause (Identity Split): The ComposerRepository and RagaRepository in the DAL were using simpler normalization than the API’s NameNormalizationService. “Muthuswami” vs “Muttuswami” resulted in separate DB IDs, breaking the metadata-based deduplication.
Resolution:
- Shared Support: Created TransliterationCollapse.kt in the shared-domain module to unify transliteration rules.
- Normalization Unification: Aligned DAL repositories and API services to use the same shared logic.
- Fuzzy Thresholds: Refined DeduplicationService to use a 75% similarity threshold when metadata (Composer/Raga) matches, and 85% otherwise.

3. Final Verification Results

Composition Count: 37 clean, deduplicated compositions.
Multi-Language Support: Confirmed that a single Krithi now correctly holds up to 6 distinct variants (EN-IAST, EN-CSV, SA-Devanagari, and Indic scripts).
Variant Consistency: Variant bloat (duplicate language/script entries) was resolved through repository-level idempotency guards.
Data Quality: Example anantabālakṛṣṇa verified as stored with correct Unicode diacritics.

4. Technical Files Modified

Python: worker.py (normalization), extractor.py (OCR trigger), diacritic_normalizer.py (new Rules).
Kotlin (API): VariantMatchingService.kt (persistence), ImportService.kt (enrichment-on-match), NameNormalizationService.kt (shared logic).
Kotlin (DAL): KrithiRepository.kt (idempotency), ComposerRepository.kt (normalization), VariantMatchRepository.kt (findById).
Kotlin (Shared): TransliterationCollapse.kt (canonical rules).
SQL: 01_reference_data.sql (seed alignment), 30__fix_entity_resolution_cache_schema.sql.

5. Closure Notes

The “System of Record” is now significantly more robust. While 3 sets of duplicates remain in the current test database due to the earlier split-identity issues, the underlying logic is now patched to prevent this in all future imports. A final manual cleanup of these 3 records via the Admin UI will bring the system to 100% health.

Report generated by Sangita Grantha Architect.

Section index · Documentation home · Feature status