Sangeetha-Grantha

Metadata	Value
Status	Active
Version	1.0.0
Last Updated	2026-09-10
Author	Sangeetha Grantha Team
Document Type	Evidence record

Import Pipeline Analysis & Remediation Report

---

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current feature map.

---

Metadata	Value
Status	Active
Version	1.0.0
Last Updated	2026-02-11
Author	Sangita Grantha Architect

1. Executive Summary

This report documents the end-to-end investigation and remediation of the Krithi import pipeline. The system was failing to correctly deduplicate compositions from disparate sources (PDF vs. Blogspot), resulting in data “spam” and process loops. Through a series of code patches and data fixes, we have hardened the “System of Record” to handle transliteration variances and typos.

2. Issues Identified & Resolved

2.1. Process Loops in Entity Resolution

• Problem: The entity_resolution job was stuck in a restart loop.
• Root Cause: ImportRepository.findOrCreateSource was creating duplicate records for “BulkImportCSV”. When querying for the source, the code expected a single result but found 20, causing a singleOrNull() exception that crashed the worker threads.
• Fix:
  • Updated repository to use limit(1) during source lookup.
  • Applied a database-level UNIQUE constraint on import_sources.name.
  • Consolidated 20 duplicate sources into one.

2.2. Incorrect Title Extraction

• Problem: Krithis were being saved with titles like Guru Guha Vaibhavam: Dikshitar Kriti - ... - Raga ....
• Root Cause: The deterministic scraper was falling back to the HTML <title> because the metadata parser failed to match lines that lacked a Tala specification.
• Fix:
  • Added metaRagaPattern to support Title - Raga format.
  • Implemented cleanTitle() to strip common prefixes.
  • Performed a one-time SQL fix to clean ~480 existing records.

2.3. Deduplication Failures (Fuzzy Matching)

• Problem: “Ananada Natana Prakaasam” (typo in Blogspot) was not matching “ānandanaṭanaprakāśam” (correct IAST in PDF).
• Root Cause: Deduplication relied on strict equality of normalized titles. Normalization also preserved spaces, preventing a match between compounded and split words.
• Fix:
  • Implemented Robust Fuzzy Matching using Levenshtein distance.
  • Metadata Prioritization: If the Composer and Raga match, the title similarity threshold is lowered to 75%. Otherwise, it remains at 85%.
  • Added “Compressed Title” matching (ignoring spaces) in KrithiRepository.

2.4. Composer Duplication

• Problem: Multiple records for “Muthuswami Dikshitar” vs “Muttuswami Dikshitar”.
• Fix: Updated NameNormalizationService to map both muthuswami and mutuswami to the canonical muttuswami diksitar. Aligned seed data to match.

3. Integration Test Results

A custom integration test script (tools/test_integration_pipeline.sh) was executed to simulate the real-world environment (Docker + Python Worker + Kotlin Backend).

• PDF Ingestion: Successfully extracted 21 Krithis from mdeng.pdf (Pages 17-37).
• CSV Ingestion: Successfully triggered a batch import of 20 compositions.
• Provenance: Source evidence correctly linked multiple sources to the same Krithi.
• Multi-script: Successfully verified that a single Krithi now holds up to 6 lyric variants (English, Sanskrit, Tamil, Telugu, Kannada, Malayalam).

4. Current Database State

• Total Krithis: 35 (clean subset from test range).
• Composers: Unique record for Muthuswami Dikshitar.
• Uniqueness: Most duplicates resolved.
• Residual Findings: A few duplicates persist (e.g., “Abhayamba Jagadamba”) where the Raga name extracted from PDF (Kalyāṇi) and Blogspot (Kalyani) differed enough to miss the “Metadata Bonus” threshold. Further refinement of Raga name normalization is recommended.

5. Technical Patches Applied

1. KrithiStructureParser.kt: Added title cleaning and improved regex.
2. ExtractionResultProcessor.kt: Implemented Levenshtein fuzzy logic and metadata-aware thresholds.
3. ImportRepository.kt: Fixed singleton source lookup.
4. NameNormalizationService.kt: Hardened composer and vowel normalization.
5. KrithiRepository.kt: Added compressed title search.
6. 30__fix_entity_resolution_cache_schema.sql: Fixed missing updated_at column.

---

Report generated by Sangita Grantha Architect.

---

Section index · Documentation home · Feature status