Sangeetha-Grantha

Metadata	Value
Status	Active
Version	1.0.0
Last Updated	2026-09-10
Author	Sangita Grantha Architect
Document Type	Evidence record

Implementation Summary: Data Remediation & Deduplication Hardening (TRACK-040, TRACK-041, TRACK-061)

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current quality checks.

Purpose

This changeset hardens the deduplication logic to handle transliteration variances (e.g., sh vs s) and typos in source data. It also introduces services for identifying and merging duplicate Krithis and cleaning metadata artifacts from imported text.

Changes

Normalization Logic

modules/backend/api/.../services/NameNormalizationService.kt: Implemented transliteration-aware normalization (e.g., ks ≈ ksh, sh ≈ s). Added canonical maps for common composer name variants.
modules/backend/api/.../services/scraping/KrithiStructureParser.kt: Enhanced deterministic parsing to strip boilerplate and handle subscript digits.
modules/backend/api/.../services/MetadataCleanupService.kt: Utility for stripping “Meaning:”, “Notes:”, and “Updated on:” lines from imported lyric blobs.

Deduplication Engine

modules/backend/api/.../services/DeduplicationService.kt: Implemented Levenshtein fuzzy matching.
Metadata Prioritization: If Composer and Raga IDs match, the title similarity threshold is lowered to 75% (vs 85% default).
modules/backend/dal/.../repositories/KrithiRepository.kt: Added findDuplicateCandidates using “Compressed Title” matching (REPLACE spaces) to find near-matches regardless of word splitting.

Data Seeding

database/seed_data/01_reference_data.sql: Aligned canonical composer names to match the new normalization rules.

Verification Results

Compressed Matching: Verified “Ananada Natana Prakaasam” correctly matches “anandanatanaprakasam”.
Composer Aliases: Verified “Muthuswami Dikshitar” and “Muttuswami Diksitar” resolve to the same entity.
Cleanup: Verified all literal ` ` characters and technical noise are stripped during ingestion.

Commit Reference

Ref: application_documentation/07-quality/track-040-remediation-deduplication-implementation.md

Section index · Documentation home · Feature status