Sangeetha-Grantha

Metadata Value
Status Completed — 108 → 0 mismatch rows; durable pallavi-echo parser 2026-09-05
Version 1.3.0
Last Updated 2026-09-05
Author Sangeetha Grantha Team
Priority P3 — backlog; 2.4% of corpus, no data loss
Depends on TRACK-093 (corpus imported)
Interacts with TRACK-100 (multi-pass Indic parsing), TRACK-079 (previous section-consistency remediation)

TRACK-133: Section-Count Mismatch Remediation (29 krithis)

Implementation summary: track-133-section-mismatch-remediation.md.

Goal

Resolve the residual section-count mismatches left by the Trinity import: 29 krithis (2.4% of 1,226), 108 variant rows. Tracked rather than left to attrition because the failures are not 29 independent data problems — the majority share one root cause and should fall to one fix.

Current state (dev DB, 2026-07-19)

Re-verified 2026-08-30 against the restored corpus (dump sangita_grantha_20260830_post_v56, Flyway v57, after TRACK-136/137 raga-identity work): 108 mismatch variant rows across 29 krithis, all “fewer than canon” (0 rows exceed canon), 1 krithi with no canonical sections, 1,226 total. Every figure is unchanged — the raga-identity/orphan-cleanup migrations do not touch section data, and the restore reproduced the same state. The count is now visible in the live curator dashboard (/v1/admin/curator/stats → sectionIssuesCount: 108). This track’s diagnosis and plan below still stand; nothing needs re-adjudication.

WITH canon AS (SELECT krithi_id, COUNT(*) c FROM krithi_sections GROUP BY 1),
var AS (SELECT v.krithi_id, v.language::text AS lang, COUNT(s.id) c
  FROM krithi_lyric_variants v LEFT JOIN krithi_lyric_sections s ON s.lyric_variant_id = v.id
  GROUP BY 1,2)
SELECT k.title, COALESCE(canon.c,0) AS canon_sections, COUNT(*) AS bad_variants,
       MIN(var.c) AS min_actual, MAX(var.c) AS max_actual, string_agg(DISTINCT var.lang, ',') AS langs
FROM var JOIN krithis k ON k.id = var.krithi_id
LEFT JOIN canon ON canon.krithi_id = var.krithi_id
WHERE var.c <> COALESCE(canon.c,0)
GROUP BY k.title, canon.c ORDER BY bad_variants DESC, k.title;

The dominant pattern — one root cause, not 29

20 of the 29 krithis have every non-primary-language variant collapsed to exactly 1 section, while the canonical structure has 2–17. Examples:

Krithi Canon sections Variants affected Actual
sAdhincenE 11 5 (kn, ml, sa, ta, te) 1 each
rAmAbhirAma raghurAma 8 5 1 each
rAma Eva daivataM 7 5 1 each
cUDarE celulAra 10 5 1 each
Alakalallalaadaga 4 5 1 each

That is the classic “script variant never got segmented — the whole lyric landed in one section” failure, i.e. the section splitter matched headings in the primary script but not in the transliterated scripts. Fixing the splitter for those scripts should clear the bulk of the 108 rows at once, and any per-krithi curation should happen only after that, against the remainder.

The rest

Scope

  1. Diagnose before fixing. Confirm the 20-krithi cluster really is one splitter failure by re-running extraction on 2–3 of them and inspecting where segmentation stops. Do not write per-krithi data fixes until the shared cause is either confirmed or ruled out.
  2. Fix the splitter in tools/krithi-extract-enrich-worker for the affected scripts; add regression cases from the cluster above. Coordinate with TRACK-100’s multi-pass architecture rather than bolting on a parallel path.
  3. Re-extract and re-verify the affected krithis; the mismatch query above should shrink to the genuinely-hard remainder.
  4. Curate the remainder by hand — route through the carnatic-musicologist subagent, since deciding the correct section count for e.g. a 17-section Sri Rama Jaya Rama is a lakshana judgement, not a parsing one.
  5. Investigate the single krithi with no canonical sections at all (1 of 1,226).

Folded-in cleanup

CuratorService.getStats() counts section issues by loading every row of krithi_sections and krithi_lyric_sections into in-memory maps and diffing them in Kotlin (CuratorService.kt:67-84). At 1,226 krithis / 6,809 variants that is two full table scans on every curator-dashboard load, to produce five integers. Replace with a SQL aggregate while this track is already in the file.

Definition of done

Diagnosis (2026-09-02) — evidence from the live pipeline

Re-ran the real extraction pipeline (HtmlTextExtractor.extract → normalize_garbled_diacritics → StructureParser.parse) against the live source pages for three cluster krithis, on the current worker code (post TRACK-100–104), and compared per-variant section counts to the DB mismatch rows.

Krithi DB (imported) Live re-parse now (en / sa / ta / te / kn / ml) Verdict
cUDarE celulAra 1 each non-primary 10 / 10 / 10 / 10 / 10 / 10 Already correct
rAmAbhirAma raghurAma 1 each non-primary 8 / 8 / 8 / 8 / 8 / 8 Already correct
sAdhincenE 1 each non-primary 11 / 5 / 5 / 5 / 5 / 5 Real residual gap

The single-shared-cause hypothesis (as written) is REFUTED — but the theme holds

The premise “the splitter matches headings in the primary script but collapses every transliterated variant to exactly 1 section” is not what the current code does. The DB’s “1 section each” rows are stale pre-fix import residue: TRACK-093 imported the corpus before TRACK-100–104 landed the multi-pass Indic parser, and the corpus was never re-extracted. Two of the three canonical cluster exemplars now parse fully in every script. The failures split into three buckets:

Evidence, detect-column excerpt for sAdhincenE (Devanagari), showing the miss:

detect=SWARA_SAHITYA  | 'स्वर साहित्य'          ← lone full header, matched
detect=None           | 'स्व2. रं(गे)शुडु ...'   ← ordinal marker, MISSED → collapses
detect=None           | 'स्व3. गोपी जन ...'      ← MISSED
detect=None           | 'स्व4. वनितल ...'        ← MISSED  (also 'स्व4(A). ...')
...

Net: re-extract first clears Bucket A; a single scoped splitter addition (inline Indic sva+ordinal marker) clears Bucket B; Bucket C is musicologist work, out of the splitter’s scope.

Spec

Spec Status: Draft (awaiting acceptance — do not implement product code yet)

Requirements

Design

Flagged concerns

Open questions

Plan

Plan Status: Accepted (2026-09-02). Decisions: Q1 → re-extract via Worker CLI batch against the dev DB (5432, now up); Q2 → verify te/kn/ml sva-marker forms against real DB data before finalizing the regex. Kotlin TransliterationCollapse parity mirror is deferred to the kotlin-backend-engineer step (notes retained under Flagged concerns), out of this worker task’s scope.

Files that will change

  1. src/structure_parser.py — add INLINE_INDIC_SWARA_PATTERNS + _INLINE_INDIC_SWARA_PROBE; gate in _detect_section_header (mirrors the INLINE_INDIC_PAC gating, ~L318–350 / detection seam).
  2. tests/fixtures/structure_parser/sadhincene_swara_multiscript.{txt,expected.json} (or a captured tests/fixtures/html/tv_sadhincene_swara.html) — deterministic, no network.
  3. tests/test_structure_parser.py (or new tests/test_indic_swara_markers.py) — the R4 regressions.

Order of work

  1. Close Q2: pull te/kn/ml variant text for sAdhincenE and confirm the sva+ordinal marker form per script; finalize the regex character set.
  2. Capture deterministic fixture(s): sAdhincenE (all scripts) + a Bucket-A “must-stay-correct” fixture (cUDarE or rAmAbhirAma raghurAma).
  3. Add failing tests (Indic sAdhincenE = 11 incl. 7 SWARA_SAHITYA; stays-correct guard; false-positive guard).
  4. Implement INLINE_INDIC_SWARA_PATTERNS + probe gating; make tests pass.
  5. ruff check . --fix → ruff format . → mypy . → pytest (unit + integration).
  6. Restart stack: make dev-down then make dev (sangita-restart-on-kotlin-change covers worker .py).
  7. Re-extract the 29 krithis through the ingestion path (Q1); re-run the track’s mismatch SQL.
  8. Emit the residual list (Bucket C + anything Bucket B did not clear) → carnatic-musicologist.
  9. Route the CuratorService.getStats() SQL-aggregate cleanup to kotlin-backend-engineer (separate change; tracked here, not in the worker diff).

Risks

Proof (from CLAUDE.md “Verifying your work”)

Implementation & Results (2026-09-02)

Code change (landed, splitter only)

src/structure_parser.py — added INLINE_INDIC_SWARA_PATTERNS + _INLINE_INDIC_SWARA_PROBE ((?:स्व|ஸ்வ|స్వ|ಸ್ವ|സ്വ)\d+(?:\([A-Za-z]\))?\s*\.), an _inline_indic_swara_enabled per-block flag set in _build_blocks, and a detection loop in _detect_section_header — mirroring the existing context-gated INLINE_INDIC_PAC seam (no parallel path). The bare स्वर साहित्य group-title line that precedes स्व1. becomes an empty-body block and is dropped by the existing _extract_sections empty-block guard, so the Indic count lands at parity with English (N sub-blocks, not N+1) — verified, not assumed.

Tests (added, none weakened)

Predicted residual (live-source re-parse; DB re-extract still pending — see below)

Re-parsed all 16 named krithis through the real pipeline with the fix applied. 14 of 16 now parse CLEAN (every variant matches canon), including all three swara-sahitya cluster exemplars and every partial/outlier that was named. Two show a uniform −1 across all Indic scripts, and both are lakshana section-count questions, not splitter gaps → route to carnatic-musicologist:

Krithi Canon (en) Indic (sa/ta/te/kn/ml) Nature
rAma Eva daivataM 7 (P + 6×C) 6 (P + 5×C) One charanam merged in Indic; English charanams carry footnote digits — correct charanam count is a structural call.
alakalallalADaga 4 (P + 2×A + C) 3 (P + 1×A + C) English splits a second ANUPALLAVI (celuvu mIraganu); Indic keeps one. Whether that is a distinct anupallavi is lakshana — the Indic reading may be the more correct one.

Live re-extract results (2026-09-02, orchestrator host)

The blocked DB steps below were subsequently run on the live dev stack:

Residual 9 krithis (46 rows) → musicologist input:

Krithi Canon Actual Langs
Rama Rama Rama Sita 14 6 en,kn,ml,sa,ta,te
rAma sItA rAma 10 6 en,kn,ml,sa,ta,te
Alakalallalaadaga 4 3 kn,ml,sa,ta,te
Raanidi Raadu 4 3 kn,ml,sa,ta,te
ennEramum un pAda 6 2 kn,ml,sa,ta,te
enta bhAgyamu 3 2 kn,ml,sa,ta,te
kaNTa jUDumi 3 2 kn,ml,sa,ta,te
rAma Eva daivataM 7 6 kn,ml,sa,ta,te
ramA ramaNa rArA 8 7 kn,ml,sa,te

Plus the 1 krithi with no canonical sections: mAdhavO mAM pAtu.

Blocked steps (environment) — RESOLVED, see above

The dev DB (5432) and Docker were unreachable from the worker’s execution context (pg_isready exit 2; docker ps empty), so these accepted steps could not be run in that context and were handed back for a host with DB access (now completed — see “Live re-extract results” above):

  1. Restart stack — make dev-down && make dev (sangita-restart-on-kotlin-change; covers worker .py).
  2. Re-extract the 29 via the worker route. The worker CLI (src/cli.py) exposes only single-input extract / transliterate; DB-backed re-extraction runs through the queue worker (python -m src.worker, DATABASE_URL=..., polls extraction_queue). Confirm the enqueue path used for the 29 (curator re-extract endpoint vs. seeding extraction_queue) before the batch run.
  3. Re-verify with the track’s mismatch SQL; expect it to collapse to the Bucket-C remainder (the two above + any of the ~13 unnamed cluster krithis the query enumerates — most expected to clear, matching the 14/16 named hit rate).
  4. verify-import on the re-extracted krithis (junction krithi_lyric_sections populated).

Handoffs

Musicologist adjudication (Bucket C) — 2026-09-02

Lakshana review of the residual 9 mismatch krithis (46 rows) + the 1 krithi with no canonical sections. Each verdict is grounded in the actual per-variant krithi_lyric_sections text on the live DB (5432). Proposals only — no DB changes applied here. Two findings overturn the “fewer than canon = variant is short” premise: in several cases the canon is wrong and the shorter Indic reading is the correct one.

Summary of verdicts

Krithi Canon Correct Verdict One-line reason
mAdhavO mAM pAtu 0 10 (ragamalika) FIX (parse + ragamalika) Dashavatara ragamalika; §6.2 violated
ennEramum un pAda 6 6 FIX-parse 4 charanams present but merged into anupallavi
enta bhAgyamu 3 3 FIX-parse charanam present but merged into anupallavi
kaNTa jUDumi 3 3 FIX-parse charanam present but merged into anupallavi
rAma Eva daivataM 7 7 FIX-parse one charanam merged (च4 marker, no period)
rAma sItA rAma 10 6 CANON-WRONG canon charanams 6–9 are empty phantom rows
Rama Rama Rama Sita 14 6 CANON-WRONG canon charanams 6–13 are empty phantom rows
Alakalallalaadaga 4 3 CANON-WRONG / ACCEPT-shorter English mis-split the pallavi; Indic (3) correct
Raanidi Raadu 4 3 CANON-WRONG / ACCEPT-shorter English mis-split the charanam; Indic (3) correct
ramA ramaNa rArA 8 7 CANON-WRONG / ACCEPT-shorter en+ta false-split on word tvac-caraNam; Indic (7) correct

FIX-parse — a second, distinct splitter gap (hand to python-engineer)

These four are not lakshana problems — the full sahitya is present in every Indic variant, but a charanam/section marker in the Indic scripts is not detected, so the charanam(s) collapse into the preceding block. This is the exact analogue of the Bucket-B swara-marker gap, one class over:

Recommendation: extend the inline-Indic section-header detection (same seam/gating as INLINE_INDIC_PAC_PATTERNS / the new INLINE_INDIC_SWARA_PATTERNS) to cover (a) bare pa/anu/ca akshara markers, (b) the full-word inline forms, and (c) the digit-without-period form. Add false-positive guards (a lyric line merely beginning with च… must not split — see ramA ramaNa rArA below for why this matters). Re-extract these four; content is intact, so they should reach canon parity.

CANON-WRONG — canon over-specified; the shorter reading is correct (hand to postgres-engineer / curator)

Incorrect — ragamalika collapsed to one raga (§6.2 violation)

Routing

Parser phase COMPLETE (2026-09-02)

Two splitter fixes (Bucket B swara markers + gap-#2 charanam markers) plus a calibration pass against real extracted source text (not synthetic fixtures) cleared the parser-owned residual.

Stage Bad variant rows Krithis
Baseline 108 29
After splitter fix (round 1) 46 9
After gap-#2 + real-data calibration (round 2) 26 5

Cleared by parser work (24 krithis): the Bucket-A stale-import set, sAdhincenE (swara), plus rAma Eva daivataM (digit च4), ennEramum un pAda (full-word चरनम्/Tamil சரனம்), enta bhAgyamu and kaNTa jUDumi (bare च/ச charanam marker). Worker tests: 352 passed.

Key fixes in structure_parser.py: (a) bare-ca detection relaxed to a self-gated charanam-only pattern with a trailing-whitespace discriminator (च /ச marker vs चॆन्त/செந்த lyric word); (b) Tamil charanam class extended to alveolar ன (U+0BA9); (c) digit-no-period च4; (d) tvac-caraNam false-positive guard.

Remaining 26 rows / 5 krithis are all CANON-side data errors (postgres-engineer phase), not parser gaps: Rama Rama Rama Sita, rAma sItA rAma, Alakalallalaadaga, Raanidi Raadu, ramA ramaNa rArA.

mAdhavO mAM pAtu — reclassified. Not a splitter issue: the HTML extractor yields empty text for the Dikshitar source page (extraction_queue INGESTED, result_count=1, 0 variants persisted). The RAGA_SEGMENT segmentation is correct when text is present (fixture proves 10 stanzas → 10 sections) but has nothing to segment. Route to the html_extractor owner (upstream), plus the postgres-engineer metadata fix (is_ragamalika + 10 krithi_ragas). Do not expect re-extract alone to fix it.

Postgres corrections (Bucket C data fixes) — 2026-09-02

Authored by postgres-engineer. The DB was not reachable from the authoring context, so this is author-and-hand-off: SQL below is applied by the orchestrator on the live stack (sangita_grantha, container sangeetha-grantha-db-1).

Delivery vehicle decision — Flyway versioned migrations (not the curator/API path)

These are one-off corpus-data repairs. The established convention here for exactly this class is a Flyway VNN__ migration carrying its own audit_log write — see V45 (remove stale anupallavi), V46 (delete incomplete variant), V47 (demerge ragamalika), V38, V48. Rationale over the curator/API route: the corrections must survive make db-reset and CI Testcontainers (a curator edit is discarded on reset), are checksum-tracked and reproducible across environments, and are reviewable in git. Each block writes audit_log inline, satisfying the mutation-audit rule (the curator path’s automatic audit is not available to raw SQL, so the audit is explicit). No constraint is weakened; every block is a no-op if the target krithi is absent (fresh reset before corpus load), matching V45–V47 behaviour.

Grouping so the orchestrator can act now vs. hold:

Group Vehicle Status Contents
1 database/migrations/V58__track133_delete_phantom_empty_charanam_sections.sql (written) APPLY NOW phantom-empty deletes: rAma sItA rAma 10→6, Rama Rama Rama Sita 14→6
2 held SQL → promote to V59__ on sign-off HOLD for user canon over-count merges: Alakalallalaadaga 4→3, Raanidi Raadu 4→3, ramA ramaNa rArA 8→7
3 held SQL → promote to V60__ on sign-off HOLD for user mAdhavO mAM pAtu ragamalika metadata (is_ragamalika + 10 krithi_ragas)

Held SQL (Groups 2 & 3) is intentionally not placed under database/migrations/ — a VNN file there auto-applies on the next make migrate. It lives at scratchpad/TRACK-133-held-V59-V60.sql (session scratchpad) until approved; promote to the next free version numbers at that time (do not pre-assign — V58 may not be the last committed migration by then).

Group 1 — APPLY NOW (V58, written)

Adjudication is unambiguous: for both krithis the canonical charanams beyond position 5 have zero text in every variant, so the true structure is Pallavi + 5 Charanams = 6. The migration is self-verifying: it only deletes sections with no non-blank text in any variant and asserts the surviving count is exactly 6, rolling back otherwise. krithi_lyric_sections.section_id is ON DELETE CASCADE, so dangling empty lyric-section rows go with the parent (no orphan/ FK risk; no constraint weakened). Contiguity holds because the empties are trailing.

Pre-check (run before applying to confirm on the live DB):

-- Proves the sections to be deleted are empty across ALL variants, and that
-- exactly 6 non-empty sections remain. deletable should be 4 and 8 respectively.
SELECT k.title,
       count(*) FILTER (WHERE nonempty)      AS keep_nonempty,   -- expect 6, 6
       count(*) FILTER (WHERE NOT nonempty)  AS deletable        -- expect 4, 8
FROM krithis k
JOIN LATERAL (
    SELECT cs.id,
           EXISTS (SELECT 1 FROM krithi_lyric_sections ls
                   WHERE ls.section_id = cs.id AND COALESCE(btrim(ls.text),'') <> '') AS nonempty
    FROM krithi_sections cs WHERE cs.krithi_id = k.id
) s ON true
WHERE k.title IN ('rAma sItA rAma','Rama Rama Rama Sita')
GROUP BY k.title;

Mutation + audit: database/migrations/V58__track133_delete_phantom_empty_charanam_sections.sql (committed migration; make migrate). It loops both titles, guards non-empty=6, deletes phantoms, re-asserts remaining=6, and writes one audit_log DELETE row per krithi.

Post-check:

SELECT k.title, count(*) AS sections,
       min(cs.order_index) AS min_ord, max(cs.order_index) AS max_ord
FROM krithis k JOIN krithi_sections cs ON cs.krithi_id = k.id
WHERE k.title IN ('rAma sItA rAma','Rama Rama Rama Sita')
GROUP BY k.title;               -- expect sections = 6; contiguous order_index
-- Audit trail:
SELECT entity_id, action, diff FROM audit_log
WHERE metadata->>'migration' = 'V58' ORDER BY changed_at;

Group 2 — HELD (V59 candidate): canon over-count merges

Alakalallalaadaga (P,A,A,C→P,A,C), Raanidi Raadu (P,A,C,A→P,A,C), ramA ramaNa rArA (8→7). Unlike Group 1 the spurious section carries real text in the offending variant (English for the first two, en+ta for the third), so the correct fix merges it into its adjudicated neighbour and then deletes it — never a blind delete. Held because the exact (keep, drop) section pair must be confirmed against live rows first. Full SQL — read-only diagnostic, a reusable merge helper block (append text per variant → repoint variant-only rows → delete → close the order_index gap → audit MERGE_SECTIONS), and a post-check expecting 3/3/7 — is in scratchpad/TRACK-133-held-V59-V60.sql. No FK/section-order invariant is broken: the gap-close keeps (krithi_id, order_index) contiguous and the unique constraint satisfied.

Group 3 — HELD (V60 candidate): mAdhavO mAM pAtu ragamalika metadata (metadata only)

Sets is_ragamalika = true and replaces the single krithi_ragas row with 10 ordered rows (order_index 1..10) for the Dashavatara sequence: nATa, SrI gauLa, SrI, Arabhi, varALi, kEdAra, vasanta, suraTi, saurAshTra, madhyamAvati — resolving each raga through the TRACK-136/137 identity fold (ragas.match_key ∪ raga_aliases.match_key via raga_match_key()), so no duplicate raga rows are created. The block RAISES on any unresolved or ambiguous name so identity is fixed first rather than guessed. order_index is 1-based to match the ragamalika convention actually in use (V57 asserts 1..34), not V02’s stale “0-based” comment — flagged for confirmation. primary_raga_id is set to the pallavi raga (nATa) as a display headline with a curator toggle to NULL; §6.2’s “never collapse to one raga” is satisfied by the 10 krithi_ragas rows, which are authoritative. Full SQL in scratchpad/TRACK-133-held-V59-V60.sql.

BLOCKED upstream — scope boundary: the lyric/section side of mAdhavO cannot be populated here. The html_extractor yields empty text for the Dikshitar source page (0 variants persisted), so no canonical sections can be built until that extractor is fixed (html_extractor owner). This migration is metadata-only; it deliberately does not touch krithi_sections for this krithi. After extraction is fixed and text lands, the RAGA_SEGMENT parser (already proven on fixtures) builds the 10 sections via the normal ingestion path — not via a migration.

Handoff summary

Postgres phase progress (2026-09-02, orchestrator-applied)

Delivery vehicle: Flyway VNN__ migrations with inline audit_log writes (per ADR-013, precedent V45–V48). Applied on the live stack via make migrate.

Held for musicologist Round 2 (concrete per-variant data captured):

Deferred: mAdhavO mAM pAtu (Group 3, V60 drafted but HELD). Doubly-blocked — (a) html_extractor yields empty text for the Dikshitar page (0 sections buildable), (b) 3 of 10 Dashavatara ragas (SrI gauLa, kEdAra, saurAshTra) don’t resolve to canonical raga IDs. Spun out to a dedicated follow-up (extractor fix + raga-identity resolution, then promote V60). Not applied this pass.

Round 2 — pre-postgres-fix confirmations (2026-09-02)

Confirmed against live per-variant text. Both proposed framings need correcting — the naive “merge the two adjacent same-type sections” and “Indic is missing the last charanam” are each wrong.

1) Alakalallalaadaga — verdict CANON-WRONG, correct = 3 (P + A + C). But NOT the proposed merge.

The Indic variants already encode the correct reading: te oi1 PALLAVI holds both pallavi lines (alakalallalADaga kaniya / rAN-muniyeTu pongenO), oi2 ANUPALLAVI = celuvu mIraganu mArIcuni…, oi4 CHARANAM. That is the standard lakshana of this Utsava-Sampradaya kriti: the pallavi sentence is “alakalallalADaga kani(y)A rAN-muni(y)eTu pongenO” (“seeing the swaying curls, how the sage-king thrilled”) — rAN-muniyeTu pongenO is the tail of the PALLAVI, not an anupallavi. The true anupallavi is celuvu mIraganu mArIcuni madam(a)NacE vELa.

The English variant over-split: it broke the pallavi across oi1+oi2 and pushed the real anupallavi to oi3. So canon oi2 (rAN-muniyeTu pongenO, labelled ANUPALLAVI) is the spurious section.

2) ramA ramaNa rArA — verdict: true structure P + 6C = 7. Canon (7) is CORRECT — keep it. The Indic variants are NOT missing text.

Six distinct charanams exist and all six are present in every variant, including the Indic ones. The kriti’s charanams are: (1) samAnamevaru… (2) budhAdyavana… (3) kalArtha bhUsha… (4) raNAdhi SUra SaraNAgata tvac-caraNam bhava tAraNambu cEsunu (5) mukhAbjamunu Sata mukhAri… durmukhAsura haraNa (6) birAna brOvaga rAdA… tyAgarAja sannuta.

The Indic variants read 6 only because charanams 4 and 5 are MERGED into one section (verified in sa, te, ml: oi5 contains both raNAdhi…cEsunu (ramA) and mukhAbjamunu…haraNa (ramA)), which shifts C6 (birAna…) up to oi6 and leaves canonical oi7 empty. The “empty oi7” is a cascade artifact of that merge — not a missing stanza. en/ta at 7 are correct (V59 having merged their earlier tvac-caraNam over-split gives the correct C4 = raNAdhi… tvac-caraNam bhava tAraNambu cEsunu).

Net for the postgres fix: Alakalallalaadaga → drop oi2 (reindex to P,A,C=3) + repair en variant text; ramA ramaNa rArA → no canon change (parser re-split of the 4 Indic variants instead).

Last remaining — ramA ramaNa rArA (parser, not canon). Musicologist Round-2: canon 7 is correct; the 4 Indic variants (sa,te,kn,ml) under-segment — charanams 4 & 5 are glued into one section, each ending with the pallavi-echo refrain (రమా). The section must split at the internal …cEsunu (రమా) | ముఖాబ్జమును… boundary. Same class as rAma Eva daivataM’s (राम) refrain split (which works), so a narrow refrain-boundary detection gap for this case.

Kotlin phase + final state (2026-09-02)

Kotlin phase complete.

Live verification: /v1/admin/curator/stats → sectionIssuesCount: 4, matching the mismatch query.

Accepted residual (Definition of Done)

Final tally

Stage Mismatch rows Krithis
Baseline 108 29
Parser (splitter fixes + real-data calibration) 26 5
V58 (phantom-empty deletes) 14 3
V59 (mis-split merges) 9 2
V60 (Alakalallalaadaga pallavi fix) 4 1
Accepted residual 4 1 (ramA ramaNa rArA)

104 of 108 rows resolved (96%); the remaining 4 are consciously accepted. Migrations V58–V60 (audited). Worker regression suite covers the marker forms (352 tests). Curator dashboard now reflects the corrected corpus via a SQL aggregate.

Follow-up spun out

Round 3 — mAdhavO mAM pAtu ragamalika (2026-09-02)

Lakshana confirmation for the Dashavatara Ragamalika. Propose only.

1. Section structure — 10 sections is correct; do NOT make 20. Each avatara stanza is one section. The (madhyama kAla sAhityam) block is the tempo-doubled tail of the same stanza in the same raga — it is not a raga change and not a structural section of its own, so it is folded into its stanza’s section. Splitting into 20 would falsely imply 20 raga/structure boundaries. Correct = 10.

SectionType: keep all 10 as OTHER — do NOT type them PALLAVI/ANUPALLAVI/CHARANAM. This kriti has no refrain: the header note (“first eight vibhaktis in order, last two in the second vibhakti”) confirms it is ten parallel avatara verses in successive Sanskrit grammatical cases, with no returning pallavi and no anupallavi. Typing stanza 1 as PALLAVI would misrepresent a non-refrain opening verse as a refrain; PALLAVI + 9×CHARANAM is therefore wrong. OTHER per stanza is the domain-honest choice for ragamalika verse-sections. (If a more descriptive type is later wanted, uniform CHARANAM is the only acceptable fallback — never PALLAVI+9.) Recommend each section carry a label = its raga name (e.g. nATa (Matsya)) and, if captured, mark the madhyamakala tail in notes, so the fold-in is not lost.

2. Raga sequence — confirmed. The ten ragas, order, spellings, and avatara mapping are the standard Dashavatara sequence and all check against the sahitya:

# Raga Avatara (from sahitya)
1 nATa Matsya (matsyAvatArO)
2 gauLa Kurma (kUrmAvatAraM)
3 SrI Varaha (bhUmi pAla sUkarENa)
4 Arabhi Narasimha (narasiMhAya namastE)
5 varALi Vamana (vAmanAt…)
6 kEdAra Parasurama (paraSu rAmasya)
7 vasanta Rama (rAma candra svAmini)
8 suraTi Balarama (bala rAma)
9 saurAshTra Krishna (SrI kRshNaM bhajarE)
10 madhyamAvati Kalki (kali yuga vara vEnkaTESaM)

Note on #2: the round-1 reading “SrI gauLa” was an artifact — the SrI in the imported latin blob is an honorific prefix (as it also precedes deity names elsewhere in the text, e.g. SrI kRshNaM, SrI dharENa). The Devanagari source’s गौळ रागं = plain gauLa. Confirmed #2 = gauLa, not a compound “SrIgauLa”. §6.2 action stands: set is_ragamalika = true and add 10 ordered krithi_ragas rows (order_index 1–10); primary_raga_id may remain nATa as the opening-raga pointer but is not the sole representation.

3. Raga identity / alias calls — both confirmed same raga; alias, do not duplicate.

Both are just anusvara/final-consonant spelling variants; aliasing avoids duplicate raga rows. High confidence on structure, sequence, and both identities. Authorship (Muthuswami Dikshitar) is consistent with the vibhakti-based style but is taken as given here — not independently re-verified against a source.

mAdhavO mAM pAtu — RESOLVED (2026-09-02), was never actually blocked

The “html_extractor yields empty text” conclusion was wrong. Real diagnosis + fix:

Findings to route separately (noted; minor)

  1. Ingestion re-persistence gap — reviewImport only persists sections in the create/first-promote path; once a krithi is APPROVED+mapped, re-approving short-circuits (alreadyPromoted early-return), so a re-extraction of an already-mapped krithi never re-persists sections. Worked around here by resetting the import to in_review and re-approving. Only bites stale already-mapped krithis; fresh db-reset imports persist correctly on first promotion. Worth a backend follow-up.
  2. Alika title mis-parse — the metadata parser reads “mAlika” (from “Dasa Raga Malika”) as a raga. V61 overwrites the result, but the parser bug recurs on re-extract. Minor worker follow-up.
  3. Tamil MKS marker — the Tamil (madhyama kala sahitya) form isn’t demoted (stays inline); the 10-way split is unaffected. Minor worker follow-up.

Final tally (all krithis)

108 → 4 mismatch rows / 1 krithi (ramA ramaNa rArA, consciously accepted & documented). mAdhavO fully resolved. Migrations V58–V61 (all audited). Parser: 3 marker fixes + ragamalika multiscript split. Kotlin: CuratorService SQL aggregate. Curator sectionIssuesCount reflects the corrected corpus.

Findings resolved (2026-09-02)

The four findings surfaced during remediation were fixed in-session (not deferred):

  1. /re-extract 1000-row cap — new DAL ExtractionQueueRepository.findIdsBySourceUrlPattern(pattern, status?) does the match in SQL (lower(source_url) LIKE, wildcard-escaped, uncapped); the route iterates the returned ids. Response shape/audit unchanged. Test seeds 1001 rows and asserts all requeued.
  2. Ingestion re-persistence gap — new ImportService.reingestMappedKrithi(id) + POST /v1/admin/imports/{id}/reingest: atomically clears variants and re-runs persistLyricVariants/ persistFromCanonical onto the mapped krithi (idempotent, no duplicate krithi, audited REINGEST_MAPPED_KRITHI). Tests: 0→3 sections after reingest + double-reingest idempotency.
  3. Alika title mis-parse — metadata_parser.py now recognises the “rAga mAlikA”/”daSa rAga mAlikA” ragamalika descriptor, sets is_ragamalika, and emits no bogus raga (tala still recovered).
  4. Tamil MKS marker — structure_parser.py Tamil MKS regex now tolerates the grantha-numeral form (மத் 4 யம கால ஸாஹித்யம்), so Tamil demotes MKS like the other scripts.

Worker suite 358 passed; backend ReExtractCapTest + ReingestMappedKrithiTest green; dal/api compile clean.

Last residual CLOSED — 0 mismatches (2026-09-02)

ramA ramaNa rArA was reclassified from “accepted residual” to fixed after it was inspected in the Curator UI: the 4 Indic variants (sa/te/kn/ml) visibly glued charanams 4+5 under “CHARANAM 4” and skipped to “CHARANAM 6”, while English/Tamil were correct.

V62 applied — splits each Indic variant’s merged oi5 at the internal pallavi-echo refrain “(ramA)”: part A (C4) stays in oi5, part B (C5) → oi6, and the shifted C6 (birAna) → a new oi7 row. Canon (7) unchanged; en/ta untouched. Self-guarding (no-op if oi7 already present / structure differs), audited (RESPLIT_SECTIONS). All 6 variants now read 7, content verified (oi5=raNAdhi/C4, oi6=mukhAbjamunu/C5, oi7=birAna/C6).

Known parser gap (durable fix; resolved below on 2026-09-05): the splitter still doesn’t split a charanam at an internal pallavi-echo refrain, so a fresh re-extract could reintroduce the glue and V62 would need re-running. The corpus-repair migrations (V58–V62) are tied to the imported snapshot by design; the durable fix is a parser refrain-split.

FINAL: mismatch query returns 0 rows / 0 krithis (was 108 / 29). Definition of Done met.

Durable pallavi-echo parser fix — verified 2026-09-05

Plan Status: Accepted — user explicitly requested implementation of the documented durable worker fix, four Indic regressions, English/Tamil and ordinary-refrain protection, the full worker suite, and validation against the source URL.

Implemented: StructureParser._split_charanam_pallavi_echoes splits variant blocks before canonical mapping. For an Indic variant with fewer charanams than canon, it recognises a line-final parenthesised echo matching both the variant pallavi’s opening word and its closing refrain. An internal echo closes the preceding charanam; the following lyric starts the next charanam. The refrain stays attached to C4, C5 gets its own source span, and the existing birAna C6 maps to oi7. The final stanza must also close with the same echo, the next line must use the same script, and exactly one glued CHARANAM block may be repaired — that block’s extra stanza count must equal the charanam deficit. Summing cuts across several charanams is rejected (a spurious mid-stanza echo plus a real glue elsewhere must not add up to a repair). Otherwise no repair is attempted. Already aligned variants, Latin text, Anupallavi bodies, inline parentheses, and terminal-only refrains are unchanged. No composition title, database ID, or V62 dependency is embedded in the parser.

This runs in the shared variant-block path, including TRACK-100 post-metadata Indic extraction. Kotlin parity remains as established above: ingestion persists worker sections without re-segmenting them; the deprecated Kotlin scraper is not part of this path. No DTO or migration changes.

Fixtures and tests: fixture provenance and focused tests. The four Indic repairs assert exact text/type/order/label for all seven sections, distinct C4/C5/C6 source spans, and identical canonical English output, before and after a metadata boundary. English/Tamil retain their complete text, including the tvac-caraNaM continuation. Additional regressions cover explicit headings, ordinary internal and terminal refrains, inline parentheses, non-pallavi parentheses, missing closing cues, non-charanam blocks, ambiguous/excess/insufficient boundaries, mixed true+false cuts across blocks, two missing headings in one glued block, an Anupallavi whose echo lines stay unsplit, and stable re-parsing. Deliberate test edits used SANGITA_ALLOW_TEST_EDITS=1.

Verification evidence

Commands run from tools/krithi-extract-enrich-worker with the existing Python 3.14.7 environment:

.venv/bin/pytest tests/test_pallavi_echo_split.py -q
Before fix: 9 failed, 17 passed (four Indic repairs × two paths, plus re-parse)
After fix:  26 passed in 0.14s

.venv/bin/pytest -q
384 passed, 94 warnings in 20.01s

.venv/bin/ruff check src/structure_parser.py tests/test_pallavi_echo_split.py
All checks passed!

.venv/bin/mypy .
Success: no issues found in 62 source files

Warnings originate from third-party deprecations (google-genai, PyMuPDF/SWIG, and indic-transliteration). Ruff formatting and git diff --check also passed.

Live source validation: fetched the complete original source page and ran HtmlTextExtractor.extract(html, base_url=url) → normalize_garbled_diacritics → StructureParser.parse. Canon = 7, and en/sa/ta/te/kn/ml = 7 each. Every variant’s complete text/type/order/label matches the pre-change live snapshot in the expected JSON fixture.

Reproduction distinction: the current live page explicitly contains all C5 headings and already parses to seven sections before this change. It does not reproduce the historical failure as-is. Removing only the four Indic C5 marker prefixes from the full extracted live text reproduces the documented merged C4+C5 shape: before the fix, sa/te/kn/ml = 6; after it, all six variants = 7, with exact equality to the intact live-page output. The committed fixture records this controlled transformation explicitly; it is not presented as an original failing HTML snapshot.

Runtime verification: started Docker Desktop and ran make dev-down then make dev. The worker image rebuilt successfully, and the Compose source mount is present. A standalone docker compose run --rm --no-deps -T --entrypoint python extraction - check parsed the merged fixture and verified all six variants against the expected JSON. It logged the new TRACK-133 pallavi-echo split: restored 1 ... charanam boundary(s) marker once for each of Devanagari, Telugu, Kannada, and Malayalam.

Environment limitation: full-stack startup stopped at PostgreSQL with FATAL: could not write lock file "postmaster.pid": No space left on device. Backend health and DB/API verification therefore could not complete. No corpus re-ingestion, data deletion, or V62 rerun was performed; source validation here is worker extraction/segmentation, not a claim of newly persisted database state. Docker storage must be freed before the stack can start. V62 remains the historical snapshot repair; the missing-heading case now has worker coverage.

Docker startup blocker resolved (2026-09-05)

The user requested resolution of the Docker disk error. Host storage had 82 GiB free, but Docker’s 60 GiB virtual disk was full (df: 59G filesystem, 56G used, 0 available, 100%). Inodes were only 38% used. Removed build cache older than seven days and old dangling images, then reclaimed the build layers released by those images. Docker reported 1.799 GB + 2.463 GB + 3.462 GB reclaimed; the filesystem now has 7.7 GiB available (87% used). No volumes or application containers were pruned.

make dev subsequently succeeded: PostgreSQL completed recovery, Flyway validated 72 migrations and confirmed schema V62 was current without applying migrations, and the extraction worker connected to the database. GET http://localhost:8080/health returned OK; the admin UI at http://localhost:5001/ returned HTTP 200. A read-only count confirmed 1,226 krithis remain. This resolves the runtime-startup limitation above; no corpus re-ingestion was performed.

Post-validation hardening (2026-09-05)

Independent review of the uncommitted parser found no production bugs. Two nits were then applied before commit:

  1. Anupallavi negative test now maps Telugu against an English Anupallavi, so the assertion is the unsplit Anupallavi body (type + full text), not “Pallavi survived after canonical mapping dropped the Anupallavi.”
  2. Single glued block — _split_charanam_pallavi_echoes repairs only when exactly one CHARANAM block has internal echo boundaries and that extra-stanza count equals the deficit. A mixed true+false cut across two charanams is rejected. A single block short by two headings still splits.

Independent re-run after those edits: focused pallavi-echo + charanam-guard 54 passed; full worker suite 386 passed; ruff and mypy . (62 files) clean. Live mismatch SQL still 0 rows; ramA ramaNa rArA remains 6 variants × 7 non-empty lyric sections. The mounted extraction image parsed the merged fixture to 7 sections in every language.