Sangeetha-Grantha

Metadata Value
Status Active
Version 1.0.0
Last Updated 2026-09-10
Author Sangita Grantha Team
Document Type Evidence record

TRACK-064 Unified Extraction Migration Handoff (2026-02-12)

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current feature map.

Purpose

This document captures the implementation status, validation runs, detailed non-convergence analysis, and next actions for the extraction unification work (HTML + PDF + queue/worker/ingest).

Use this as the starting context for the next session.

Follow-up (2026-02-13)

Snapshot (as of February 12, 2026)

Implemented Fixes

1) Database / migrations

2) Kotlin backend (queue-first import + matching)

3) Python extractor (HTML + OCR fallback + metadata parsing)

4) CLI E2E and regression capability

5) New tests

Recent worker/parser verification:

Validation Runs Performed

Blogspot HTML scenario (direct CLI)

Ran multiple extraction-e2e --scenario blogspot-html tasks for specific URLs; each reached INGESTED and created evidence.

Examples:

Focused 3-row convergence run (latest)

Command:

cargo run --manifest-path /Users/seshadri/project/sangeetha-grantha/tools/sangita-cli/Cargo.toml -- \
  test extraction-e2e \
  --scenario dikshitar-a-series \
  --csv-path /tmp/dikshitar-a-next-batch.csv \
  --english-pdf-path /Users/seshadri/Downloads/mdeng-A-series.pdf \
  --sanskrit-pdf-path /Users/seshadri/Downloads/mdskt-A-series.pdf \
  --skip-migrations \
  --skip-extraction-start \
  --timeout-seconds 300

Run summary:

PDF baseline task ids from this run:

HTML task ids from this run:

Detailed Non-Convergence Analysis (Current Batch)

Requested pairs

Row 1: abhyaambaam-bhaktim

Row 2: abhayaambikaayaah-anyam

Row 3: abhayaambikaayai

Additional Findings

Evidence insertion/page-index gaps still observed

For run tag 1770913249:

This matches the repeated symptom: result_count = 26, but evidence rows are lower (24/25).

Metadata improvements validated

OCR fallback behavior validated

Likely Root Causes (Prioritized)

1) PDF payload-to-evidence persistence mismatch (highest impact)

2) Existing duplicate/canonical split across previously created Krithis

3) Remaining title-variant normalization gaps

  1. Fix PDF evidence persistence gap first.
    • Trace from extraction_queue.result_payload -> ExtractionResultProcessor loop -> krithi_source_evidence inserts.
    • Add deterministic test asserting evidence_count == result_count for a controlled multi-page fixture.
  2. Re-run focused 3-row batch immediately after step 1.
    • Same command and same page mappings.
    • Validate whether row 3 can converge once English p13 + Sanskrit p16 evidence is actually present.
  3. Add targeted match reconciliation for known A-series duplicates.
    • Include raga-aware preference and script-variant affinity in tie-break when title scores are close.
    • Avoid creating/retaining split IDs for same composition across sources.
  4. Keep the 3-row CSV scenario as a permanent regression test (fast signal) plus Akhila 3-source as broader guardrail.

Useful Queries/Checks (for next session)

  1. Verify missing pages for a run: 
    WITH eng AS (
  SELECT page_range::int AS p
  FROM krithi_source_evidence
  WHERE source_url = '<eng-url-with-run>'
 AND page_range ~ '^[0-9]+$'
),
skt AS (
  SELECT page_range::int AS p
  FROM krithi_source_evidence
  WHERE source_url = '<skt-url-with-run>'
 AND page_range ~ '^[0-9]+$'
)
SELECT 'eng' AS src, gs AS missing_page
FROM generate_series(1,26) gs
WHERE gs NOT IN (SELECT p FROM eng)
UNION ALL
SELECT 'skt' AS src, gs AS missing_page
FROM generate_series(1,26) gs
WHERE gs NOT IN (SELECT p FROM skt)
ORDER BY src, missing_page;
  2. Map HTML source URL to current krithi id: 
    SELECT source_url, krithi_id, extracted_at, raw_extraction->>'title' AS extracted_title
FROM krithi_source_evidence
WHERE source_url LIKE 'http://guru-guha.blogspot.com/2007/08/dikshitar-kriti-abh%'
ORDER BY extracted_at DESC;
  3. Inspect PDF payload titles by index: 
    WITH task AS (
  SELECT result_payload
  FROM extraction_queue
  WHERE id = '<pdf-task-id>'::uuid
)
SELECT ordinality AS idx, elem->>'title' AS title
FROM task, jsonb_array_elements(task.result_payload) WITH ORDINALITY AS t(elem, ordinality)
ORDER BY ordinality;

Files Most Relevant for Next Debug Pass

Section index · Documentation home · Feature status