Sangeetha-Grantha

Track: Bulk Import - Backend Orchestration

ID: TRACK-001 Status: Completed Owner: Sangita Grantha Architect Created: 2026-01-20 Updated: 2026-01-21

Goal

Build the backend orchestration engine for bulk importing Krithis. This includes the database schema, Ktor services, coroutine workers, and management APIs. (UI/UX is handled in TRACK-003 and TRACK-004).

Context

• CSV Files: Located in database/for_import/
• Architecture: Ktor backend with coroutine-based background workers, Postgres-backed orchestration.
• Reference: /application_documentation/01-requirements/features/bulk-import/bulk-import-orchestration-ops-plan-goose.md

Implementation Plan (Phased)

Phase A: Foundation (Current)

• Update TRACK-001 with orchestration plan
• Create database migration for orchestration tables
• Create DAL layer (tables, repositories)
• Implement BulkImportOrchestrationService (batch/job/task lifecycle + admin controls)
• Implement BulkImportWorkerService (background coroutine workers + polling)
• Create API routes (/v1/admin/bulk-import/...) for batch management
• Implement CSV manifest ingestion worker
• Integrate workers into App.kt lifecycle
• Hardening: idempotency keys, rate limiting, stuck-task watchdog, richer error taxonomy

Phase B: Scrape & Enrich Workers

• Implement scrape worker coroutines with rate limiting
• Integrate with WebScrapingService and ImportService
• Implement enrichment logic (Scraping is the enrichment)

Phase C: Entity Resolution Workers

• Implement entity resolution worker (Levenshtein based)
• Add confidence scoring and blocked state logic (Basic confidence implemented)
• Implement APIs for retrieving resolution candidates (Saved to resolution_data)

Phase D: Review Workflow APIs

• Implement APIs for fetching tasks ready for review (GET /imports?status=PENDING)
• Implement APIs for approving/rejecting tasks (POST /imports/{id}/review)
• Implement APIs for manual entity resolution (overrides) (Supported via overrides in ImportReviewRequest)

Phase E: Hardening & Performance

• Stuck task detector
• SLO monitoring and alerts
• Performance tuning
• Load test on ~1,240 entries
• Duplicate Entity Fix: Implemented findOrCreate for Composers, Ragas, and Talas to prevent race conditions during review.
• Batch Cleanup: Added DELETE /batches/{id} endpoint and UI control for managing stuck/test batches.
• Error Handling: Improved error reporting in reviewImport to return meaningful messages instead of 500s.

Progress Log

(Previous logs preserved)

2026-01-22: Hardening & Cleanup

• ✅ Investigated talas_name_normalized_uq error in logs.
• ✅ Implemented findOrCreate in TalaRepository, RagaRepository, ComposerRepository to handle race conditions safely.
• ✅ Wrapped ImportService.reviewImport in try-catch to provide user-friendly error messages.
• ✅ Added DELETE /v1/admin/bulk-import/batches/{id} endpoint and corresponding “Delete” button in Admin UI.

2026-01-21: Entity Resolution & Workers

• ✅ Implemented EntityResolutionService with fuzzy matching (Levenshtein) for Composers, Ragas, Talas.
• ✅ Added resolution_data JSONB column to imported_krithis.
• ✅ Updated BulkImportWorkerService to include EntityResolution worker loop.
• ✅ Implemented orchestration transition: Scrape -> Entity Resolution.
• ✅ Wired up services in App.kt.

2026-01-21: Track Restructuring

• 🔄 Moved UI/UX scope to TRACK-003 and TRACK-004.
• 🔄 Renamed track to “Bulk Import - Backend Orchestration”.

2026-01-21: Hardening Complete

• ✅ Added idempotency_key column + unique index via 11__bulk-import-hardening.sql
• ✅ Worker improvements: dedupe manifest rows, reuse scrape jobs per batch, task idempotency on insert
• ✅ Scrape loop: rate limiting (per-domain/global), retryable errors until max attempts, terminal failures increment counters
• ✅ Watchdog loop: marks stuck RUNNING tasks as RETRYABLE with audit events

---

Phase E (Hardening) descoped to post-launch. See conductor/handoff-simplify-and-ship.md