Sangeetha-Grantha

Metadata Value
Status Active
Version 1.1.0
Last Updated 2026-09-10
Author Sangeetha Grantha Team
Document Type Design reference

Bulk Import Master Analysis

[!NOTE] Design/reference material: this page may include proposals or earlier implementation assumptions. Use current ingestion guide for implemented behavior and current operating steps.

Document Consolidation Note

This document consolidates analysis from multiple sources:

Key insights from all sources have been integrated into this master document.

1. Executive Summary

This document provides a comprehensive analysis of building a bulk import capability for Krithis from multiple web sources. The analysis covers:

Note: The Dikshitar Kritis List source is particularly important for Trinity composer coverage (Tyagaraja, Dikshitar, Shyama Shastri).

Key Finding: The import capability requires a multi-stage pipeline with AI-powered extraction, validation, and human-in-the-loop moderation. Koog offers strong orchestration capabilities but may be over-engineered for initial phases. A hybrid approach combining existing Gemini integration with targeted workflow orchestration is recommended.

2. Data Source Analysis

2.1 Source Overview

Note on Trinity Composers: The Carnatic music tradition recognizes three composers as the “Trinity” (Trimurti):

Comprehensive coverage of these composers is critical for the Sangeetha Grantha project. The Dikshitar Kritis List source is particularly important for ensuring Trinity composer completeness.

2.2 Source Overview Table

Source URL Content Type Structure Estimated Volume
Karnatik.com https://karnatik.com/lyrics.shtml Structured HTML Well-organized, consistent format ~10,000+ compositions
Guru Guha Blog https://guru-guha.blogspot.com/ Blog posts Semi-structured, variable format ~500-1,000 posts
Dikshitar Kritis List https://guru-guha.blogspot.com/2009/04/dikshitar-kritis-alphabetical-list.html Blog archive List-based, alphabetical ~500+ compositions
Syama Krishna Vaibhavam https://syamakrishnavaibhavam.blogspot.com/2011/03/alphabetica-list-of-kritis.html Blog archive List-based, alphabetical ~1,000+ compositions
Thyagaraja Vaibhavam https://thyagaraja-vaibhavam.blogspot.com/2009/03/tyagaraja-kritis-alphabetical-list.html Blog archive List-based, alphabetical ~700+ compositions
TempleNet http://templenet.com/ Temple database Structured temple information ~1,000+ temples

2.3 Source-Specific Characteristics

2.2.1 Karnatik.com (https://karnatik.com/lyrics.shtml)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐ (Low-Medium)

2.2.2 Guru Guha Blog (https://guru-guha.blogspot.com/)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐⭐⭐ (High)

2.2.3 Syama Krishna Vaibhavam Blog (https://syamakrishnavaibhavam.blogspot.com/)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐⭐ (Medium-High)

2.2.4 Dikshitar Kritis List (https://guru-guha.blogspot.com/2009/04/dikshitar-kritis-alphabetical-list.html)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐⭐ (Medium-High)

Note: This source is critical for comprehensive Trinity composer coverage (Tyagaraja, Dikshitar, Shyama Shastri).

2.2.5 Thyagaraja Vaibhavam Blog (https://thyagaraja-vaibhavam.blogspot.com/)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐⭐ (Medium-High)

2.2.6 TempleNet (http://templenet.com/)

Strengths:

Challenges:

Data Quality:

Extraction Complexity: ⭐⭐⭐ (Medium)

2.4 Cross-Source Data Quality Challenges

2.3.1 Metadata Inconsistencies

Composer Name Variations:

Raga Name Variations:

Tala Variations:

Deity Name Variations:

2.3.2 Structural Variations

Section Identification:

Language/Script Variations:

2.3.3 Completeness Issues

Missing Metadata:

Incomplete Lyrics:

3. Import Capability Requirements

3.1 Core Data Entities to Import

Based on the Sangeetha Grantha domain model, the import pipeline must extract and map:

3.1.1 Krithi Core Data

3.1.2 Raga Association

3.1.3 Deity Association

3.1.4 Temple/Kshetra Association

3.1.5 Lyric Structure

3.1.6 Additional Metadata

3.2 Import Pipeline Stages

Stage 1: Discovery & URL Collection

Stage 2: Web Scraping

Stage 3: Content Extraction

Stage 4: Entity Resolution & Mapping

Stage 5: Data Cleansing

Stage 6: De-duplication

Stage 7: Validation

Stage 8: Staging & Review

Stage 9: Human Moderation

Stage 10: Canonicalization

3.3 Data Cleansing Requirements

3.3.1 Text Normalization

3.3.2 Name Normalization

3.3.3 Structural Cleanup

3.4 De-duplication Strategy

3.4.1 Duplicate Detection Criteria

Primary Keys for Matching:

  1. Composer + Title + Incipit: Strong match
  2. Composer + Title: Medium match (verify incipit)
  3. Title + Incipit: Weak match (may be different composers)
  4. Incipit Only: Very weak (many compositions share opening lines)

Fuzzy Matching:

3.4.2 Cross-Source Duplicate Handling

Scenario: Same Krithi from multiple sources

3.4.3 Duplicate Resolution Workflow

  1. Automatic Detection: Flag potential duplicates during import
  2. Confidence Scoring: Assign confidence to duplicate matches
  3. Review Queue: High-confidence duplicates auto-merged, others flagged
  4. Manual Review: Human verification for edge cases

3.5 Moderation Requirements

3.5.1 Pre-Moderation Checks

Automatic Validation:

Confidence Scoring:

3.5.2 Human Moderation Workflow

Review Interface Requirements:

Moderation Actions:

3.5.3 Quality Gates

Before Publishing:

4. Orchestration Options Analysis

4.1 Option A: Koog-Based Orchestration

4.1.1 Koog Capabilities for Import Pipeline

Strengths:

Workflow Example:

Discovery Node → Scraping Node → Extraction Node → 
Entity Resolution Node → Cleansing Node → 
Deduplication Node → Validation Node → Staging Node

4.1.2 Koog Integration Points

Existing Infrastructure:

New Components Needed:

4.1.3 Pros & Cons

Pros:

Cons:

Effort: High (2-3 weeks for initial implementation)

4.2 Option B: Custom Workflow Engine (Kotlin Coroutines)

4.2.1 Approach

Build a lightweight workflow engine using Kotlin Coroutines and structured concurrency:

class ImportPipeline {
    suspend fun processImport(source: ImportSource, urls: List<String>) {
        urls.map { url ->
            async {
                val html = scrape(url)
                val extracted = extractMetadata(html)
                val mapped = resolveEntities(extracted)
                val cleaned = cleanse(mapped)
                val validated = validate(cleaned)
                stageForReview(validated)
            }
        }.awaitAll()
    }
}

4.2.2 Pros & Cons

Pros:

Cons:

Effort: Medium (1-2 weeks)

4.3.1 Strategy

Phase 1: Custom Workflow (Immediate)

Phase 2: Evaluate Koog (After Phase 1)

Phase 3: Optimize Based on Learnings

4.3.2 Implementation Plan

Immediate (Weeks 1-4):

  1. Enhance WebScrapingService for multi-source support
  2. Build EntityResolutionService for composer/raga/deity/temple mapping
  3. Create ImportPipelineService using coroutines
  4. Add de-duplication logic
  5. Enhance import review UI

Evaluation (Weeks 5-6):

  1. Build Koog POC for extraction stage
  2. Compare with custom implementation
  3. Document findings

Decision Point:

4.4 Option D: Third-Party ETL Tools

4.4.1 Options Considered

Verdict: Not recommended for this use case. Kotlin-native solutions preferred.

5. Technical Architecture Recommendations

┌─────────────────────────────────────────────────────────┐
│                   Import API Endpoints                   │
│  POST /v1/admin/imports/sources/{id}/scrape            │
│  POST /v1/admin/imports/batch                           │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│              ImportPipelineService                      │
│  (Kotlin Coroutines-based workflow orchestration)      │
└──────┬──────────┬──────────┬──────────┬───────────────┘
       │          │          │          │
       ▼          ▼          ▼          ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Scraping │ │Extraction│ │  Entity  │ │Validation│
│ Service  │ │ Service  │ │Resolver  │ │ Service  │
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
     │           │            │            │
     ▼           ▼            ▼            ▼
┌─────────────────────────────────────────────────────┐
│         Existing Services (Enhanced)                 │
│  - WebScrapingService (Gemini-powered)              │
│  - TransliterationService                           │
│  - Entity Repositories (Composer, Raga, etc.)       │
└─────────────────────────────────────────────────────┘

5.2 Service Layer Design

5.2.1 ImportPipelineService

Responsibilities:

Key Methods: suspend fun importFromSource( sourceId: UUID, urls: List, options: ImportOptions ): ImportBatchResult

suspend fun processImportBatch(
    batchId: UUID
): ImportBatchStatus

5.2.2 EntityResolutionService (New)

Responsibilities:

**Key Methods:**
suspend fun resolveComposer(name: String): EntityMatch<Composer>
suspend fun resolveRaga(name: String): EntityMatch<Raga>
suspend fun resolveDeity(name: String): EntityMatch<Deity>
suspend fun resolveTemple(name: String, context: String?): EntityMatch<Temple>

5.2.3 DeduplicationService (New)

Responsibilities:

Key Methods: suspend fun findDuplicates( imported: ImportedKrithiDto ): List

suspend fun mergeDuplicates(
    primaryId: UUID,
    duplicateIds: List<UUID>
): MergeResult

5.2.4 ValidationService (Enhanced)

Responsibilities:

5.3 Database Schema Enhancements

5.3.1 Import Batch Tracking

**New Table: `import_batches`**
CREATE TABLE import_batches (
  id UUID PRIMARY KEY,
  import_source_id UUID REFERENCES import_sources(id),
  status TEXT, -- pending, processing, completed, failed
  total_urls INT,
  processed_urls INT,
  successful_imports INT,
  failed_imports INT,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  error_summary JSONB
);

5.3.2 Enhanced Imported Krithis

Add Fields to imported_krithis:

5.4 TempleNet Integration

5.4.1 Integration Strategy

Option 1: Pre-fetch and Cache

Option 2: On-Demand Lookup

Option 3: Hybrid

5.4.2 Temple Matching Logic

suspend fun matchTemple(
    name: String,
    deity: String?,
    location: String?
): List<TempleMatch> {
    // 1. Exact name match
    // 2. Fuzzy name match
    // 3. Deity + location match
    // 4. Multilingual name match
    // Return ranked matches with confidence
}

6. Implementation Roadmap

Phase 1: Foundation (Weeks 1-4)

Goals:

Deliverables:

  1. Enhanced WebScrapingService with source-specific handlers
  2. EntityResolutionService with fuzzy matching
  3. ImportPipelineService (coroutine-based)
  4. Basic de-duplication logic
  5. Import batch tracking
  6. Enhanced import review UI with confidence scores

Success Criteria:

Phase 2: Data Quality (Weeks 5-8)

Goals:

Deliverables:

  1. DeduplicationService with cross-source detection
  2. DataCleansingService for normalization
  3. ValidationService with musicological checks
  4. Quality scoring system
  5. Batch operations in review UI

Success Criteria:

Phase 3: TempleNet Integration & Trinity Sources (Weeks 9-10)

Goals:

Deliverables:

  1. TempleNet scraper/cache
  2. Temple matching service
  3. Integration with import pipeline
  4. Temple entity management UI enhancements
  5. Dikshitar List handler (similar to Thyagaraja handler)

Success Criteria:

Phase 4: Advanced Features (Weeks 11-12)

Goals:

Deliverables:

  1. Koog POC for extraction stage
  2. Comparison analysis
  3. Enhanced error handling
  4. Performance optimizations
  5. Monitoring and alerting

Success Criteria:

7. Risk Assessment & Mitigation

7.1 Technical Risks

Risk Impact Probability Mitigation
Source website changes High Medium Version source handlers, cache HTML, graceful degradation
Rate limiting Medium High Implement backoff, respect robots.txt, batch processing
Entity resolution accuracy High Medium Human review workflow, confidence thresholds, manual override
Duplicate detection false positives Medium Medium Conservative matching, human review, merge workflow
TempleNet availability Low Low Cache data, fallback matching, manual curation
Performance at scale Medium Low Batch processing, async operations, database optimization

7.2 Data Quality Risks

Risk Impact Probability Mitigation
Incorrect metadata extraction High Medium Multi-pass validation, human review, source comparison
Missing sections Medium Medium Section detection validation, manual correction workflow
Language/script misidentification Medium Low Language detection validation, manual override
Composer misattribution High Low Cross-reference validation, expert review for edge cases

7.3 Operational Risks

Risk Impact Probability Mitigation
Import pipeline failures Medium Medium Comprehensive error handling, retry logic, manual recovery
Database performance Medium Low Batch processing, indexing, connection pooling
Human review bottleneck Medium Medium Batch operations, confidence-based prioritization, automation

8. Success Metrics

8.1 Import Volume Metrics

8.2 Data Quality Metrics

8.3 Operational Metrics

9. Open Questions & Decisions Needed

9.1 Technical Decisions

  1. Koog Adoption: Proceed with POC or defer?
  2. TempleNet Integration: Pre-fetch vs. on-demand?
  3. De-duplication Strategy: Automatic merge vs. always review?
  4. Retry Policy: How many retries? Exponential backoff parameters?

9.2 Data Quality Decisions

  1. Confidence Thresholds: What scores trigger automatic approval?
  2. Source Priority: How to handle conflicting information?
  3. Missing Data: Allow incomplete imports or require all fields?
  4. Validation Strictness: How strict should musicological validation be?

9.3 Operational Decisions

  1. Review Workflow: Single reviewer or multi-stage review?
  2. Batch Size: Optimal batch size for processing?
  3. Scheduling: Scheduled imports or on-demand only?
  4. Attribution: How to handle source attribution in UI?

10. Recommendations

10.1 Immediate Actions (Next 2 Weeks)

  1. Start with Custom Workflow: Build coroutine-based pipeline for immediate needs
  2. Focus on Karnatik.com First: Highest quality, most structured source
  3. Build Entity Resolution Service: Critical for data quality
  4. Enhance Import Review UI: Human-in-the-loop is essential

10.2 Medium-Term (Next 2-3 Months)

  1. Evaluate Koog: Build POC after custom workflow is stable
  2. Add Trinity Sources: Dikshitar List (important for Trinity composer coverage)
  3. Add Remaining Sources: Expand to other blog sources
  4. TempleNet Integration: Pre-fetch and cache temple data
  5. Advanced De-duplication: Cross-source duplicate detection

10.3 Long-Term (6+ Months)

  1. Koog Decision: Adopt if value is clear, otherwise enhance custom solution
  2. Automation: Increase confidence thresholds for auto-approval
  3. Continuous Import: Scheduled imports for updated sources
  4. Quality Improvements: Machine learning for better entity resolution

11. Conclusion

The bulk import capability is a critical feature for scaling Sangeetha Grantha’s content. The recommended hybrid approach balances:

Key Success Factors:

  1. Strong entity resolution (composer, raga, deity, temple)
  2. Effective de-duplication across sources
  3. Intuitive human review workflow
  4. Robust error handling and recovery
  5. Performance at scale

The import pipeline will be a significant engineering effort but is essential for building a comprehensive Carnatic music compendium.

12. References

Documentation home · Feature status