Sangeetha-Grantha

Metadata Value
Status Archived
Version 1.0.0
Last Updated 2026-09-10
Author Sangeetha Grantha Team
Document Type Archive

Krithi Data Import and Orchestration - Comprehensive Analysis

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current ingestion guide.

Document Version: 1.0 Date: January 19, 2026 Author: Analysis for Sangeetha Grantha Application

Executive Summary

This document provides a comprehensive analysis of building an import capability for Carnatic music Krithis from multiple web sources, with associated metadata including Raga, Deity, and Temple/Kshetra information. The analysis covers data source characteristics, technical challenges, architecture options, and recommendations for orchestration approaches.

1. Data Source Analysis

1.1 Primary Krithi Sources

Source 1: Karnatik.com

Source 2: Guru-Guha Blogspot

Source 3: Syama Krishna Vaibhavam

Source 4: Thyagaraja Vaibhavam

Source 5: TempleNet.com

1.2 Data Source Comparison Matrix

Source Structure Metadata Richness Extract Difficulty Data Quality Update Frequency
Karnatik.com Semi-structured High Medium High Low
Guru-Guha Unstructured Medium High Medium Low
Syama Krishna List-based Low-Medium Medium Medium Very Low
Thyagaraja List-based Low-Medium Medium Medium-High Very Low
TempleNet Semi-structured High Medium High Low

2. Data Model Requirements

2.1 Core Entities

Krithi
├── id (UUID)
├── name (primary, with transliteration variants)
├── name_variants (array of transliterations: IAST, Tamil, Telugu, Devanagari)
├── composer_id (FK)
├── raga_id (FK)
├── tala_id (FK)
├── language_id (FK)
├── lyrics (full text with lines/sections)
├── meaning (optional)
├── pallavi, anupallavi, charanam (structured lyrics)
├── created_at
├── updated_at
├── source_urls (array)
├── verification_status (enum: unverified, verified, expert_verified)

Raga
├── id
├── name (primary)
├── name_variants
├── melakarta_number (if applicable)
├── arohanam
├── avarohanam
├── janya_of (FK to parent raga)

Composer
├── id
├── name
├── name_variants
├── period (time period)
├── language_primary
├── bio

Deity
├── id
├── name
├── name_variants (Vishnu/Perumal/Narayana)
├── pantheon
├── iconography

Temple/Kshetra
├── id
├── name
├── name_variants
├── location
├── primary_deity_id (FK)
├── secondary_deities (array of FK)
├── kshetra_type (Divya Desam, Jyotirlinga, Shakti Peetha, etc.)
├── region

Krithi_Deity (many-to-many)
├── krithi_id
├── deity_id
├── is_primary (boolean)

Krithi_Temple (many-to-many)
├── krithi_id
├── temple_id
├── reference_type (composed_at, about, dedicated_to)

2.2 Data Relationships

3. Data Challenges and Cleansing Requirements

3.1 Data Quality Issues

Name Variations and Transliteration

Challenge: Same krithi appears with different spellings

Solution Approach:

Raga Name Variations

Challenge: Different schools use different names

Solution Approach:

Composer Attribution

Challenge: Multiple composers with similar names

Solution Approach:

Deity Identification

Challenge: Multiple names for same deity

Solution Approach:

Temple Association

Challenge: Linking krithis to temples

Solution Approach:

3.2 De-duplication Strategy

Level 1: Exact Match

Level 2: Fuzzy Matching

Level 3: Semantic Matching

Level 4: Manual Review

Proposed De-duplication Pipeline

Raw Import →
  Exact Match (automated merge) →
  Fuzzy Match (high confidence: auto-merge, medium: flag) →
  Semantic Match (all flagged) →
  Manual Review Queue →
  Final Database

3.3 Data Moderation Requirements

Verification Levels

  1. Unverified: Auto-imported, not reviewed
  2. System-Verified: Passed all automated checks
  3. Human-Verified: Reviewed by contributor
  4. Expert-Verified: Verified by recognized authority

Moderation Workflow

4. Technical Architecture Options

4.1 Option A: Custom Python-based Pipeline

Architecture

Web Scrapers (BeautifulSoup/Scrapy) →
  Raw Data Storage (JSON/CSV) →
  ETL Pipeline (Python/Pandas) →
  Cleansing Engine (Custom Scripts) →
  De-duplication Engine (FuzzyWuzzy/Dedupe) →
  Staging Database (PostgreSQL) →
  Manual Review Interface (Web App) →
  Production Database

Pros

Cons

Technology Stack

Estimated Effort

Cost

4.2 Option B: Koog.ai-based Orchestration

Architecture (Hypothetical based on typical orchestration platforms)

Koog Connectors/Scrapers →
  Koog Data Pipeline →
  Koog Transformation Rules →
  Koog Quality Checks →
  Koog De-duplication →
  Review Interface →
  Target Database

Expected Koog.ai Capabilities (typical for such platforms)

Pros

Cons

Considerations for Koog.ai

Need to Verify:

Estimated Effort

Cost

4.3 Option C: Hybrid Approach

Architecture

Custom Scrapers (Python) →
  Data Lake (S3/MinIO) →
  Koog.ai or Airflow for Orchestration →
  Custom Cleansing Scripts →
  Koog De-duplication or Custom →
  Staging DB →
  Custom Review Interface →
  Production DB

Pros

Cons

4.4 Option D: Cloud-Native ETL (AWS Glue/Azure Data Factory/Google Dataflow)

Architecture

Lambda/Cloud Functions (Scrapers) →
  S3/Blob Storage (Raw Data) →
  AWS Glue/Azure Data Factory (ETL) →
  Custom Processing (Containers) →
  RDS/Cloud SQL (Staging) →
  Application Review Interface →
  Production Database

Pros

Cons

Estimated Cost

5. Orchestration Requirements

5.1 Workflow Stages

Stage 1: Data Extraction

Stage 2: Data Transformation

Stage 3: Data Enrichment

Stage 4: De-duplication

Stage 5: Quality Checks

Stage 6: Staging and Review

Stage 7: Production Deployment

5.2 Orchestration Features Needed

Essential Features

Advanced Features

6. Detailed Comparison: Custom vs Koog.ai

6.1 Development Complexity

Aspect Custom Python Koog.ai Hybrid
Web Scraping High (build scrapers) Medium (if supported) Medium (custom)
Orchestration High (Airflow setup) Low (built-in) Low
Data Transformation High (custom code) Medium (visual rules) Medium
De-duplication High (algorithm dev) Low (if built-in) Medium
NLP Integration High (custom) Unknown High
Review Interface High (build from scratch) Medium (if included) High
Maintenance High Low-Medium Medium

6.2 Feature Comparison

Feature Custom Koog.ai Hybrid
Web Scraping ✅ Full control ❓ Unknown ✅ Full control
Indian Language NLP ✅ IndicNLP ❓ Unknown ✅ Custom
Complex De-dup Logic ✅ Custom algorithms ⚠️ May be limited ✅ Custom
Visual Workflow ❌ Code-based ✅ Likely ⚠️ Partial
Scalability ⚠️ Manual optimization ✅ Built-in ✅ Good
Cost Low infra, high dev High subscription Medium
Flexibility ✅ Unlimited ⚠️ Platform limits ✅ High
Time to Market Slow (3-4 months) Fast (1-2 months) Medium (2-3 months)

6.3 Cost Analysis (Annual Estimates)

Custom Python Solution

Koog.ai Solution (Estimated)

Hybrid Solution

Approach: Custom Python Pipeline (Minimal)

Rationale:

Scope:

  1. Build scraper for ONE source (Karnatik.com recommended - highest quality)
  2. Create basic ETL pipeline
  3. Implement simple de-duplication
  4. Build minimal review interface
  5. Manual temple/deity association
  6. Import 50-100 krithis end-to-end

Technology:

Timeline: 6-8 weeks

Cost: ~$8,000 - $12,000

Success Criteria:

7.2 Phase 2: Production Pipeline

Approach: Hybrid (Custom Code + Airflow Orchestration)

Rationale:

Scope:

  1. Extend to all 4 krithi sources
  2. Integrate TempleNet data
  3. Full de-duplication pipeline
  4. Enhanced review interface
  5. Airflow workflows for orchestration
  6. Monitoring and alerting
  7. API for application integration

Technology:

Timeline: 3-4 months

Cost: ~$30,000 - $45,000

7.3 Koog.ai Evaluation Criteria

Consider Koog.ai if:

Evaluation Steps:

  1. Request Koog.ai demo focused on web scraping use case
  2. Ask about Indian language support
  3. Test with sample data from one source
  4. Compare development time vs custom approach
  5. Get detailed pricing
  6. Check exit strategy (data export)

Decision Point: After PoC completion, evaluate if Koog.ai can reduce Phase 2 timeline by >40% at reasonable cost

8. Implementation Roadmap

Phase 1: Proof of Concept (Weeks 1-8)

Week 1-2: Setup and First Scraper

Week 3-4: ETL Pipeline

Week 5-6: De-duplication

Week 7-8: Review Interface and Testing

Phase 2: Production System (Months 3-6)

Month 3: Multi-Source Integration

Month 4: Advanced Processing

Month 5: Review System

Month 6: Production Deployment

Phase 3: Ongoing Operations (Month 7+)

Monthly:

Quarterly:

Annually:

9. Risk Analysis and Mitigation

9.1 Technical Risks

Risk Impact Probability Mitigation
Website structure changes break scrapers High Medium Version scrapers, automated tests, alerts
Data quality worse than expected High Medium PoC validates first, manual review queue
De-duplication too complex Medium Medium Phased approach, expert review layer
Indian language NLP challenges Medium High Use IndicNLP library, manual fallback
Scale issues with thousands of krithis Medium Low Database optimization, caching, pagination
Temple association ambiguity Medium High Accept as limitation, expert curation

9.2 Operational Risks

Risk Impact Probability Mitigation
Insufficient manual review capacity Medium Medium Prioritize high-confidence items, community
Source websites go offline Medium Low Archive raw data, diversify sources
Copyright/legal issues with lyrics High Low Ensure only metadata, first lines, links
Platform lock-in if using Koog Medium Medium Avoid Koog initially, keep data portable
Maintenance burden too high Medium Medium Good documentation, simple architecture

9.3 Business Risks

Risk Impact Probability Mitigation
Budget overruns Medium Medium Phased approach, PoC first
Timeline delays Medium Medium Conservative estimates, agile approach
User adoption of review interface Low Medium Simple UX, expert engagement
Data completeness concerns Medium High Set expectations, continuous improvement

10. Alternative Approaches

10.1 Community Crowdsourcing

Model: Wikipedia-style contribution platform

Approach:

Pros:

Cons:

Recommendation: Combine with automated import - use automation for bulk, community for enhancement

10.2 Commercial Data Purchase

Approach: License data from commercial Carnatic music platforms

Pros:

Cons:

Recommendation: Explore as supplement, not primary strategy

10.3 Academic Partnership

Approach: Partner with music universities/research institutions

Pros:

Cons:

Recommendation: Good for validation and expertise, not primary data source

11. Success Metrics

11.1 Import Pipeline Metrics

11.2 Data Quality Metrics

11.3 Operational Metrics

12. Technology Stack Recommendation

12.1 Core Technologies

Programming Language: Python 3.11+

Web Scraping:

Data Processing:

NLP and Matching:

Orchestration:

Database:

API Layer:

Frontend (Review Interface):

Infrastructure:

Monitoring:

13. Conclusion and Recommendations

13.1 Primary Recommendation

Implement a phased, custom Python-based approach with Airflow orchestration:

  1. Start with Proof of Concept (8 weeks, ~$10K)
    • Validate technical approach
    • Understand data quality
    • Test de-duplication logic
    • Single source (Karnatik.com)
  2. Evaluate Koog.ai in parallel
    • Request demo and trial
    • Test with PoC data
    • Compare cost/benefit
    • Make informed decision
  3. Build Production System (4 months, ~$35K)
    • If Koog.ai proves valuable: Hybrid approach
    • If not: Full custom with Airflow
    • All sources integrated
    • Complete review workflow
  4. Launch and Iterate (Ongoing)
    • Community feedback
    • Continuous data quality improvement
    • Add sources as discovered

13.2 Koog.ai Specific Recommendation

Do not commit to Koog.ai without thorough evaluation:

Required Validation:

Use Koog.ai if:

Avoid Koog.ai if:

13.3 Critical Success Factors

  1. Start Small: PoC before major investment
  2. Data Quality First: Perfect 100 records better than poor 10,000
  3. Expert Involvement: Engage Carnatic music experts for validation
  4. Flexible Architecture: Avoid lock-in, keep options open
  5. Incremental Value: Release data in phases, don’t wait for perfection
  6. Community Engagement: Consider crowdsourcing for enhancement
  7. Documentation: Document all transformations and business rules

13.4 Next Steps

Immediate (Week 1-2):

  1. Approve recommended approach
  2. Set up development environment
  3. Request Koog.ai demo (parallel track)
  4. Identify 1-2 Carnatic music experts for consultation
  5. Define initial database schema

Short-term (Month 1-2):

  1. Build first scraper
  2. Create basic ETL pipeline
  3. Test de-duplication approaches
  4. Evaluate Koog.ai trial results
  5. Make build-vs-buy decision

Medium-term (Month 3-6):

  1. Expand to all sources
  2. Production pipeline development
  3. Review interface
  4. Testing and refinement
  5. Initial data release

14. Appendices

Appendix A: Sample Data Schema (SQL)

– Core Krithi Table CREATE TABLE krithis ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name VARCHAR(255) NOT NULL, name_iast VARCHAR(255), name_tamil VARCHAR(255), name_telugu VARCHAR(255), composer_id UUID REFERENCES composers(id), raga_id UUID REFERENCES ragas(id), tala_id UUID REFERENCES talas(id), language VARCHAR(50), lyrics_pallavi TEXT, lyrics_anupallavi TEXT, lyrics_charanam TEXT, lyrics_full TEXT, meaning TEXT, source_urls JSONB, verification_status VARCHAR(50) DEFAULT ‘unverified’, created_at TIMESTAMP DEFAULT NOW(), updated_at TIMESTAMP DEFAULT NOW(), created_by VARCHAR(100), UNIQUE(name, composer_id) );

– Raga Table CREATE TABLE ragas ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name VARCHAR(100) NOT NULL UNIQUE, name_variants JSONB, melakarta_number INT, arohanam VARCHAR(255), avarohanam VARCHAR(255), parent_raga_id UUID REFERENCES ragas(id) );

– Composer Table CREATE TABLE composers ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name VARCHAR(100) NOT NULL, name_variants JSONB, period_start INT, period_end INT, language_primary VARCHAR(50), biography TEXT );

– Deity Table CREATE TABLE deities ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name VARCHAR(100) NOT NULL, name_variants JSONB, pantheon VARCHAR(50), description TEXT );

– Temple Table CREATE TABLE temples ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name VARCHAR(255) NOT NULL, name_variants JSONB, location VARCHAR(255), city VARCHAR(100), state VARCHAR(100), country VARCHAR(50), primary_deity_id UUID REFERENCES deities(id), kshetra_type VARCHAR(100), description TEXT );

– Krithi-Deity Many-to-Many CREATE TABLE krithi_deities ( krithi_id UUID REFERENCES krithis(id) ON DELETE CASCADE, deity_id UUID REFERENCES deities(id) ON DELETE CASCADE, is_primary BOOLEAN DEFAULT FALSE, PRIMARY KEY (krithi_id, deity_id) );

– Krithi-Temple Many-to-Many CREATE TABLE krithi_temples ( krithi_id UUID REFERENCES krithis(id) ON DELETE CASCADE, temple_id UUID REFERENCES temples(id) ON DELETE CASCADE, reference_type VARCHAR(50), – ‘composed_at’, ‘about’, ‘dedicated_to’ PRIMARY KEY (krithi_id, temple_id) );

– Temple-Deity Many-to-Many CREATE TABLE temple_deities ( temple_id UUID REFERENCES temples(id) ON DELETE CASCADE, deity_id UUID REFERENCES deities(id) ON DELETE CASCADE, is_primary BOOLEAN DEFAULT FALSE, PRIMARY KEY (temple_id, deity_id) );

– Indices for Performance CREATE INDEX idx_krithis_name_trgm ON krithis USING gin (name gin_trgm_ops); CREATE INDEX idx_krithis_raga ON krithis(raga_id); CREATE INDEX idx_krithis_composer ON krithis(composer_id); CREATE INDEX idx_krithis_verification ON krithis(verification_status); CREATE INDEX idx_krithi_deities_deity ON krithi_deities(deity_id); CREATE INDEX idx_krithi_temples_temple ON krithi_temples(temple_id);

-- Full-text search
CREATE INDEX idx_krithis_lyrics_fts ON krithis USING gin(to_tsvector('english', lyrics_full));

Appendix B: Sample Airflow DAG Structure

from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime, timedelta

default_args = { ‘owner’: ‘sangeetha-grantha’, ‘depends_on_past’: False, ‘start_date’: datetime(2026, 1, 1), ‘email_on_failure’: True, ‘email_on_retry’: False, ‘retries’: 2, ‘retry_delay’: timedelta(minutes=5), }

with DAG( ‘krithi_import_pipeline’, default_args=default_args, description=’Import krithis from multiple sources’, schedule_interval=’@weekly’, catchup=False ) as dag:

# Stage 1: Extraction
scrape_karnatik = PythonOperator(
    task_id='scrape_karnatik',
    python_callable=scrape_karnatik_com,
)

scrape_guruguha = PythonOperator(
    task_id='scrape_guruguha',
    python_callable=scrape_guruguha_blog,
)

scrape_syamakrishna = PythonOperator(
    task_id='scrape_syamakrishna',
    python_callable=scrape_syamakrishna_blog,
)

scrape_thyagaraja = PythonOperator(
    task_id='scrape_thyagaraja',
    python_callable=scrape_thyagaraja_blog,
)

# Stage 2: Transformation
transform_data = PythonOperator(
    task_id='transform_data',
    python_callable=transform_raw_data,
)

# Stage 3: Enrichment
enrich_metadata = PythonOperator(
    task_id='enrich_metadata',
    python_callable=enrich_with_metadata,
)

# Stage 4: De-duplication
deduplicate = PythonOperator(
    task_id='deduplicate',
    python_callable=run_deduplication,
)

# Stage 5: Quality Checks
quality_checks = PythonOperator(
    task_id='quality_checks',
    python_callable=run_quality_checks,
)

# Stage 6: Load to Staging
load_staging = PythonOperator(
    task_id='load_staging',
    python_callable=load_to_staging_db,
)

# Dependencies
```text
[scrape_karnatik, scrape_guruguha, scrape_syamakrishna, scrape_thyagaraja] >> transform_data
transform_data >> enrich_metadata >> deduplicate >> quality_checks >> load_staging ```

Appendix C: Deity Name Normalization Examples

Sample deity normalization dictionary

DEITY_SYNONYMS = { ‘vishnu’: [‘perumal’, ‘narayana’, ‘hari’, ‘govinda’, ‘madhava’, ‘venkateshwara’], ‘rama’: [‘raghuvira’, ‘raghunatha’, ‘dasarathi’, ‘kodanda rama’, ‘sitapati’], ‘krishna’: [‘gopala’, ‘madhava’, ‘yadava’, ‘devaki nandana’, ‘vaasudeva’], ‘shiva’: [‘shankara’, ‘ishwara’, ‘mahadeva’, ‘hara’, ‘chandrashekara’], ‘murugan’: [‘kartikeya’, ‘skanda’, ‘subramanya’, ‘guha’, ‘shanmukha’], ‘ganesha’: [‘ganapati’, ‘vinayaka’, ‘pillayar’, ‘vighneshwara’], }

def normalize_deity_name(name): “"”Normalize deity name to canonical form””” name_lower = name.lower().strip()

for canonical, synonyms in DEITY_SYNONYMS.items():
    if name_lower == canonical or name_lower in synonyms:
        return canonical

```text
return name_lower  # Return as-is if not found ```

Appendix D: Fuzzy Matching Configuration

Sample de-duplication configuration

DEDUP_CONFIG = { ‘exact_match_threshold’: 1.0, # 100% match ‘high_confidence_threshold’: 0.92, # Auto-merge ‘medium_confidence_threshold’: 0.85, # Flag for review ‘low_confidence_threshold’: 0.75, # Ignore

'field_weights': {
    'name': 0.5,
    'composer': 0.2,
    'raga': 0.15,
    'pallavi': 0.15,
},

```text
'phonetic_algorithms': ['metaphone', 'soundex'],
'use_trigram': True,
'trigram_threshold': 0.7, } ```

Appendix E: Glossary

Document End

This analysis is based on general knowledge of Carnatic music data sources, typical data engineering challenges, and orchestration platform capabilities. Specific website structures and Koog.ai features should be validated through direct examination and vendor consultation.

Documentation home · Feature status