Sangeetha-Grantha

Metadata Value
Status Active
Version 1.1.0
Last Updated 2026-09-10
Author Sangeetha Grantha Team
Document Type Design reference

Web Source Detailed Analysis for Krithi Import

[!NOTE] Design/reference material: this page may include proposals or earlier implementation assumptions. Use current ingestion guide for implemented behavior and current operating steps.

1. Overview

This document provides detailed technical analysis of each web source for Krithi import, including HTML structure, extraction strategies, and source-specific challenges.

2. Karnatik.com (https://karnatik.com/lyrics.shtml)

2.1 Site Structure

Organization:

URL Patterns:

2.2 HTML Structure Analysis

**Typical Composition Page Structure:**
<div class="composition">
  <h1>Composition Title</h1>
  <div class="metadata">
    <span>Raga: [Raga Name]</span>
    <span>Tala: [Tala Name]</span>
    <span>Composer: [Composer Name]</span>
  </div>
  <div class="lyrics">
    <h2>Pallavi</h2>
    <p>Lyric text...</p>
    <h2>Anupallavi</h2>
    <p>Lyric text...</p>
    <h2>Charanam</h2>
    <p>Lyric text...</p>
  </div>
</div>

Key Characteristics:

2.3 Extraction Strategy

Step 1: Discovery

Step 2: Scraping

Step 3: Extraction

Step 4: Normalization

2.4 Challenges & Solutions

Challenge Solution
Rate Limiting Implement exponential backoff, respect robots.txt
Pagination Follow “next” links, track visited URLs
Multiple Scripts Detect script, create separate lyric variants
Metadata Variations Use AI extraction for edge cases
Broken Links Skip and log, continue processing

2.5 Data Quality Assessment

Recommended Priority: ⭐⭐⭐⭐⭐ (Highest - Start here)

3. Guru Guha Blog (https://guru-guha.blogspot.com/)

3.1 Site Structure

Organization:

URL Patterns:

3.2 HTML Structure Analysis

**Typical Blog Post Structure:**
<article class="post">
  <header>
    <h1 class="post-title">Post Title</h1>
    <div class="post-meta">Date, Author</div>
  </header>
  <div class="post-body">
    <!-- Mixed content: explanations, lyrics, notation -->
    <p>Introduction text...</p>
    <div class="lyrics-section">
      <h3>Pallavi</h3>
      <p>Lyric text mixed with commentary...</p>
    </div>
    <!-- More mixed content -->
  </div>
</article>

Key Characteristics:

3.3 Extraction Strategy

Step 1: Discovery

Step 2: Scraping

Step 3: Extraction (AI-Powered)

Step 4: Normalization

3.4 Challenges & Solutions

Challenge Solution
Mixed Content AI-powered extraction to separate lyrics from commentary
Variable Structure Flexible parsing, AI fallback
Metadata Extraction AI extraction from post content and tags
Section Identification AI identifies implicit section boundaries
Blogspot Limitations Handle platform-specific quirks

3.5 Data Quality Assessment

Recommended Priority: ⭐⭐⭐ (Medium - After Karnatik)

4. Syama Krishna Vaibhavam Blog

4.1 Site Structure

Organization:

URL Patterns:

4.2 HTML Structure Analysis

**List Page Structure:**
<div class="post-body">
  <h2>A</h2>
  <ul>
    <li><a href="...">Composition Title 1</a></li>
    <li><a href="...">Composition Title 2</a></li>
    <!-- ... -->
  </ul>
  <h2>B</h2>
  <!-- ... -->
</div>

Individual Post Structure:

4.3 Extraction Strategy

Step 1: Discovery

Step 2: Scraping

Step 3: Extraction

Step 4: Cross-Reference

4.4 Challenges & Solutions

Challenge Solution
Link Rot Validate links, skip broken ones, log for manual review
Title-Only Entries Use for discovery, cross-reference with other sources
External Links Follow if same domain, otherwise use as reference only
Older Content Expect some data quality issues, prioritize newer sources
Alphabetical Organization Parse list structure, extract all links

4.5 Data Quality Assessment

Recommended Priority: ⭐⭐ (Low - Use for discovery, cross-reference)

5. Dikshitar Kritis List (Guru Guha Blog)

5.1 Site Structure

Organization:

URL Patterns:

5.2 HTML Structure Analysis

**Similar to other alphabetical list sources:**
<div class="post-body">
  <h2>A</h2>
  <ul>
    <li><a href="...">Composition Title 1</a></li>
    <li><a href="...">Composition Title 2</a></li>
    <!-- ... -->
  </ul>
  <h2>B</h2>
  <!-- ... -->
</div>

Key Characteristics:

5.3 Extraction Strategy

Step 1: Discovery

Step 2: Scraping

Step 3: Extraction

Step 4: Cross-Reference

5.4 Challenges & Solutions

Challenge Solution
Link Rot Validate links, skip broken ones, log for manual review
Title-Only Entries Use for discovery, cross-reference with other sources
External Links Follow if same domain (Guru Guha), otherwise use as reference only
Older Content Expect some data quality issues, prioritize newer sources
Alphabetical Organization Parse list structure, extract all links
Same Domain as Main Blog Reuse Guru Guha blog handler, differentiate by URL pattern

5.5 Data Quality Assessment

Recommended Priority: ⭐⭐⭐ (Medium - Important for Trinity composer coverage)

Note: This source is particularly valuable because:

  1. Dikshitar is one of the Trinity composers (Tyagaraja, Dikshitar, Shyama Shastri)
  2. Provides composer-specific discovery
  3. Complements other Trinity sources (Thyagaraja Vaibhavam)
  4. May have compositions not in Karnatik

6. Thyagaraja Vaibhavam Blog

6.1 Site Structure

Organization:

URL Patterns:

6.2 HTML Structure Analysis

Similar to Syama Krishna Vaibhavam:

6.3 Extraction Strategy

Similar to Syama Krishna Vaibhavam:

6.4 Challenges & Solutions

Same as Syama Krishna Vaibhavam, plus:

6.5 Data Quality Assessment

Recommended Priority: ⭐⭐ (Low - Use for Tyagaraja-specific discovery)

7. TempleNet (http://templenet.com/)

6.1 Site Structure

Organization:

URL Patterns:

6.2 HTML Structure Analysis

**Typical Temple Page Structure:**
<div class="temple-info">
  <h1>Temple Name</h1>
  <div class="location">
    <span>City: [City]</span>
    <span>State: [State]</span>
    <span>District: [District]</span>
  </div>
  <div class="deity">
    <h2>Primary Deity</h2>
    <p>Deity name and details...</p>
  </div>
  <div class="history">
    <h2>History</h2>
    <p>Historical information...</p>
  </div>
  <div class="names">
    <h2>Other Names</h2>
    <ul>
      <li>Alternative name 1</li>
      <li>Alternative name 2</li>
    </ul>
  </div>
</div>

Key Characteristics:

6.3 Extraction Strategy

Step 1: Discovery

Step 2: Scraping

Step 3: Extraction

Step 4: Normalization

6.4 Integration with Krithi Import

Matching Strategy:

  1. Extract temple names from Krithi lyrics/title
  2. Match against TempleNet database
  3. Use fuzzy matching for name variations
  4. Consider deity context for disambiguation
  5. Geographic hints if available

Example:

6.5 Challenges & Solutions

Challenge Solution
Name Variations Fuzzy matching, multilingual name support
Geographic Disambiguation Use location context from lyrics if available
Deity Associations Match by deity name, handle multiple deities
TempleNet Availability Pre-fetch and cache, fallback to manual matching
Volume Batch processing, incremental updates

6.6 Data Quality Assessment

Recommended Priority: ⭐⭐⭐⭐ (High - Essential for temple associations)

7. Cross-Source Comparison

7.1 Source Priority Matrix

Source Data Quality Volume Structure Priority Effort
Karnatik.com ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ 1st Low
TempleNet ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ 2nd Medium
Guru Guha ⭐⭐⭐ ⭐⭐⭐ ⭐⭐ 3rd High
Dikshitar List ⭐⭐ ⭐⭐ ⭐⭐ 4th Medium
Syama Krishna ⭐⭐ ⭐⭐⭐ ⭐⭐ 5th Medium
Thyagaraja ⭐⭐ ⭐⭐ ⭐⭐ 6th Medium
  1. Phase 1: Karnatik.com (foundation, highest quality)
  2. Phase 2: TempleNet (essential for temple associations)
  3. Phase 3: Guru Guha (supplementary, rich content)
  4. Phase 4: Dikshitar List (Trinity composer coverage - important for completeness)
  5. Phase 5: Syama Krishna & Thyagaraja (discovery, cross-reference)

Note: Dikshitar List is prioritized over other blog lists because:

7.3 Cross-Source Deduplication Strategy

Matching Criteria:

  1. Strong Match: Same composer + same title + same incipit
  2. Medium Match: Same composer + same title (verify incipit)
  3. Weak Match: Same title + similar incipit (may be different compositions)
  4. Discovery Match: Title-only match (use for discovery, verify with primary source)

Priority for Conflicts:

  1. Karnatik.com (highest authority)
  2. Guru Guha (good supplementary)
  3. Dikshitar List (Trinity composer - use for discovery and completeness)
  4. Other blog sources (use for discovery only)

8. Technical Implementation Notes

8.1 Scraping Best Practices

Rate Limiting:

Error Handling:

Caching:

8.2 Source Handler Architecture

interface SourceHandler { suspend fun discoverUrls(): List suspend fun scrape(url: String): RawContent suspend fun extract(content: RawContent): ExtractedMetadata fun getSourceId(): UUID }

class KarnatikSourceHandler : SourceHandler { // Karnatik-specific implementation }

class BlogspotSourceHandler : SourceHandler {
    // Generic Blogspot handler, configurable per blog
}

8.3 AI Extraction Prompts

For Structured Sources (Karnatik):

For Unstructured Sources (Blogs):

Prompt Template:

Extract the following from this HTML content:
1. Composition title
2. Composer name
3. Raga name
4. Tala name (if mentioned)
5. Deity name (if mentioned)
6. Temple/Kshetra name (if mentioned)
7. Language and script
8. Sections (Pallavi, Anupallavi, Charanam) with text
9. Any notation (separate from lyrics)

Return as structured JSON matching this schema: [schema]

9. Data Quality Metrics by Source

9.1 Expected Extraction Accuracy

Source Title Composer Raga Tala Deity Temple Sections Overall
Karnatik 99% 99% 99% 95% 80% 60% 98% 95%
Guru Guha 90% 85% 80% 70% 70% 50% 75% 75%
Dikshitar List 85% 95% 70% 60% 60% 40% 50% 65%
Syama Krishna 85% 90% 70% 60% 60% 40% 50% 65%
Thyagaraja 85% 95% 70% 60% 60% 40% 50% 65%
TempleNet N/A N/A N/A N/A 95% 98% N/A 96%

9.2 Human Review Requirements

Automatic Approval Threshold:

Review Time Estimates:

10. Recommendations

10.1 Immediate Actions

  1. Start with Karnatik.com: Highest ROI, lowest risk
  2. Build Generic Scraper: Reusable for other sources
  3. Implement Source Handlers: Modular architecture
  4. Add Comprehensive Logging: Track extraction accuracy

10.2 Medium-Term

  1. TempleNet Integration: Essential for temple associations
  2. Dikshitar List Handler: Important for Trinity composer coverage
  3. Blog Source Handlers: More complex but valuable
  4. Cross-Source Deduplication: Critical for quality
  5. Quality Metrics Dashboard: Monitor extraction accuracy

10.3 Long-Term

  1. Continuous Import: Scheduled updates
  2. Source Monitoring: Detect structure changes
  3. Automated Quality Checks: Reduce human review
  4. Community Contributions: Allow manual corrections

11. Conclusion

The web sources vary significantly in structure and quality. A tiered approach is recommended:

  1. Tier 1 (Karnatik, TempleNet): High-quality, structured sources - prioritize
  2. Tier 2 (Guru Guha): Good content, requires AI extraction - secondary priority
  3. Tier 3 (Dikshitar List): Trinity composer source - important for completeness, medium priority
  4. Tier 4 (Other Blog Lists): Discovery sources, cross-reference with Tier 1 - lower priority

Note: Dikshitar List is elevated to Tier 3 (separate from other blog lists) because:

Key Success Factors:

12. References

Documentation home · Feature status