| Metadata |
Value |
| Status |
Archived |
| Version |
1.0.0 |
| Last Updated |
2026-09-10 |
| Author |
Sangeetha Grantha Team |
| Document Type |
Archive |
Bulk Import Documentation Consolidation Summary
[!NOTE]
Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current ingestion guide.
| Date |
2026-01-20 |
| Status |
✅ Complete |
What Was Done
1. Organized 12 Newly Added Files
Original Files:
- 3 Koog evaluation documents
- 4 general bulk import analysis documents
- 2 CSV import strategy documents
- 1 technical implementation guide
- 1 web source analysis
- 1 update/correction document
2. Created Consolidated Structure
bulk-import/
├── README.md (Master index with feature roadmap)
├── 01-strategy/
│ ├── master-analysis.md (Consolidated from 4 analysis files)
│ ├── csv-import-strategy.md (Merged from 2 CSV files)
│ └── koog-evaluation.md (Merged from 3 Koog files)
├── 02-implementation/
│ └── technical-implementation-guide.md (Moved, kept as-is)
├── 03-sources/
│ └── web-source-analysis.md (Moved, kept as-is)
└── archive/
└── [All 12 original files preserved]
3. Enhanced Documents
- CSV Import Strategy: Added Phase 1 details from Google analysis (SQL seed file generation approach)
- Master Analysis: Added consolidation note referencing all source documents
- Koog Evaluation: Added consolidation note
4. Created Master Index
- Feature roadmap with 4 phases
- Quick reference guide
- Key decisions summary
- Next steps identification
First Feature for Implementation
✅ Feature 1: CSV Manifest Ingestion (Phase 1)
Priority: HIGH
Complexity: LOW
Timeline: 1 week
Status: Ready to start
Objective: Parse CSV files and load into imported_krithis staging table
Key Documents:
Deliverables:
- Rust CLI command:
tools/sangita-cli/src/commands/import.rs (see Technical Analysis for rationale)
- Parse 3 CSV files from
/database/for_import/
- Generate SQL seed file:
database/seed_data/04_initial_manifest_load.sql
- Handle data cleaning and SQL escaping
- Note: Originally suggested Python, but Rust is recommended to align with existing CLI architecture
- Source registry:
database/seed_data/03_import_sources.sql
- Register 3 blogspot sources with stable UUIDs
- Verification:
- Run
tools/sangita-cli -- db seed
- Verify ~1,240 entries in
imported_krithis table
- All records have
import_status = 'pending'
Implementation Steps:
- Verify
import_sources table exists (migration 04__import-pipeline.sql)
- Generate UUIDs for 3 sources
- Add CSV parsing to Rust CLI (
csv = "1.3" dependency, new import command)
- Test with one CSV file first
- Generate all SQL seed files
- Run
cargo run -- db seed and verify data
Success Criteria:
- ✅ All 3 CSV files parse successfully
- ✅ ~1,240 entries loaded into staging table
- ✅ Source attribution correctly mapped
- ✅ Ready for Phase 2 (web scraping)
Next Steps
- Review Phase 1 Design: Review CSV Import Strategy - Phase 1
- Start Implementation: Begin with CSV parser script
- Plan Phase 2: Review web scraping requirements after Phase 1 complete
Document References
Documentation home · Feature status