Sangeetha-Grantha

Metadata	Value
Status	Active
Version	1.1.0
Last Updated	2026-09-10
Author	Sangeetha Grantha Team
Document Type	Evidence record

Bulk Import Fixes - Implementation Plan

[!NOTE] Historical evidence: results, counts, commands, and observations below belong to the original work described here. The editorial update date is not a new test or corpus verification. For present behavior, use current quality checks.

Executive Summary

This document consolidates findings from three comprehensive code reviews (Claude, Goose, Codex) and provides a prioritized implementation plan to address critical issues, gaps, and improvements in the bulk import system.

Review Summary

Overall Assessment:

Architecture Grade: A- (Excellent foundation, follows v2 technical guide)
Implementation Grade: B+ (Well-executed core pipeline, Phase 4 features pending)
Production Readiness: 75% (Core pipeline production-ready, review workflow needed for full automation)

Key Findings:

✅ Excellent unified dispatcher pattern and event-driven architecture
✅ Robust error handling and idempotency mechanisms
⚠️ Missing quality scoring system (critical for Phase 4)
⚠️ Incomplete review workflow APIs
🔴 Critical correctness issues (manifest failure handling, task stuck detection)
🔴 Security concerns (file upload vulnerabilities)

Issue Categorization

Critical (Must Fix - Blocking Production)

Manifest Ingest Failure Handling (All 3 reviews)
- Batch not marked FAILED when manifest ingest fails with zero tasks
- Violates clarified requirements (2026-01)
- Track: TRACK-010
Task Stuck Detection Race Condition (Codex)
- Tasks marked RUNNING at claim time, but watchdog may mark as RETRYABLE before execution
- Risk of double-processing and duplicate side effects
- Track: TRACK-010
File Upload Security Vulnerabilities (Codex, Goose)
- Path traversal risk (no filename sanitization)
- No file size limits (OOM risk)
- Null filename handling
- Track: TRACK-010
Quality Scoring System Missing (All 3 reviews)
- Strategy specifies quality tiers (EXCELLENT, GOOD, FAIR, POOR)
- No calculation or persistence of quality scores
- Blocks review prioritization and automation
- Track: TRACK-011

High Priority (Should Fix - Quality & Completeness)

Review Workflow APIs Incomplete (Claude, Goose)
- Missing bulk-review and auto-approve queue endpoints
- Limited batch-scale moderation workflows
- Track: TRACK-012
Auto-Approval Logic Incomplete (All 3 reviews)
- Hardcoded heuristics instead of configurable rules
- Missing quality tier integration
- Track: TRACK-012
Entity Resolution Cache Issues (Goose, Codex)
- In-memory only, no database persistence
- Cache invalidation missing when new entities created
- Multi-node deployment divergence risk
- Track: TRACK-013
Deduplication Service Incomplete (All 3 reviews)
- Missing intra-batch deduplication during processing
- O(N^2) performance for large batches
- Track: TRACK-013

Medium Priority (Performance & Scalability)

Stage Completion Checks O(N) per Task (Claude, Codex)
- checkAndTriggerNextStage loads all tasks on every completion
- ~1,200 unnecessary DB queries per batch
- Track: TRACK-013
Rate Limiting Too Conservative (Claude, Codex)
- Defaults (12/min per domain, 50/min global) vs strategy (120/min)
- 5-10x slower than strategy estimates
- Track: TRACK-013
CSV Validation Deferred (Codex)
- Validation happens at manifest ingest, not upload
- Invalid CSVs fail minutes later instead of fast-failing
- Track: TRACK-013
CSV Parsing Issues (Codex)
- Platform default charset (diacritic handling risk)
- File readers not closed (file descriptor leaks)
- Track: TRACK-010

Low Priority (Code Quality & Polish)

Normalization Bugs (Codex)
- Honorific removal regex broken ("\b" is backspace, not word boundary)
- Normalized lookup maps drop collisions
- Track: TRACK-013
Memory Leak in Rate Limiter (Claude)
- perDomainWindows map grows unbounded
- Track: TRACK-013
Testing Gaps (All 3 reviews)
- No unit tests for normalization/resolution logic
- No integration tests for full pipeline
- Track: TRACK-014
Frontend Improvements (Goose)
- Missing batch filter in review queue
- No quality tier filtering
- Track: TRACK-012

Implementation Tracks

TRACK-010: Critical Fixes & Security Hardening

Priority: CRITICAL
Estimated Effort: 2-3 days
Status: Proposed

Scope:

Fix manifest ingest failure handling (mark batch FAILED)
Fix task stuck detection race condition (QUEUED state or delayed RUNNING)
Fix file upload security (path traversal, size limits, null handling)
Fix CSV parsing (charset, file handles)

Dependencies: None
Blocks: Production deployment

TRACK-011: Quality Scoring System

Priority: HIGH
Estimated Effort: 3-4 days
Status: Proposed

Scope:

Implement quality score calculation (completeness, confidence, validation)
Add quality tier assignment (EXCELLENT, GOOD, FAIR, POOR)
Persist scores to imported_krithis table
Integrate with review workflow and auto-approval

Dependencies: None
Blocks: Review prioritization, auto-approval automation

TRACK-012: Review Workflow Completion

Priority: HIGH
Estimated Effort: 4-5 days
Status: Proposed

Scope:

Complete review workflow APIs (bulk-review, auto-approve queues)
Enhance auto-approval with configurable rules and quality tiers
Add batch filtering to review queue (frontend)
Add quality tier filtering (frontend)

Dependencies: TRACK-011 (quality scoring)
Blocks: Full automation, batch-scale moderation

TRACK-013: Performance & Scalability Improvements

Priority: MEDIUM
Estimated Effort: 5-7 days
Status: Proposed

Scope:

Optimize stage completion checks (counter-based instead of O(N))
Implement entity resolution cache (database-backed)
Complete deduplication service (intra-batch, optimized queries)
Tune rate limiting (after real-world testing)
Fix normalization bugs (honorific regex, collision handling)
Fix rate limiter memory leak (LRU cache)
Add CSV validation at upload (fast-fail)

Dependencies: None
Blocks: Scale beyond 5,000 krithis

TRACK-014: Testing & Quality Assurance

Priority: MEDIUM
Estimated Effort: 5-7 days
Status: Proposed

Scope:

Unit tests for normalization service
Unit tests for entity resolution logic
Unit tests for deduplication heuristics
Integration tests for full pipeline (manifest → scrape → resolution → review)
Performance tests for large batches (100+ entries)
Error recovery scenario tests

Dependencies: TRACK-010, TRACK-011, TRACK-012, TRACK-013
Blocks: Confidence in production stability

Implementation Timeline

Phase 1: Critical Fixes (Week 1)

TRACK-010: Critical Fixes & Security Hardening
- Days 1-2: Manifest failure handling, task stuck detection
- Day 3: File upload security, CSV parsing fixes

Phase 2: Quality & Review (Weeks 2-3)

TRACK-011: Quality Scoring System
- Days 1-2: Score calculation logic
- Days 3-4: Database persistence, tier assignment
TRACK-012: Review Workflow Completion
- Days 1-2: Backend APIs (bulk-review, auto-approve queues)
- Days 3-4: Auto-approval enhancements
- Day 5: Frontend improvements

Phase 3: Performance & Testing (Weeks 4-5)

TRACK-013: Performance & Scalability
- Days 1-2: Stage completion optimization, entity cache
- Days 3-4: Deduplication improvements
- Days 5-6: Rate limiting tuning, normalization fixes
- Day 7: CSV validation at upload
TRACK-014: Testing & Quality Assurance
- Days 1-3: Unit tests
- Days 4-5: Integration tests
- Days 6-7: Performance tests, error recovery tests

Success Criteria

TRACK-010 (Critical Fixes)

✅ Manifest ingest failures mark batch as FAILED (even with zero tasks)
✅ Tasks only marked RUNNING when worker begins execution
✅ File uploads sanitized, size-limited, null-safe
✅ CSV parsing uses UTF-8, closes file handles

TRACK-011 (Quality Scoring)

✅ Quality scores calculated for all imports
✅ Quality tiers assigned (EXCELLENT ≥0.90, GOOD ≥0.75, FAIR ≥0.60, POOR <0.60)
✅ Scores persisted to database
✅ Scores visible in review UI

TRACK-012 (Review Workflow)

✅ Bulk-review API endpoint functional
✅ Auto-approve queue API endpoint functional
✅ Auto-approval uses configurable rules and quality tiers
✅ Batch filter in review queue (frontend)
✅ Quality tier filter in review queue (frontend)

TRACK-013 (Performance)

✅ Stage completion checks use counters (O(1) instead of O(N))
✅ Entity resolution cache database-backed with invalidation
✅ Deduplication optimized (DB queries, intra-batch support)
✅ Rate limiting tuned based on real-world testing
✅ Normalization bugs fixed
✅ Rate limiter memory leak fixed

TRACK-014 (Testing)

✅ Unit test coverage >80% for normalization, resolution, deduplication
✅ Integration test for full pipeline
✅ Performance test for 100+ entry batches
✅ Error recovery tests pass

Risk Assessment

High Risk

TRACK-010: Security vulnerabilities could be exploited in production
TRACK-011: Missing quality scoring blocks review automation goals

Medium Risk

TRACK-012: Incomplete review workflow limits batch-scale operations
TRACK-013: Performance issues may not surface until scale testing

Low Risk

TRACK-014: Testing gaps don’t block production but reduce confidence

Dependencies & Blockers

Critical Path

TRACK-010 → Must complete before production deployment
TRACK-011 → Blocks TRACK-012 (review workflow needs quality scores)
TRACK-012 → Depends on TRACK-011
TRACK-013 → Can proceed in parallel with TRACK-011/012
TRACK-014 → Should follow TRACK-010-013 completion

External Dependencies

Real-world rate limiting testing requires access to blogspot.com scraping
Performance testing requires test data fixtures (1,200+ entries)

Notes

Clarified Requirements (2026-01)

The implementation correctly follows most clarified requirements:

✅ URL validation is syntax-only (no HEAD/GET required)
✅ CSV Raga column is optional (scraped values are authoritative)
⚠️ Manifest ingest failure handling needs fix (TRACK-010)
✅ Scraping failure handling: CSV metadata not used in scraping stage (N/A)

Architecture Decisions

Unified Dispatcher pattern (TRACK-007) is correct and well-implemented
Event-driven wakeup eliminates polling latency concerns
Three-level hierarchy (Batch → Job → Task) provides excellent granularity

Performance Considerations

Current rate limits (12/min per domain) are conservative by design
Strategy estimates (120/min) should be validated with real-world testing
Entity resolution caching is critical for scale beyond 5,000 krithis

References

Implementation Plan Created: 2026-01-23
Next Review: After TRACK-010 completion

Section index · Documentation home · Feature status