Sangeetha-Grantha

Metadata Value
Status Active
Version 1.1.1
Last Updated 2026-02-19
Author Sangeetha Grantha Team

Sangita CLI

A unified command-line tool for the Sangita Grantha project.

Prerequisites

Recommended: Use mise for toolchain management (see .mise.toml in project root).

Tools managed by mise (see current versions):

System requirements (not managed by mise):

Installation

Build the tool from source:

cd tools/sangita-cli
cargo build --release

The binary will be at target/release/sangita-cli.

Usage

Since Rust and other tools are managed by mise, run sangita-cli through mise:

# Development workflow
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- dev --start-db

# Database management - Reset existing database database 
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- db reset

# Database management - Applying database schema and seed data changes 
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- db migrate

# Testing
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- test steel-thread

# Extraction E2E (backend + queue + Python worker)
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- test extraction-e2e --timeout-seconds 600

# Blogspot HTML E2E scenario (auto-picks one URL from Dikshitar-Krithi-Test-20.csv)
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- test extraction-e2e --scenario blogspot-html --timeout-seconds 600

# Akhila 3-source regression (Blogspot HTML + Roman PDF + Sanskrit PDF fixtures)
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- test extraction-e2e --scenario akhila-three-source --timeout-seconds 600

# Large-set Dikshitar key collision scan (flags first10+raga+tala outliers, continues run)
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- \
  test extraction-e2e --scenario dikshitar-key-collision \
  --csv-path database/for_import/Dikshitar-Krithi-For-Import.csv \
  --skip-migrations --skip-extraction-start --timeout-seconds 600

# Setup check
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- setup

This ensures:

Without mise (Fallback)

If mise is not available, ensure tools are installed manually with correct versions:

cd tools/sangita-cli
cargo run -- dev --start-db

Note: You must ensure tool versions match .mise.toml requirements manually.

Setup

Check environment and dependencies:

# Via mise (recommended)
mise exec -- cargo run --manifest-path tools/sangita-cli/Cargo.toml -- setup

# Or without mise
cd tools/sangita-cli
cargo run -- setup

Commit Guardrails

Validate commit messages and manage Git hooks:

# Validate a commit message (reads from stdin if --message not provided)
cargo run -- commit check --message "Your commit message here"

# Install Git hooks (commit-msg and pre-commit)
cargo run -- commit install-hooks

# Remove installed Git hooks
cargo run -- commit uninstall-hooks

# Manually scan staged files for sensitive data
cargo run -- commit scan-sensitive

Commit Message Format: All commits must include a reference to a documentation file in application_documentation/:

<subject line>

Ref: application_documentation/01-requirements/features/my-feature.md

<optional body>

Features:

Database Management

Reset database (Drop → Create → Migrate → Seed):

cargo run -- db reset

Run migrations only:

cargo run -- db migrate

Development

Start Backend and Frontend servers:

cargo run -- dev

Start the full stack (DB + Backend + Frontend) for manual verification:

cargo run -- dev --start-db

This will:

  1. Clean up existing processes using the configured ports (database: 5432, backend: 8080, frontend: 5001)
  2. Ensure Postgres is running (if --start-db)
  3. Wait for the backend /health endpoint
  4. Launch the frontend

Press Ctrl+C to gracefully stop all services.

Network & Mobile Setup

Smoke Test

Quick health + search check:

cargo run -- test upload --base-url http://192.168.0.42:8080

Mobile Docs

Show where the mobile testing guides/checklists live:

cargo run -- mobile guide

PDF Extraction Service

Manage the containerised Python PDF extraction service for multi-format data ingestion:

# Build the PDF extractor Docker image
cargo run -- extraction build

# Start the extraction service (with database)
cargo run -- extraction start --with-db

# Start and follow logs
cargo run -- extraction start --with-db --follow

# View logs
cargo run -- extraction logs -f

# Check service and queue status
cargo run -- extraction status

# Stop the service
cargo run -- extraction stop

# Restart the service
cargo run -- extraction restart

Commands:

Requirements:

Documentation Management

Sync version information from source files to documentation:

# Generate/update current-versions.md
cargo run -- docs sync-versions

# Check if versions are in sync (for CI - exits with error if out of sync)
cargo run -- docs sync-versions --check

# Validate documentation links (future feature)
cargo run -- docs validate-links

Sources of Truth:

Output: application_documentation/00-meta/current-versions.md

Testing

Extraction E2E Integration Test

Run full backend extraction integration checks (no Playwright/UI dependency):

cargo run -- test extraction-e2e

What it validates automatically:

  1. Database is running (and migrations/seed unless skipped)
  2. Backend starts and passes /health
  3. Extraction worker container is running (or started by the command)
  4. Auth token issuance and extraction submission through API
  5. Queue lifecycle reaches INGESTED (PENDING -> PROCESSING -> DONE -> INGESTED)
  6. Database integrity checks for the submitted task:
    • extraction_queue.status = INGESTED
    • non-empty extraction_method and extractor_version
    • result_count threshold validation
    • matching result_payload sanity checks
    • source evidence rows in krithi_source_evidence

Useful options:

# Run Blogspot HTML scenario (source URL is auto-selected from database/for_import/Dikshitar-Krithi-Test-20.csv)
cargo run -- test extraction-e2e --scenario blogspot-html

# Run Akhila 3-source convergence regression
# (uses fixtures in tools/sangita-cli/fixtures/extraction/)
cargo run -- test extraction-e2e --scenario akhila-three-source

# Run full Dikshitar collision scan and fail only if you request strict mode
cargo run -- test extraction-e2e --scenario dikshitar-key-collision --max-rows 200
cargo run -- test extraction-e2e --scenario dikshitar-key-collision --fail-on-collision

# Keep services up for manual debugging after test
cargo run -- test extraction-e2e --keep-services

# Reuse an already-running worker, skip migration step
cargo run -- test extraction-e2e --skip-extraction-start --skip-migrations

# Adjust source/page range and assertions
cargo run -- test extraction-e2e --source-url "<pdf-url>" --page-range "17-18" --min-result-count 1

TRACK-068 Markdown Ingestion Harness

Run focused harness checks for the TRACK-068 markdown workflow (mdskt.md/mdeng.md):

# Regenerate parser outputs + validate parity/artifacts
cargo run -- test track068-harness

# Validate existing artifacts only (skip parser/generator rerun)
cargo run -- test track068-harness --skip-regenerate

# Enforce strict ingestion gates (fail on unknown metadata/missing pallavi)
cargo run -- test track068-harness --enforce-ingestion-gates

# Skip semantic EN<->SA title/raga/tala alignment gate (diagnostics only)
cargo run -- test track068-harness --skip-semantic-alignment

What it verifies:

  1. Parser/generator scripts run successfully (unless --skip-regenerate)
  2. skt_krithis.json and eng_krithis.json both contain the expected count (default 479)
  3. ID parity is contiguous (1..N) and identical across Sanskrit/English datasets
  4. Generated artifacts exist and align with expected counts:
    • final_mdskt.md
    • final_mdeng.md
    • krithi_comparison_report.csv
  5. Semantic EN↔SA alignment scoring from krithi_comparison_report.csv:
    • Transliteration-aware fuzzy checks for title/raga/tala
    • Fails by default if mismatch ratios exceed configured thresholds
  6. Writes a structured summary report:
    • database/for_import/track_068_harness_report.json

Requirements for semantic scoring:

Steel Thread Test

Run the end-to-end smoke verification:

cargo run -- test steel-thread

What it checks:

  1. Database connectivity + migrations
  2. Backend health (/health)
  3. Krithi search endpoint (/v1/krithis/search)
  4. Admin audit log access (using ADMIN_TOKEN)
  5. Frontend dev server startup

Manual Verification: After automated checks pass, the system stays running for manual verification:

Press Ctrl+C to stop servers and exit.

Configuration

Environment variables (set in .env file):

Troubleshooting

Backend fails to start

  1. Check Gradle logs for compilation errors
  2. Verify database is running: cargo run -- db health
  3. Check if port 8080 is already in use: lsof -i :8080
  4. Try running backend manually: ./gradlew :modules:backend:api:run

Database connection issues

  1. Ensure Docker is running (or PostgreSQL is installed)
  2. Check database health: cargo run -- db health
  3. Try starting database: cargo run -- db start
  4. Review PostgreSQL logs in the configured pg_data directory

Frontend fails to start

  1. Ensure Bun dependencies are installed: cd modules/frontend/sangita-admin-web && bun install
  2. Check if port 5001 is available: lsof -i :5001
  3. Try starting frontend manually: cd modules/frontend/sangita-admin-web && bun run dev

Rust-Based Git Guardrails Implementation

The commit guardrails are implemented in tools/sangita-cli/src/commands/commit.rs.

Technical Implementation

Dependencies The implementation uses the following dependencies (already in Cargo.toml):

regex = "1.11.1"  # Commit message parsing and sensitive data pattern matching
clap = { version = "4.5.53", features = ["derive"] }  # CLI argument parsing

Command Structure The CLI command structure (tools/sangita-cli/src/commands/commit.rs):

#[derive(Subcommand)]
pub enum CommitCommands {
    /// Validate commit message format and reference
    Check {
        /// Commit message to validate (or read from stdin)
        #[arg(long)]
        message: Option<String>,
    },
    /// Scan staged files for sensitive data (used by pre-commit hook)
    ScanSensitive,
    /// Install Git hooks for commit validation
    InstallHooks,
    /// Remove installed Git hooks
    UninstallHooks,
}

Git Hook Scripts When install-hooks is run, it creates two hooks:

  1. .git/hooks/commit-msg - Validates commit message format: ```bash #!/bin/sh

    Sangita Grantha Commit Guardrails Hook

    This hook validates commit messages to ensure they reference documentation

exec “/path/to/sangita-cli” commit check –message “$(cat “$1”)”


2. **`.git/hooks/pre-commit`** - Scans for sensitive data:
```bash
#!/bin/sh
# Sangita Grantha Pre-commit Hook
# This hook scans staged files for sensitive data

exec "/path/to/sangita-cli" commit scan-sensitive

The hooks automatically detect the binary location (release/debug) or fall back to cargo run for development.

Validation Logic

  1. Extract Reference: Parses commit message using regex pattern (?i)ref:\s*(.+?)(?:\n|$) to find Ref: <path>.
  2. Single Reference Check: Ensures only one reference exists per commit (enforces 1:1 mapping).
  3. File Existence Check: Verifies that the referenced path exists and is within application_documentation/ directory.
  4. Path Normalization: Handles relative paths, paths starting with application_documentation/, and normalizes to absolute paths.
  5. Exit Code: Returns 0 on success, 1 on failure (which blocks the commit in the hook).

Sensitive Data Scanning The scan-sensitive command: