urmi-ai/docs/BATCH_JOBS_TODO.md

Batch jobs — migrate / improve later

Campaign CSV runs and scheduled sends are UI + data-model only for now. Actual delivery is stubbed.

Current stack (Docker)

Service	Status	Role
Redis	Running, unused	Recommended job broker
Kafka	Profile data-plane, off	Event stream at scale
Postgres	Running	Job state + queue (fallback)
ClickHouse	Profile analytics, off	Analytics rollups
MinIO	Profile object-storage, off	CSV / export storage

Recommended migration path

Phase 1 — Redis worker (short term)

1. Wire REDIS_URL=redis://redis:6379/0 into backend and a new worker service (same image as API, different command).
2. Add Celery, RQ, or ARQ to backend/requirements.txt.
3. Move campaign run logic from stub endpoints to background tasks:
   • run_campaign_csv_upload(upload_id)
   • run_campaign_automation(campaign_id) — batch all validated CSVs
4. Campaign CSV upload with run_mode=on_upload should enqueue a job after validation instead of inline stub.
5. Scheduled campaigns (run_mode=scheduled) need a periodic beat/cron worker to enqueue due runs.

Phase 2 — Kafka (when volume grows)

1. docker compose --profile data-plane up -d
2. Publish outbound SMS events; consume with dedicated send workers.
3. Use ClickHouse profile for delivery analytics.

Phase 3 — Object storage

1. docker compose --profile object-storage up -d
2. Store campaign CSVs in MinIO instead of local uploads/campaigns/.

Campaign send integration checklist

• Replace stub POST .../run handlers with worker enqueue
• Implement Ultronsms batch/chunked send in worker
• Persist per-row send results (success / failure / message_id)
• Retry failed rows with backoff
• Respect scheduled_at for scheduled campaigns
• Idempotency key per CSV row to avoid duplicate sends on retry
• Rate limiting per sender / DLT template

Files to touch when implementing

• backend/app/routes_campaigns.py — replace _stub_run_upload with enqueue
• backend/app/campaign_csv_service.py — row iteration + send hook
• backend/app/message_send_service.py — reuse template validation + gateway client
• docker-compose.yml — worker service + Redis dependency on backend
• frontend/src/features/campaigns/ — show real run progress / row counts