first commit

docs/BATCH_JOBS_TODO.md

@@ -0,0 +1,54 @@
+# Batch jobs — migrate / improve later
+
+Campaign CSV runs and scheduled sends are **UI + data-model only** for now. Actual delivery is stubbed.
+
+## Current stack (Docker)
+
+| Service | Status | Role |
+|---------|--------|------|
+| **Redis** | Running, unused | Recommended job broker |
+| **Kafka** | Profile `data-plane`, off | Event stream at scale |
+| **Postgres** | Running | Job state + queue (fallback) |
+| **ClickHouse** | Profile `analytics`, off | Analytics rollups |
+| **MinIO** | Profile `object-storage`, off | CSV / export storage |
+
+## Recommended migration path
+
+### Phase 1 — Redis worker (short term)
+
+1. Wire `REDIS_URL=redis://redis:6379/0` into backend and a new `worker` service (same image as API, different `command`).
+2. Add **Celery**, **RQ**, or **ARQ** to `backend/requirements.txt`.
+3. Move campaign run logic from stub endpoints to background tasks:
+   - `run_campaign_csv_upload(upload_id)`
+   - `run_campaign_automation(campaign_id)` — batch all validated CSVs
+4. Campaign CSV upload with `run_mode=on_upload` should enqueue a job after validation instead of inline stub.
+5. Scheduled campaigns (`run_mode=scheduled`) need a periodic beat/cron worker to enqueue due runs.
+
+### Phase 2 — Kafka (when volume grows)
+
+1. `docker compose --profile data-plane up -d`
+2. Publish outbound SMS events; consume with dedicated send workers.
+3. Use ClickHouse profile for delivery analytics.
+
+### Phase 3 — Object storage
+
+1. `docker compose --profile object-storage up -d`
+2. Store campaign CSVs in MinIO instead of local `uploads/campaigns/`.
+
+## Campaign send integration checklist
+
+- [ ] Replace stub `POST .../run` handlers with worker enqueue
+- [ ] Implement Ultronsms batch/chunked send in worker
+- [ ] Persist per-row send results (success / failure / message_id)
+- [ ] Retry failed rows with backoff
+- [ ] Respect `scheduled_at` for scheduled campaigns
+- [ ] Idempotency key per CSV row to avoid duplicate sends on retry
+- [ ] Rate limiting per sender / DLT template
+
+## Files to touch when implementing
+
+- `backend/app/routes_campaigns.py` — replace `_stub_run_upload` with enqueue
+- `backend/app/campaign_csv_service.py` — row iteration + send hook
+- `backend/app/message_send_service.py` — reuse template validation + gateway client
+- `docker-compose.yml` — `worker` service + Redis dependency on backend
+- `frontend/src/features/campaigns/` — show real run progress / row counts