# Batch jobs — migrate / improve later

Campaign CSV runs and scheduled sends are **UI + data-model only** for now. Actual delivery is stubbed.

## Current stack (Docker)

| Service | Status | Role |
|---------|--------|------|
| **Redis** | Running, unused | Recommended job broker |
| **Kafka** | Profile `data-plane`, off | Event stream at scale |
| **Postgres** | Running | Job state + queue (fallback) |
| **ClickHouse** | Profile `analytics`, off | Analytics rollups |
| **MinIO** | Profile `object-storage`, off | CSV / export storage |

## Recommended migration path

### Phase 1 — Redis worker (short term)

1. Wire `REDIS_URL=redis://redis:6379/0` into backend and a new `worker` service (same image as API, different `command`).
2. Add **Celery**, **RQ**, or **ARQ** to `backend/requirements.txt`.
3. Move campaign run logic from stub endpoints to background tasks:
   - `run_campaign_csv_upload(upload_id)`
   - `run_campaign_automation(campaign_id)` — batch all validated CSVs
4. Campaign CSV upload with `run_mode=on_upload` should enqueue a job after validation instead of inline stub.
5. Scheduled campaigns (`run_mode=scheduled`) need a periodic beat/cron worker to enqueue due runs.

### Phase 2 — Kafka (when volume grows)

1. `docker compose --profile data-plane up -d`
2. Publish outbound SMS events; consume with dedicated send workers.
3. Use ClickHouse profile for delivery analytics.

### Phase 3 — Object storage

1. `docker compose --profile object-storage up -d`
2. Store campaign CSVs in MinIO instead of local `uploads/campaigns/`.

## Campaign send integration checklist

- [ ] Replace stub `POST .../run` handlers with worker enqueue
- [ ] Implement Ultronsms batch/chunked send in worker
- [ ] Persist per-row send results (success / failure / message_id)
- [ ] Retry failed rows with backoff
- [ ] Respect `scheduled_at` for scheduled campaigns
- [ ] Idempotency key per CSV row to avoid duplicate sends on retry
- [ ] Rate limiting per sender / DLT template

## Files to touch when implementing

- `backend/app/routes_campaigns.py` — replace `_stub_run_upload` with enqueue
- `backend/app/campaign_csv_service.py` — row iteration + send hook
- `backend/app/message_send_service.py` — reuse template validation + gateway client
- `docker-compose.yml` — `worker` service + Redis dependency on backend
- `frontend/src/features/campaigns/` — show real run progress / row counts