54 lines
2.4 KiB
Markdown
54 lines
2.4 KiB
Markdown
# Batch jobs — migrate / improve later
|
|
|
|
Campaign CSV runs and scheduled sends are **UI + data-model only** for now. Actual delivery is stubbed.
|
|
|
|
## Current stack (Docker)
|
|
|
|
| Service | Status | Role |
|
|
|---------|--------|------|
|
|
| **Redis** | Running, unused | Recommended job broker |
|
|
| **Kafka** | Profile `data-plane`, off | Event stream at scale |
|
|
| **Postgres** | Running | Job state + queue (fallback) |
|
|
| **ClickHouse** | Profile `analytics`, off | Analytics rollups |
|
|
| **MinIO** | Profile `object-storage`, off | CSV / export storage |
|
|
|
|
## Recommended migration path
|
|
|
|
### Phase 1 — Redis worker (short term)
|
|
|
|
1. Wire `REDIS_URL=redis://redis:6379/0` into backend and a new `worker` service (same image as API, different `command`).
|
|
2. Add **Celery**, **RQ**, or **ARQ** to `backend/requirements.txt`.
|
|
3. Move campaign run logic from stub endpoints to background tasks:
|
|
- `run_campaign_csv_upload(upload_id)`
|
|
- `run_campaign_automation(campaign_id)` — batch all validated CSVs
|
|
4. Campaign CSV upload with `run_mode=on_upload` should enqueue a job after validation instead of inline stub.
|
|
5. Scheduled campaigns (`run_mode=scheduled`) need a periodic beat/cron worker to enqueue due runs.
|
|
|
|
### Phase 2 — Kafka (when volume grows)
|
|
|
|
1. `docker compose --profile data-plane up -d`
|
|
2. Publish outbound SMS events; consume with dedicated send workers.
|
|
3. Use ClickHouse profile for delivery analytics.
|
|
|
|
### Phase 3 — Object storage
|
|
|
|
1. `docker compose --profile object-storage up -d`
|
|
2. Store campaign CSVs in MinIO instead of local `uploads/campaigns/`.
|
|
|
|
## Campaign send integration checklist
|
|
|
|
- [ ] Replace stub `POST .../run` handlers with worker enqueue
|
|
- [ ] Implement Ultronsms batch/chunked send in worker
|
|
- [ ] Persist per-row send results (success / failure / message_id)
|
|
- [ ] Retry failed rows with backoff
|
|
- [ ] Respect `scheduled_at` for scheduled campaigns
|
|
- [ ] Idempotency key per CSV row to avoid duplicate sends on retry
|
|
- [ ] Rate limiting per sender / DLT template
|
|
|
|
## Files to touch when implementing
|
|
|
|
- `backend/app/routes_campaigns.py` — replace `_stub_run_upload` with enqueue
|
|
- `backend/app/campaign_csv_service.py` — row iteration + send hook
|
|
- `backend/app/message_send_service.py` — reuse template validation + gateway client
|
|
- `docker-compose.yml` — `worker` service + Redis dependency on backend
|
|
- `frontend/src/features/campaigns/` — show real run progress / row counts
|