258 lines
6.7 KiB
Markdown
258 lines
6.7 KiB
Markdown
# Browser Monitoring & Notification Agent
|
|
|
|
A Python application that continuously monitors websites using a **real browser** installed on your machine. It visits configured sites in round-robin order, captures snapshots (screenshots + metadata), and is designed for future OpenAI-based page understanding and alert notifications.
|
|
|
|
## Features
|
|
|
|
- **Browser-first monitoring** — Uses Playwright with your installed Chrome, Chromium, Edge, or Firefox
|
|
- **Persistent browser profile** — Reuses cookies, sessions, and login state across runs
|
|
- **Round-robin scheduling** — Visits one website at a time, sequentially
|
|
- **Snapshots** — Screenshots, page title, visible text, HTML, and metadata
|
|
- **Snapshot retention** — Configurable cleanup to prevent unbounded disk growth
|
|
- **OpenAI stub** — Ready for future page understanding integration
|
|
- **Notification framework** — SMS and email abstractions (logging by default)
|
|
- **Error resilience** — One site failing does not stop the monitor
|
|
- **Browser recovery** — Automatic restart on browser crashes
|
|
- **Graceful shutdown** — Clean exit on Ctrl+C
|
|
|
|
## Requirements
|
|
|
|
- Python 3.10+
|
|
- A supported browser installed locally (Chrome recommended)
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
cd browser_automation
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Playwright is used with your **installed browser**, not a separately downloaded Playwright browser bundle. If auto-detection fails, set `browser.executable_path` in `config.yaml`.
|
|
|
|
Copy environment variable template:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
Never commit `.env` — it is listed in `.gitignore`.
|
|
|
|
## Browser Configuration
|
|
|
|
Edit `config.yaml`:
|
|
|
|
```yaml
|
|
browser:
|
|
type: chrome # chrome | chromium | edge | firefox
|
|
executable_path: null # set if auto-detection fails
|
|
headless: false # false = visible browser window
|
|
profile_directory: "./profiles/default"
|
|
```
|
|
|
|
### Persistent Profile / Login Sessions
|
|
|
|
1. Set `profile_directory` to a dedicated folder (e.g. `./profiles/default`).
|
|
2. Run the **login launcher** to open that profile and sign in manually.
|
|
3. Close the browser or press Ctrl+C when finished.
|
|
4. Start the monitor — sessions and cookies are preserved.
|
|
|
|
Do **not** store passwords in `config.yaml`. Use the browser profile for authentication.
|
|
|
|
### Login launcher
|
|
|
|
Use `login.py` to open the monitor profile without starting the monitoring loop:
|
|
|
|
```bash
|
|
# Open profile with a blank tab
|
|
python login.py
|
|
|
|
# Open BookMyShow (or any URL)
|
|
python login.py --url "https://in.bookmyshow.com"
|
|
|
|
# Open all enabled websites from config.yaml as tabs
|
|
python login.py --websites
|
|
|
|
# Open a specific configured site by id
|
|
python login.py --website-id website_1
|
|
```
|
|
|
|
The launcher always opens a **visible** browser window, even if `headless: true` in config.
|
|
|
|
### Auto-Detection Paths (Linux)
|
|
|
|
| Browser | Typical paths |
|
|
|----------|---------------|
|
|
| Chrome | `/usr/bin/google-chrome`, `/usr/bin/google-chrome-stable` |
|
|
| Chromium | `/usr/bin/chromium`, `/usr/bin/chromium-browser` |
|
|
| Edge | `/usr/bin/microsoft-edge` |
|
|
| Firefox | `/usr/bin/firefox` |
|
|
|
|
## Configuration
|
|
|
|
`config.yaml` controls monitoring behaviour:
|
|
|
|
```yaml
|
|
monitor:
|
|
round_robin_delay_seconds: 5 # pause between website visits
|
|
page_load_timeout_seconds: 30
|
|
render_wait_seconds: 3 # extra wait for JavaScript rendering
|
|
snapshot_directory: "./snapshots"
|
|
snapshot_retention_count: 50 # max screenshots per website
|
|
|
|
websites:
|
|
- id: website_1
|
|
name: Website One
|
|
url: "https://example.com"
|
|
enabled: true
|
|
|
|
notifications:
|
|
sms:
|
|
enabled: false
|
|
email:
|
|
enabled: false
|
|
```
|
|
|
|
Add as many websites as needed. Disabled sites are skipped.
|
|
|
|
## Usage
|
|
|
|
### Start monitoring
|
|
|
|
```bash
|
|
python main.py
|
|
```
|
|
|
|
### Custom config file
|
|
|
|
```bash
|
|
python main.py --config config.yaml
|
|
```
|
|
|
|
### Debug logging
|
|
|
|
```bash
|
|
python main.py --debug
|
|
```
|
|
|
|
### Test browser setup
|
|
|
|
```bash
|
|
python main.py --test-browser
|
|
python main.py --test-browser --test-url "https://example.com"
|
|
```
|
|
|
|
### Graceful shutdown
|
|
|
|
Press `Ctrl+C`. The monitor stops scheduling new checks, closes the browser, and exits cleanly.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Browser → Snapshot → OpenAI Processor (stub) → Condition Engine → Notifications
|
|
```
|
|
|
|
| Layer | Responsibility |
|
|
|-------|----------------|
|
|
| OpenAI (future) | Page understanding, value extraction |
|
|
| Python | Scheduling, condition evaluation, notifications |
|
|
| Playwright | Real browser automation |
|
|
|
|
### Project Structure
|
|
|
|
```
|
|
browser_automation/
|
|
├── main.py
|
|
├── config.yaml
|
|
├── monitor/
|
|
│ ├── browser.py # Browser discovery, launch, navigation
|
|
│ ├── scheduler.py # Round-robin loop
|
|
│ ├── snapshot.py # Page capture
|
|
│ ├── processor.py # OpenAI stub + condition evaluation
|
|
│ ├── models.py # PageSnapshot, ProcessingResult, etc.
|
|
│ └── config.py # YAML loading
|
|
├── notifications/
|
|
│ ├── base.py # NotificationService abstraction
|
|
│ ├── sms.py # SMS provider stub
|
|
│ └── email.py # SMTP / logging email
|
|
├── storage/
|
|
│ └── snapshots.py # Screenshot storage + retention
|
|
├── profiles/ # Persistent browser profiles
|
|
└── snapshots/ # Captured screenshots
|
|
```
|
|
|
|
## Snapshots
|
|
|
|
Screenshots are saved per website:
|
|
|
|
```
|
|
snapshots/
|
|
website_1/
|
|
2026-09-30T13-20-00.png
|
|
website_2/
|
|
2026-09-30T13-20-05.png
|
|
```
|
|
|
|
Older snapshots are removed automatically when `snapshot_retention_count` is exceeded.
|
|
|
|
## Notifications
|
|
|
|
By default, notifications are **logged** to the console:
|
|
|
|
```
|
|
SMS WOULD BE SENT:
|
|
...
|
|
|
|
EMAIL WOULD BE SENT:
|
|
Subject: ...
|
|
```
|
|
|
|
### Email (SMTP)
|
|
|
|
Set environment variables in `.env`:
|
|
|
|
```
|
|
SMTP_HOST=smtp.example.com
|
|
SMTP_PORT=587
|
|
SMTP_USER=your-user
|
|
SMTP_PASSWORD=your-password
|
|
EMAIL_FROM=monitor@example.com
|
|
EMAIL_TO=you@example.com
|
|
```
|
|
|
|
Enable in `config.yaml`:
|
|
|
|
```yaml
|
|
notifications:
|
|
email:
|
|
enabled: true
|
|
```
|
|
|
|
### SMS
|
|
|
|
Configure provider credentials via environment variables (`SMS_PROVIDER`, `SMS_API_KEY`, etc.). Enable in config when a provider is implemented.
|
|
|
|
## Error Handling
|
|
|
|
- Navigation timeouts, DNS failures, and page errors are logged per website
|
|
- The monitor continues to the next site
|
|
- Browser crashes trigger automatic restart with backoff
|
|
- Bot/automation blocks are logged and skipped — no CAPTCHA or bypass logic
|
|
|
|
## Security
|
|
|
|
- No credentials in source code or `config.yaml`
|
|
- Use `.env` for secrets (not committed)
|
|
- No CAPTCHA bypass, Cloudflare bypass, or fingerprint spoofing
|
|
|
|
## Future Extensions
|
|
|
|
The architecture supports adding:
|
|
|
|
- OpenAI page understanding and condition triggers
|
|
- Per-site instructions and check intervals
|
|
- Multiple browser profiles
|
|
- Alert deduplication and cooldown
|
|
- Urmi, Twilio, WhatsApp integrations
|
|
- Dashboard and database storage
|