first commit
This commit is contained in:
commit
dc1245f164
22 changed files with 1666 additions and 0 deletions
258
README.md
Normal file
258
README.md
Normal file
|
|
@ -0,0 +1,258 @@
|
|||
# Browser Monitoring & Notification Agent
|
||||
|
||||
A Python application that continuously monitors websites using a **real browser** installed on your machine. It visits configured sites in round-robin order, captures snapshots (screenshots + metadata), and is designed for future OpenAI-based page understanding and alert notifications.
|
||||
|
||||
## Features
|
||||
|
||||
- **Browser-first monitoring** — Uses Playwright with your installed Chrome, Chromium, Edge, or Firefox
|
||||
- **Persistent browser profile** — Reuses cookies, sessions, and login state across runs
|
||||
- **Round-robin scheduling** — Visits one website at a time, sequentially
|
||||
- **Snapshots** — Screenshots, page title, visible text, HTML, and metadata
|
||||
- **Snapshot retention** — Configurable cleanup to prevent unbounded disk growth
|
||||
- **OpenAI stub** — Ready for future page understanding integration
|
||||
- **Notification framework** — SMS and email abstractions (logging by default)
|
||||
- **Error resilience** — One site failing does not stop the monitor
|
||||
- **Browser recovery** — Automatic restart on browser crashes
|
||||
- **Graceful shutdown** — Clean exit on Ctrl+C
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.10+
|
||||
- A supported browser installed locally (Chrome recommended)
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd browser_automation
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Playwright is used with your **installed browser**, not a separately downloaded Playwright browser bundle. If auto-detection fails, set `browser.executable_path` in `config.yaml`.
|
||||
|
||||
Copy environment variable template:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Never commit `.env` — it is listed in `.gitignore`.
|
||||
|
||||
## Browser Configuration
|
||||
|
||||
Edit `config.yaml`:
|
||||
|
||||
```yaml
|
||||
browser:
|
||||
type: chrome # chrome | chromium | edge | firefox
|
||||
executable_path: null # set if auto-detection fails
|
||||
headless: false # false = visible browser window
|
||||
profile_directory: "./profiles/default"
|
||||
```
|
||||
|
||||
### Persistent Profile / Login Sessions
|
||||
|
||||
1. Set `profile_directory` to a dedicated folder (e.g. `./profiles/default`).
|
||||
2. Run the **login launcher** to open that profile and sign in manually.
|
||||
3. Close the browser or press Ctrl+C when finished.
|
||||
4. Start the monitor — sessions and cookies are preserved.
|
||||
|
||||
Do **not** store passwords in `config.yaml`. Use the browser profile for authentication.
|
||||
|
||||
### Login launcher
|
||||
|
||||
Use `login.py` to open the monitor profile without starting the monitoring loop:
|
||||
|
||||
```bash
|
||||
# Open profile with a blank tab
|
||||
python login.py
|
||||
|
||||
# Open BookMyShow (or any URL)
|
||||
python login.py --url "https://in.bookmyshow.com"
|
||||
|
||||
# Open all enabled websites from config.yaml as tabs
|
||||
python login.py --websites
|
||||
|
||||
# Open a specific configured site by id
|
||||
python login.py --website-id website_1
|
||||
```
|
||||
|
||||
The launcher always opens a **visible** browser window, even if `headless: true` in config.
|
||||
|
||||
### Auto-Detection Paths (Linux)
|
||||
|
||||
| Browser | Typical paths |
|
||||
|----------|---------------|
|
||||
| Chrome | `/usr/bin/google-chrome`, `/usr/bin/google-chrome-stable` |
|
||||
| Chromium | `/usr/bin/chromium`, `/usr/bin/chromium-browser` |
|
||||
| Edge | `/usr/bin/microsoft-edge` |
|
||||
| Firefox | `/usr/bin/firefox` |
|
||||
|
||||
## Configuration
|
||||
|
||||
`config.yaml` controls monitoring behaviour:
|
||||
|
||||
```yaml
|
||||
monitor:
|
||||
round_robin_delay_seconds: 5 # pause between website visits
|
||||
page_load_timeout_seconds: 30
|
||||
render_wait_seconds: 3 # extra wait for JavaScript rendering
|
||||
snapshot_directory: "./snapshots"
|
||||
snapshot_retention_count: 50 # max screenshots per website
|
||||
|
||||
websites:
|
||||
- id: website_1
|
||||
name: Website One
|
||||
url: "https://example.com"
|
||||
enabled: true
|
||||
|
||||
notifications:
|
||||
sms:
|
||||
enabled: false
|
||||
email:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
Add as many websites as needed. Disabled sites are skipped.
|
||||
|
||||
## Usage
|
||||
|
||||
### Start monitoring
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
### Custom config file
|
||||
|
||||
```bash
|
||||
python main.py --config config.yaml
|
||||
```
|
||||
|
||||
### Debug logging
|
||||
|
||||
```bash
|
||||
python main.py --debug
|
||||
```
|
||||
|
||||
### Test browser setup
|
||||
|
||||
```bash
|
||||
python main.py --test-browser
|
||||
python main.py --test-browser --test-url "https://example.com"
|
||||
```
|
||||
|
||||
### Graceful shutdown
|
||||
|
||||
Press `Ctrl+C`. The monitor stops scheduling new checks, closes the browser, and exits cleanly.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser → Snapshot → OpenAI Processor (stub) → Condition Engine → Notifications
|
||||
```
|
||||
|
||||
| Layer | Responsibility |
|
||||
|-------|----------------|
|
||||
| OpenAI (future) | Page understanding, value extraction |
|
||||
| Python | Scheduling, condition evaluation, notifications |
|
||||
| Playwright | Real browser automation |
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
browser_automation/
|
||||
├── main.py
|
||||
├── config.yaml
|
||||
├── monitor/
|
||||
│ ├── browser.py # Browser discovery, launch, navigation
|
||||
│ ├── scheduler.py # Round-robin loop
|
||||
│ ├── snapshot.py # Page capture
|
||||
│ ├── processor.py # OpenAI stub + condition evaluation
|
||||
│ ├── models.py # PageSnapshot, ProcessingResult, etc.
|
||||
│ └── config.py # YAML loading
|
||||
├── notifications/
|
||||
│ ├── base.py # NotificationService abstraction
|
||||
│ ├── sms.py # SMS provider stub
|
||||
│ └── email.py # SMTP / logging email
|
||||
├── storage/
|
||||
│ └── snapshots.py # Screenshot storage + retention
|
||||
├── profiles/ # Persistent browser profiles
|
||||
└── snapshots/ # Captured screenshots
|
||||
```
|
||||
|
||||
## Snapshots
|
||||
|
||||
Screenshots are saved per website:
|
||||
|
||||
```
|
||||
snapshots/
|
||||
website_1/
|
||||
2026-09-30T13-20-00.png
|
||||
website_2/
|
||||
2026-09-30T13-20-05.png
|
||||
```
|
||||
|
||||
Older snapshots are removed automatically when `snapshot_retention_count` is exceeded.
|
||||
|
||||
## Notifications
|
||||
|
||||
By default, notifications are **logged** to the console:
|
||||
|
||||
```
|
||||
SMS WOULD BE SENT:
|
||||
...
|
||||
|
||||
EMAIL WOULD BE SENT:
|
||||
Subject: ...
|
||||
```
|
||||
|
||||
### Email (SMTP)
|
||||
|
||||
Set environment variables in `.env`:
|
||||
|
||||
```
|
||||
SMTP_HOST=smtp.example.com
|
||||
SMTP_PORT=587
|
||||
SMTP_USER=your-user
|
||||
SMTP_PASSWORD=your-password
|
||||
EMAIL_FROM=monitor@example.com
|
||||
EMAIL_TO=you@example.com
|
||||
```
|
||||
|
||||
Enable in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
notifications:
|
||||
email:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### SMS
|
||||
|
||||
Configure provider credentials via environment variables (`SMS_PROVIDER`, `SMS_API_KEY`, etc.). Enable in config when a provider is implemented.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Navigation timeouts, DNS failures, and page errors are logged per website
|
||||
- The monitor continues to the next site
|
||||
- Browser crashes trigger automatic restart with backoff
|
||||
- Bot/automation blocks are logged and skipped — no CAPTCHA or bypass logic
|
||||
|
||||
## Security
|
||||
|
||||
- No credentials in source code or `config.yaml`
|
||||
- Use `.env` for secrets (not committed)
|
||||
- No CAPTCHA bypass, Cloudflare bypass, or fingerprint spoofing
|
||||
|
||||
## Future Extensions
|
||||
|
||||
The architecture supports adding:
|
||||
|
||||
- OpenAI page understanding and condition triggers
|
||||
- Per-site instructions and check intervals
|
||||
- Multiple browser profiles
|
||||
- Alert deduplication and cooldown
|
||||
- Urmi, Twilio, WhatsApp integrations
|
||||
- Dashboard and database storage
|
||||
Loading…
Add table
Add a link
Reference in a new issue