# SA Tender Monitor

Automated South African tender monitoring system that pulls tenders from
multiple online sources and makes them available for search, filtering,
and notification.

## Architecture

Hybrid data pipeline:

1. **eTenders OCDS API** — primary source for tender discovery (titles, dates,
   status, buyer info). Covers all government, SOE, and public entity tenders.
2. **eTenders Web UI scraping** — supplements the OCDS API by fetching the full
   `supportDocument[]` array for each tender (the OCDS API only returns 1
   document per tender; the web UI shows all attached files).
3. **Supplementary sources** — RSS feeds from Armscor, Health.gov.za, CIDB,
   sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE
   portals not fully captured in the OCDS API.

## Key Technical Detail

The eTenders OCDS API (`https://ocds-api.etenders.gov.za/api/OCDSReleases`)
returns OCDS-compliant JSON with one critical limitation: it only exposes a
single primary advert document per tender in `tender.documents[]`. Supporting
documents (additional attachments uploaded by the tender issuer) are NOT
in the OCDS API — they only appear in the web UI's internal DataTables
endpoint (`/Home/PaginatedTenderOpportunities`), which returns each tender
row with a `supportDocument[]` array containing all attached files.

This system uses a **hybrid approach**:
- OCDS API for discovery and metadata
- Headless browser scraping for the full document list per tender

## Sources

See `docs/sources.md` for the comprehensive research report covering 75+ SA
tender sources with automation details.

## Project Structure

```
sa-tender-monitor/
├── README.md
├── .gitignore
├── requirements.txt
├── docs/
│   └── sources.md          # SA tender sources research report
├── src/
│   ├── __init__.py
│   ├── ocds_client.py      # eTenders OCDS API client
│   ├── web_scraper.py      # eTenders web UI document scraper
│   ├── models.py            # Data models (Tender, Document, Buyer)
│   ├── db.py                # SQLite storage
│   ├── pipeline.py          # Orchestration: OCDS → scrape → store
│   └── config.py           # Configuration
├── data/                   # Cached/raw data (gitignored)
└── tests/
    └── test_ocds_client.py
```

## Setup

```bash
cd /root/workspace/sa-tender-monitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

## Usage

```bash
# Pull tenders from the last 7 days
python -m src.pipeline --days 7

# Pull tenders for a specific date range
python -m src.pipeline --from 2025-06-01 --to 2025-06-30
```