2b39c0aa28
- OCDS API client for eTenders.gov.za (primary tender discovery) - Web UI scraper using Playwright (fetches supportDocument[] array that the OCDS API doesn't expose — only 1 doc per tender in OCDS) - SQLite database with full schema for tenders + documents - Click CLI: pull, stats, list, show commands - 10 unit tests (all passing) covering models, OCDS parsing, web UI parsing, and document enrichment logic - Comprehensive research report: 75+ SA tender sources documented with automation details (API/RSS/scraping availability per source) - Hybrid architecture: OCDS for discovery + web scraping for full document lists (verified: web UI shows 2+ docs, OCDS shows 1)
78 lines
2.6 KiB
Markdown
78 lines
2.6 KiB
Markdown
# SA Tender Monitor
|
|
|
|
Automated South African tender monitoring system that pulls tenders from
|
|
multiple online sources and makes them available for search, filtering,
|
|
and notification.
|
|
|
|
## Architecture
|
|
|
|
Hybrid data pipeline:
|
|
|
|
1. **eTenders OCDS API** — primary source for tender discovery (titles, dates,
|
|
status, buyer info). Covers all government, SOE, and public entity tenders.
|
|
2. **eTenders Web UI scraping** — supplements the OCDS API by fetching the full
|
|
`supportDocument[]` array for each tender (the OCDS API only returns 1
|
|
document per tender; the web UI shows all attached files).
|
|
3. **Supplementary sources** — RSS feeds from Armscor, Health.gov.za, CIDB,
|
|
sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE
|
|
portals not fully captured in the OCDS API.
|
|
|
|
## Key Technical Detail
|
|
|
|
The eTenders OCDS API (`https://ocds-api.etenders.gov.za/api/OCDSReleases`)
|
|
returns OCDS-compliant JSON with one critical limitation: it only exposes a
|
|
single primary advert document per tender in `tender.documents[]`. Supporting
|
|
documents (additional attachments uploaded by the tender issuer) are NOT
|
|
in the OCDS API — they only appear in the web UI's internal DataTables
|
|
endpoint (`/Home/PaginatedTenderOpportunities`), which returns each tender
|
|
row with a `supportDocument[]` array containing all attached files.
|
|
|
|
This system uses a **hybrid approach**:
|
|
- OCDS API for discovery and metadata
|
|
- Headless browser scraping for the full document list per tender
|
|
|
|
## Sources
|
|
|
|
See `docs/sources.md` for the comprehensive research report covering 75+ SA
|
|
tender sources with automation details.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
sa-tender-monitor/
|
|
├── README.md
|
|
├── .gitignore
|
|
├── requirements.txt
|
|
├── docs/
|
|
│ └── sources.md # SA tender sources research report
|
|
├── src/
|
|
│ ├── __init__.py
|
|
│ ├── ocds_client.py # eTenders OCDS API client
|
|
│ ├── web_scraper.py # eTenders web UI document scraper
|
|
│ ├── models.py # Data models (Tender, Document, Buyer)
|
|
│ ├── db.py # SQLite storage
|
|
│ ├── pipeline.py # Orchestration: OCDS → scrape → store
|
|
│ └── config.py # Configuration
|
|
├── data/ # Cached/raw data (gitignored)
|
|
└── tests/
|
|
└── test_ocds_client.py
|
|
```
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cd /root/workspace/sa-tender-monitor
|
|
python3 -m venv venv
|
|
source venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Pull tenders from the last 7 days
|
|
python -m src.pipeline --days 7
|
|
|
|
# Pull tenders for a specific date range
|
|
python -m src.pipeline --from 2025-06-01 --to 2025-06-30
|
|
``` |