2b39c0aa28
- OCDS API client for eTenders.gov.za (primary tender discovery) - Web UI scraper using Playwright (fetches supportDocument[] array that the OCDS API doesn't expose — only 1 doc per tender in OCDS) - SQLite database with full schema for tenders + documents - Click CLI: pull, stats, list, show commands - 10 unit tests (all passing) covering models, OCDS parsing, web UI parsing, and document enrichment logic - Comprehensive research report: 75+ SA tender sources documented with automation details (API/RSS/scraping availability per source) - Hybrid architecture: OCDS for discovery + web scraping for full document lists (verified: web UI shows 2+ docs, OCDS shows 1)
SA Tender Monitor
Automated South African tender monitoring system that pulls tenders from multiple online sources and makes them available for search, filtering, and notification.
Architecture
Hybrid data pipeline:
- eTenders OCDS API — primary source for tender discovery (titles, dates, status, buyer info). Covers all government, SOE, and public entity tenders.
- eTenders Web UI scraping — supplements the OCDS API by fetching the full
supportDocument[]array for each tender (the OCDS API only returns 1 document per tender; the web UI shows all attached files). - Supplementary sources — RSS feeds from Armscor, Health.gov.za, CIDB, sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE portals not fully captured in the OCDS API.
Key Technical Detail
The eTenders OCDS API (https://ocds-api.etenders.gov.za/api/OCDSReleases)
returns OCDS-compliant JSON with one critical limitation: it only exposes a
single primary advert document per tender in tender.documents[]. Supporting
documents (additional attachments uploaded by the tender issuer) are NOT
in the OCDS API — they only appear in the web UI's internal DataTables
endpoint (/Home/PaginatedTenderOpportunities), which returns each tender
row with a supportDocument[] array containing all attached files.
This system uses a hybrid approach:
- OCDS API for discovery and metadata
- Headless browser scraping for the full document list per tender
Sources
See docs/sources.md for the comprehensive research report covering 75+ SA
tender sources with automation details.
Project Structure
sa-tender-monitor/
├── README.md
├── .gitignore
├── requirements.txt
├── docs/
│ └── sources.md # SA tender sources research report
├── src/
│ ├── __init__.py
│ ├── ocds_client.py # eTenders OCDS API client
│ ├── web_scraper.py # eTenders web UI document scraper
│ ├── models.py # Data models (Tender, Document, Buyer)
│ ├── db.py # SQLite storage
│ ├── pipeline.py # Orchestration: OCDS → scrape → store
│ └── config.py # Configuration
├── data/ # Cached/raw data (gitignored)
└── tests/
└── test_ocds_client.py
Setup
cd /root/workspace/sa-tender-monitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Usage
# Pull tenders from the last 7 days
python -m src.pipeline --days 7
# Pull tenders for a specific date range
python -m src.pipeline --from 2025-06-01 --to 2025-06-30