ddfe4bccba
Replace the JS variable polling approach with Playwright response
interception — listen for DataTables AJAX responses at the network
layer instead of waiting for the JS variable to populate.
- Use 'domcontentloaded' instead of 'networkidle' (site keeps
long-poll connections open, preventing networkidle from firing)
- Capture /PaginatedTenderOpportunities responses via page.on('response')
- Paginate through all pages using table.page('next').draw()
- Verified live: 500 rows fetched, 173 multi-doc tenders found
- Full pipeline tested: 149 OCDS tenders → 141 matched & enriched
with web UI docs → 34 tenders now have >1 document in DB
SA Tender Monitor
Automated South African tender monitoring system that pulls tenders from multiple online sources and makes them available for search, filtering, and notification.
Architecture
Hybrid data pipeline:
- eTenders OCDS API — primary source for tender discovery (titles, dates, status, buyer info). Covers all government, SOE, and public entity tenders.
- eTenders Web UI scraping — supplements the OCDS API by fetching the full
supportDocument[]array for each tender (the OCDS API only returns 1 document per tender; the web UI shows all attached files). - Supplementary sources — RSS feeds from Armscor, Health.gov.za, CIDB, sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE portals not fully captured in the OCDS API.
Key Technical Detail
The eTenders OCDS API (https://ocds-api.etenders.gov.za/api/OCDSReleases)
returns OCDS-compliant JSON with one critical limitation: it only exposes a
single primary advert document per tender in tender.documents[]. Supporting
documents (additional attachments uploaded by the tender issuer) are NOT
in the OCDS API — they only appear in the web UI's internal DataTables
endpoint (/Home/PaginatedTenderOpportunities), which returns each tender
row with a supportDocument[] array containing all attached files.
This system uses a hybrid approach:
- OCDS API for discovery and metadata
- Headless browser scraping for the full document list per tender
Sources
See docs/sources.md for the comprehensive research report covering 75+ SA
tender sources with automation details.
Project Structure
sa-tender-monitor/
├── README.md
├── .gitignore
├── requirements.txt
├── docs/
│ └── sources.md # SA tender sources research report
├── src/
│ ├── __init__.py
│ ├── ocds_client.py # eTenders OCDS API client
│ ├── web_scraper.py # eTenders web UI document scraper
│ ├── models.py # Data models (Tender, Document, Buyer)
│ ├── db.py # SQLite storage
│ ├── pipeline.py # Orchestration: OCDS → scrape → store
│ └── config.py # Configuration
├── data/ # Cached/raw data (gitignored)
└── tests/
└── test_ocds_client.py
Setup
cd /root/workspace/sa-tender-monitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Usage
# Pull tenders from the last 7 days
python -m src.pipeline --days 7
# Pull tenders for a specific date range
python -m src.pipeline --from 2025-06-01 --to 2025-06-30