Files

jjsm01 2b39c0aa28 feat: initial SA Tender Monitor project

- OCDS API client for eTenders.gov.za (primary tender discovery)
- Web UI scraper using Playwright (fetches supportDocument[] array
  that the OCDS API doesn't expose — only 1 doc per tender in OCDS)
- SQLite database with full schema for tenders + documents
- Click CLI: pull, stats, list, show commands
- 10 unit tests (all passing) covering models, OCDS parsing,
  web UI parsing, and document enrichment logic
- Comprehensive research report: 75+ SA tender sources documented
  with automation details (API/RSS/scraping availability per source)
- Hybrid architecture: OCDS for discovery + web scraping for full
  document lists (verified: web UI shows 2+ docs, OCDS shows 1)

2026-07-03 09:09:39 +00:00

2.6 KiB

Raw Permalink Blame History

SA Tender Monitor

Automated South African tender monitoring system that pulls tenders from multiple online sources and makes them available for search, filtering, and notification.

Architecture

Hybrid data pipeline:

eTenders OCDS API — primary source for tender discovery (titles, dates, status, buyer info). Covers all government, SOE, and public entity tenders.
eTenders Web UI scraping — supplements the OCDS API by fetching the full supportDocument[] array for each tender (the OCDS API only returns 1 document per tender; the web UI shows all attached files).
Supplementary sources — RSS feeds from Armscor, Health.gov.za, CIDB, sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE portals not fully captured in the OCDS API.

Key Technical Detail

The eTenders OCDS API (https://ocds-api.etenders.gov.za/api/OCDSReleases) returns OCDS-compliant JSON with one critical limitation: it only exposes a single primary advert document per tender in tender.documents[]. Supporting documents (additional attachments uploaded by the tender issuer) are NOT in the OCDS API — they only appear in the web UI's internal DataTables endpoint (/Home/PaginatedTenderOpportunities), which returns each tender row with a supportDocument[] array containing all attached files.

This system uses a hybrid approach:

OCDS API for discovery and metadata
Headless browser scraping for the full document list per tender

Sources

See docs/sources.md for the comprehensive research report covering 75+ SA tender sources with automation details.

Project Structure

sa-tender-monitor/
├── README.md
├── .gitignore
├── requirements.txt
├── docs/
│   └── sources.md          # SA tender sources research report
├── src/
│   ├── __init__.py
│   ├── ocds_client.py      # eTenders OCDS API client
│   ├── web_scraper.py      # eTenders web UI document scraper
│   ├── models.py            # Data models (Tender, Document, Buyer)
│   ├── db.py                # SQLite storage
│   ├── pipeline.py          # Orchestration: OCDS → scrape → store
│   └── config.py           # Configuration
├── data/                   # Cached/raw data (gitignored)
└── tests/
    └── test_ocds_client.py

Setup

cd /root/workspace/sa-tender-monitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Usage

# Pull tenders from the last 7 days
python -m src.pipeline --days 7

# Pull tenders for a specific date range
python -m src.pipeline --from 2025-06-01 --to 2025-06-30

2.6 KiB Raw Permalink Blame History