feat: initial SA Tender Monitor project
- OCDS API client for eTenders.gov.za (primary tender discovery) - Web UI scraper using Playwright (fetches supportDocument[] array that the OCDS API doesn't expose — only 1 doc per tender in OCDS) - SQLite database with full schema for tenders + documents - Click CLI: pull, stats, list, show commands - 10 unit tests (all passing) covering models, OCDS parsing, web UI parsing, and document enrichment logic - Comprehensive research report: 75+ SA tender sources documented with automation details (API/RSS/scraping availability per source) - Hybrid architecture: OCDS for discovery + web scraping for full document lists (verified: web UI shows 2+ docs, OCDS shows 1)
This commit is contained in:
78
README.md
Normal file
78
README.md
Normal file
@@ -0,0 +1,78 @@
|
||||
# SA Tender Monitor
|
||||
|
||||
Automated South African tender monitoring system that pulls tenders from
|
||||
multiple online sources and makes them available for search, filtering,
|
||||
and notification.
|
||||
|
||||
## Architecture
|
||||
|
||||
Hybrid data pipeline:
|
||||
|
||||
1. **eTenders OCDS API** — primary source for tender discovery (titles, dates,
|
||||
status, buyer info). Covers all government, SOE, and public entity tenders.
|
||||
2. **eTenders Web UI scraping** — supplements the OCDS API by fetching the full
|
||||
`supportDocument[]` array for each tender (the OCDS API only returns 1
|
||||
document per tender; the web UI shows all attached files).
|
||||
3. **Supplementary sources** — RSS feeds from Armscor, Health.gov.za, CIDB,
|
||||
sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE
|
||||
portals not fully captured in the OCDS API.
|
||||
|
||||
## Key Technical Detail
|
||||
|
||||
The eTenders OCDS API (`https://ocds-api.etenders.gov.za/api/OCDSReleases`)
|
||||
returns OCDS-compliant JSON with one critical limitation: it only exposes a
|
||||
single primary advert document per tender in `tender.documents[]`. Supporting
|
||||
documents (additional attachments uploaded by the tender issuer) are NOT
|
||||
in the OCDS API — they only appear in the web UI's internal DataTables
|
||||
endpoint (`/Home/PaginatedTenderOpportunities`), which returns each tender
|
||||
row with a `supportDocument[]` array containing all attached files.
|
||||
|
||||
This system uses a **hybrid approach**:
|
||||
- OCDS API for discovery and metadata
|
||||
- Headless browser scraping for the full document list per tender
|
||||
|
||||
## Sources
|
||||
|
||||
See `docs/sources.md` for the comprehensive research report covering 75+ SA
|
||||
tender sources with automation details.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
sa-tender-monitor/
|
||||
├── README.md
|
||||
├── .gitignore
|
||||
├── requirements.txt
|
||||
├── docs/
|
||||
│ └── sources.md # SA tender sources research report
|
||||
├── src/
|
||||
│ ├── __init__.py
|
||||
│ ├── ocds_client.py # eTenders OCDS API client
|
||||
│ ├── web_scraper.py # eTenders web UI document scraper
|
||||
│ ├── models.py # Data models (Tender, Document, Buyer)
|
||||
│ ├── db.py # SQLite storage
|
||||
│ ├── pipeline.py # Orchestration: OCDS → scrape → store
|
||||
│ └── config.py # Configuration
|
||||
├── data/ # Cached/raw data (gitignored)
|
||||
└── tests/
|
||||
└── test_ocds_client.py
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
cd /root/workspace/sa-tender-monitor
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Pull tenders from the last 7 days
|
||||
python -m src.pipeline --days 7
|
||||
|
||||
# Pull tenders for a specific date range
|
||||
python -m src.pipeline --from 2025-06-01 --to 2025-06-30
|
||||
```
|
||||
Reference in New Issue
Block a user