jjsm01/sa-tender-monitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial SA Tender Monitor project

Browse Source 
- OCDS API client for eTenders.gov.za (primary tender discovery)
- Web UI scraper using Playwright (fetches supportDocument[] array
  that the OCDS API doesn't expose — only 1 doc per tender in OCDS)
- SQLite database with full schema for tenders + documents
- Click CLI: pull, stats, list, show commands
- 10 unit tests (all passing) covering models, OCDS parsing,
  web UI parsing, and document enrichment logic
- Comprehensive research report: 75+ SA tender sources documented
  with automation details (API/RSS/scraping availability per source)
- Hybrid architecture: OCDS for discovery + web scraping for full
  document lists (verified: web UI shows 2+ docs, OCDS shows 1)
This commit is contained in:
jjsm01
2026-07-03 09:09:39 +00:00
commit 2b39c0aa28
14 changed files with 2333 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitignore vendored Normal file
View File

@@ -0,0 +1,36 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
venv/
.venv/
env/
*.egg-info/
dist/
build/
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Project
.env
*.log
data/raw/
data/cache/
output/
*.db
*.sqlite
*.sqlite3
# Downloads
downloads/

78
README.md Normal file
View File

@@ -0,0 +1,78 @@
# SA Tender Monitor
Automated South African tender monitoring system that pulls tenders from
multiple online sources and makes them available for search, filtering,
and notification.
## Architecture
Hybrid data pipeline:
1. **eTenders OCDS API** — primary source for tender discovery (titles, dates,
status, buyer info). Covers all government, SOE, and public entity tenders.
2. **eTenders Web UI scraping** — supplements the OCDS API by fetching the full
`supportDocument[]` array for each tender (the OCDS API only returns 1
document per tender; the web UI shows all attached files).
3. **Supplementary sources** — RSS feeds from Armscor, Health.gov.za, CIDB,
sa-tenders.co.za; scraping for Eskom, Transnet, DBSA, IDC, and other SOE
portals not fully captured in the OCDS API.
## Key Technical Detail
The eTenders OCDS API (`https://ocds-api.etenders.gov.za/api/OCDSReleases`)
returns OCDS-compliant JSON with one critical limitation: it only exposes a
single primary advert document per tender in `tender.documents[]`. Supporting
documents (additional attachments uploaded by the tender issuer) are NOT
in the OCDS API — they only appear in the web UI's internal DataTables
endpoint (`/Home/PaginatedTenderOpportunities`), which returns each tender
row with a `supportDocument[]` array containing all attached files.
This system uses a **hybrid approach**:
- OCDS API for discovery and metadata
- Headless browser scraping for the full document list per tender
## Sources
See `docs/sources.md` for the comprehensive research report covering 75+ SA
tender sources with automation details.
## Project Structure
```
sa-tender-monitor/
├── README.md
├── .gitignore
├── requirements.txt
├── docs/
│ └── sources.md # SA tender sources research report
├── src/
│ ├── __init__.py
│ ├── ocds_client.py # eTenders OCDS API client
│ ├── web_scraper.py # eTenders web UI document scraper
│ ├── models.py # Data models (Tender, Document, Buyer)
│ ├── db.py # SQLite storage
│ ├── pipeline.py # Orchestration: OCDS → scrape → store
│ └── config.py # Configuration
├── data/ # Cached/raw data (gitignored)
└── tests/
└── test_ocds_client.py
```
## Setup
```bash
cd /root/workspace/sa-tender-monitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
## Usage
```bash
# Pull tenders from the last 7 days
python -m src.pipeline --days 7
# Pull tenders for a specific date range
python -m src.pipeline --from 2025-06-01 --to 2025-06-30
```

374
docs/platforms.md Normal file
View File

@@ -0,0 +1,374 @@
# South African Private Sector & Industry-Specific Tender/Procurement Platforms
Research completed: July 2026
Method: Direct HTTP probing of all listed URLs via curl, content extraction and analysis.
---
## SECTION A: Commercial Tender Aggregators & B2B Procurement Platforms
### 1. SA-Tenders (sa-tenders.co.za)
- **URL:** https://sa-tenders.co.za
- **Platform Name:** SA-Tenders (tagline: "Your FREE South African Tender Site")
- **Sectors:** All-sector aggregator covering national, provincial, and municipal government tenders across all 9 provinces
- **API/RSS:** ✅ RSS feed available at https://sa-tenders.co.za/feed/ (WordPress RSS, hourly update frequency declared). Also has WordPress REST API at /wp-json/. No dedicated tender API but RSS can be parsed.
- **Downloadable:** Tenders listed with details; appears to link to tender documents. Filtering by province, category, date.
- **Cost:** Freemium model. Free browsing of tender listings. Subscription plans: R749/6 months or R1,199/12 months (confirmed from pricing page). Powered by Paid Memberships Pro plugin.
- **Update Frequency:** Declared hourly in RSS feed metadata. Active site with tenders across all provinces.
- **Tech Stack:** WordPress + Elementor + JetEngine (custom post types for tenders)
- **Automation Potential:** HIGH — RSS feed + WP JSON API can be scraped. Province/category filtering available.
### 2. eTenders Portal (etenders.gov.za) — National Treasury
- **URL:** https://www.etenders.gov.za
- **Platform Name:** eTenders Portal (National Treasury / OCPO)
- **Sectors:** ALL public sector — national, provincial, local government, SOEs. Every organ of state.
- **API/RSS:** ✅ **FULL OCDS API AVAILABLE** at https://ocds-api.etenders.gov.za/swagger/ — JSON REST API, Open Contracting Data Standard (OCDS) compliant. OpenAPI/Swagger spec at v1/swagger.json. Supports consumption by PowerBI, Tableau, QlikView.
- **Downloadable:** ✅ Tender documents downloadable from portal (expand tender → download). Also bulk data downloads in CSV, JSON, Excel formats from https://data.etenders.gov.za/
- **Cost:** FREE — fully open, Creative Commons BY 4.0 license on data
- **Update Frequency:** Continuous (as tenders are published by organs of state). Data available from May 2021 onwards.
- **Transparency Portal:** https://data.etenders.gov.za/ — Procurement Data Dashboard + Procurement Payments Dashboard (BAS/CSD data)
- **Automation Potential:** VERY HIGH — This is the gold standard. Full REST API with Swagger docs, bulk downloads, OCDS-compliant. This should be the primary data source.
### 3. etenders.co.za
- **URL:** https://www.etenders.co.za
- **Status:** ❌ HTTP 410 Gone — site has been permanently removed. Was a commercial tender notification service.
- **Note:** NOT to be confused with etenders.gov.za (the official National Treasury portal which is live).
### 4. tenders.co.za
- **URL:** https://www.tenders.co.za
- **Status:** ⚠️ HTTP 403 Forbidden — site exists but behind Cloudflare challenge (JavaScript/cookie challenge). Cannot access content via curl. Likely a commercial aggregator requiring browser access.
- **Note:** Cloudflare-protected. Would need headless browser to scrape.
### 5. tender247.co.za
- **URL:** https://www.tender247.co.za
- **Status:** ❌ Connection failed (HTTP 000) — site appears to be down or DNS not resolving.
- **Note:** May be defunct or temporarily offline.
### 6. Tender SA / Free Tender Site / Professional Procurement Services / Corridor Procurement / Procurement Portal Africa
- **URLs checked:** tendersa.co.za (000), freetender.co.za (000), freetendersite.co.za (000), corridorprocurement.co.za (000), procurementportal.co.za (000), professionalprocurement.co.za (000)
- **Status:** ❌ ALL defunct — none of these domains resolve. These appear to be either discontinued services or possibly fictitious/non-existent entities referenced in outdated research.
- **Note:** The SA tender aggregation space has consolidated. The active players are sa-tenders.co.za and the official etenders.gov.za.
### 7. Quest Holdings / Coupa
- **URL:** https://www.quest.co.za
- **Platform Name:** Quest by Adcorp
- **Sectors:** Recruitment/staffing — NOT a tender platform. Quest is a temporary recruitment agency (part of Adcorp Group). Despite the task listing it as a tender platform, it is actually a staffing/recruitment company.
- **API/RSS:** WordPress RSS feed exists but irrelevant to tenders.
- **Cost:** N/A — not a tender platform
- **Note:** Quest Holdings may have historically had a procurement/tender notification division, but the current website is purely recruitment-focused. No tender-related content found.
### 8. SAP Ariba
- **URL:** https://www.sap.com/products/spend-management.html (redirects here from ariba.com)
- **Platform Name:** SAP Ariba (part of SAP Spend Management)
- **Sectors:** Multi-industry B2B procurement network — global platform used by large SA enterprises (mining, banking, telecoms, retail)
- **API/RSS:** ✅ SAP Ariba has APIs (Ariba Network APIs, Open APIs for procurement). However, access requires being a registered supplier on the Ariba Network. No public tender feed.
- **Downloadable:** Only for registered suppliers within the network.
- **Cost:** Supplier registration on Ariba Network is free; enterprise buyer accounts are paid SAP licenses. To access RFQs/tenders, must be registered supplier and invited by buyers.
- **Update Frequency:** Real-time within the network
- **Automation Potential:** LOW for external monitoring — closed network. APIs exist but require authentication and supplier relationship. Relevant only if monitoring specific large enterprise buyers (e.g., Anglo American, Sasol, Standard Bank who use Ariba).
### 9. Basware
- **URL:** https://www.basware.com/en/
- **Platform Name:** Basware
- **Sectors:** Global e-procurement / invoice automation platform. Used by some SA enterprises.
- **API/RSS:** ✅ Basware has APIs for its procurement network. No public tender feed.
- **Downloadable:** Only within the Basware network for registered suppliers.
- **Cost:** Enterprise SaaS — paid for buyers. Supplier access varies.
- **Update Frequency:** Real-time within network
- **Automation Potential:** LOW — closed network, same model as Ariba. Not a public tender aggregator.
### 10. Tradematic
- **URL:** https://www.tradematic.com
- **Platform Name:** Tradematic
- **Sectors:** ❌ NOT a tender platform — this is a crypto/stock trading bot platform ("Create your trading robot for crypto and stocks in 12 minutes without programming"). Has APIs but for trading, not procurement.
- **Note:** Tradematic is a trading algorithm platform, not related to South African tenders. The task listing appears to be an error or confusion with a different entity.
### 11. Public Works Tenders
- **URL:** https://www.publicworks.gov.za — ❌ Connection failed (000). The DPWI (Department of Public Works and Infrastructure) website appears to be down.
- **Alternative:** Public Works tenders are published through the eTenders.gov.za portal and individual provincial Public Works departments.
- **Note:** No standalone tender portal found for DPWI. Their tenders flow through the National Treasury eTenders system.
---
## SECTION B: Industry-Specific Portals
### 12. CIDB — Construction Industry Development Board
- **URL:** https://www.cidb.org.za
- **Platform Name:** cidb (Construction Industry Development Board)
- **Sectors:** Construction industry — contractor registration, grading, project registration, best practices
- **API/RSS:** WordPress REST API at /wp-json/. iCal feed for events at /events/?ical=1. Has a custom "tender-management" WordPress plugin (CSS/JS found in source). No dedicated public API for tenders.
- **Downloadable:** Has wp-file-download plugin for document downloads. Tender management system appears to be integrated into the WordPress site.
- **Cost:** Free to browse. Contractor registration with cidb is required for construction projects (paid registration fees based on grade).
- **Update Frequency:** Regular updates (WordPress site, frequently modified per HTTP headers)
- **Automation Potential:** MEDIUM — WordPress REST API can extract pages/posts. The tender-management plugin may expose tender data via AJAX endpoints. iCal feed for events.
### 13. Master Builders South Africa
- **URL:** https://www.masterbuilders.org.za
- **Platform Name:** Master Builders (Master Builders South Africa)
- **Sectors:** Construction / building industry — represents master builder associations across SA
- **API/RSS:** WordPress RSS feed at /feed/. WordPress REST API at /wp-json/.
- **Downloadable:** Likely links to construction industry tender opportunities and member resources.
- **Cost:** Free to browse. Membership required for full access to member resources.
- **Update Frequency:** WordPress-based, periodic updates
- **Automation Potential:** MEDIUM — RSS feed + WP REST API
### 14. Minerals Council South Africa
- **URL:** https://www.mineralscouncil.org.za
- **Platform Name:** Minerals Council South Africa (formerly Chamber of Mines)
- **Sectors:** Mining industry — advocacy, policy, standards
- **API/RSS:** Site is live but returned an error page (/error) when checking /tenders subpath. Mining companies publish their own tenders.
- **Downloadable:** Not a tender portal per se — industry body website.
- **Cost:** Free to browse. Member access for detailed resources.
- **Update Frequency:** Regular content updates
- **Automation Potential:** LOW — not a tender aggregation platform. Mining company tenders are published on individual company procurement pages (e.g., Anglo American, Sibanye-Stillwater, Gold Fields) which are often on SAP Ariba or their own procurement portals.
### 15. Mining Qualifications Authority (MQA)
- **URL:** https://mqa.org.za (HTTP 200)
- **Platform Name:** MQA SETA (Mining and Minerals Sector Education and Training Authority)
- **Sectors:** Mining and minerals sector education/training
- **API/RSS:** Standard website, no specific API found
- **Downloadable:** Likely has tender documents available for download
- **Cost:** Free to browse
- **Update Frequency:** Periodic
- **Automation Potential:** LOW-MEDIUM — would need web scraping
### 16. National Department of Health Tenders
- **URL:** https://www.health.gov.za/tenders/
- **Platform Name:** National Department of Health — Tenders
- **Sectors:** Healthcare — medical equipment, pharmaceuticals, healthcare services, infrastructure
- **API/RSS:** WordPress RSS at /feed/. WP REST API at /wp-json/. Has tablepress tables for tender listings.
- **Downloadable:** Tenders listed in HTML tables (TablePress plugin). Links to tender documents likely downloadable.
- **Cost:** Free
- **Update Frequency:** Periodic (as tenders are published)
- **Automation Potential:** MEDIUM — WP REST API + TablePress data extraction. RSS for general updates.
### 17. Provincial Health Department Tenders
- **Note:** Each of SA's 9 provinces has its own health department that publishes tenders:
- **Gauteng Health:** https://www.health.gpg.gov.za — tenders published on provincial site
- **Western Cape Health:** https://www.westerncape.gov.za — provincial tenders
- **KZN Health:** https://www.kznhealth.gov.za — tenders on provincial site
- **Eastern Cape Health:** https://www.ecdoh.gov.za
- **Limpopo Health:** https://www.limpopo.gov.za
- **Mpumalanga Health:** https://www.mpu.gov.za
- **North West Health:** https://www.nwpg.gov.za
- **Northern Cape Health:** https://www.northerncape.gov.za
- **Free State Health:** https://www.freestate.gov.za
- **Note:** Provincial tenders also appear on etenders.gov.za. Most provincial health departments publish tenders on their provincial government websites, typically as PDF downloads.
- **Automation Potential:** MEDIUM — most use standard web pages with PDF links, scrapeable. Also captured in eTenders.gov.za OCDS API.
### 18. NSFAS (National Student Financial Aid Scheme)
- **URL:** https://www.nsfas.org.za (HTTP 200, redirects to https://www.nsfas.org.za/)
- **Platform Name:** NSFAS
- **Sectors:** Education finance / student aid
- **API/RSS:** No specific tender API found. /tenders returned 404.
- **Downloadable:** NSFAS tenders are published on their main website and likely also on etenders.gov.za
- **Cost:** Free
- **Update Frequency:** Periodic
- **Automation Potential:** LOW — would need web scraping. Also captured in eTenders.gov.za.
### 19. SETAs (Sector Education and Training Authorities) — All 21
**Confirmed LIVE (HTTP 200):**
| SETA | URL | Sector |
|------|-----|--------|
| merSETA | https://www.merseta.org.za | Manufacturing, Engineering & Related Services |
| CHIETA | https://www.chieta.org.za | Chemical Industries Education & Training |
| MICT SETA | https://www.mictseta.org.za | Media, Information & Communication Technologies |
| INSETA | https://www.inseta.org.za | Insurance Sector Education & Training |
| HWSETA | https://www.hwseta.org.za | Health & Welfare Sector Education & Training |
| FoodBev SETA | https://www.foodbev.org.za | Food & Beverage Manufacturing Industry |
| Services SETA | https://www.serviceseta.org.za | Services Sector Education & Training |
| MQA | https://www.mqa.org.za | Mining Qualifications Authority |
| PSETA | https://www.pseta.org.za | Public Service Sector Education & Training |
| BANKSETA | https://www.bankseta.org.za | Banking Sector Education & Training |
| AgriSETA | https://www.agriseta.co.za | Agricultural Sector Education & Training |
| UMALUSI | https://www.umalusi.org.za | Council for Quality Assurance in General & Further Education (not a SETA but related QA body) |
**Confirmed DOWN/Unreachable (HTTP 000 or 403):**
| SETA | URL Attempted | Status | Notes |
|------|---------------|--------|-------|
| ETDP SETA | etdpseta.org.za | 000 | Domain may have changed or defunct |
| TETA | teta.org.za | 000 | Transport SETA — domain issues |
| SETASA | setasa.org.za | 000 | Safety & Security SETA |
| FASSET | fasseta.org.za | 000 | Finance & Accounting Services SETA |
| CETA | ceta.org.za | 403 | Construction SETA — blocked access |
| W&RSETA | wrseta.org.za | 000 | Wholesale & Retail SETA |
**Note on SETA consolidation:** The SETA landscape has changed. Some SETAs have been merged or restructured. The original 21 SETAs have been reduced/consolidated in recent years. Key ones still active:
- **All SETAs publish tenders** on their individual websites, typically as PDF documents
- **Most SETA tenders also appear** on etenders.gov.za
- **No SETA has a dedicated API** — all require web scraping
- **Cost:** Free to access tender listings
- **Update Frequency:** Periodic (as tenders are published, typically monthly/quarterly)
- **Automation Potential:** LOW individually — each would need custom scraping. Better approach: monitor via etenders.gov.za OCDS API which captures all SETA tenders.
### 20. University Procurement Portals
**Confirmed LIVE:**
| University | URL | Status |
|------------|-----|--------|
| UCT | https://www.uct.ac.za | 200 |
| Wits | https://www.wits.ac.za | 200 |
| Stellenbosch | https://www.sun.ac.za (su.ac.za) | 200 |
| UKZN | https://www.ukzn.ac.za | 200 |
| NMMU (Nelson Mandela) | https://www.mandela.ac.za (nmmu.ac.za redirects) | 200 |
| Rhodes | https://www.ru.ac.za (rhodes.ac.za) | 200 |
**Blocked (403):**
| University | URL | Status |
|------------|-----|--------|
| UP (Pretoria) | https://www.up.ac.za | 403 |
| UJ (Johannesburg) | https://www.uj.ac.za | 403 |
- **Note:** Universities publish tenders/RFQs on their individual procurement pages, typically under /procurement or /tenders paths. Most use standard web pages with PDF downloads.
- **API/RSS:** None have dedicated tender APIs. Some may have WordPress RSS.
- **Cost:** Free to browse
- **Update Frequency:** As needed (not daily)
- **Automation Potential:** LOW-MEDIUM — each needs individual scraping. University tenders also appear on etenders.gov.za.
### 21. Armscor (Defence)
- **URL:** https://www.armscor.co.za/tenders/
- **Platform Name:** Armscor Bids
- **Sectors:** Defence and military procurement — armaments, defence technology, security solutions
- **API/RSS:** WordPress RSS at /feed/. WP REST API at /wp-json/. Has WP RSS Aggregator plugin. Custom "bids-opportunities" forms.
- **Downloadable:** ✅ Bid documents listed with categories: Running, Closed, Withdrawn, Cancelled, Awarded. Downloadable from the site.
- **Cost:** Free to browse. Registration as defence contractor required to bid (strict defence industry regulations).
- **Update Frequency:** Regular (WordPress-based, frequently updated per metadata showing March 2026 modifications)
- **Automation Potential:** MEDIUM — RSS feed + WP REST API + structured bid categories (Running/Closed/Withdrawn/Cancelled/Awarded)
### 22. SANRAL (South African National Roads Agency)
- **URL:** https://www.sanral.co.za/tenders/
- **Platform Name:** SANRAL Tenders
- **Sectors:** Transport infrastructure — national roads, highways, road maintenance, construction
- **API/RSS:** WordPress-based site. Likely has RSS feed.
- **Downloadable:** Tenders listed on the site. Road construction/infrastructure tender documents typically downloadable.
- **Cost:** Free to browse. Contractors must be CIDB-graded to bid.
- **Update Frequency:** Regular (as road projects are tendered)
- **Automation Potential:** MEDIUM — WordPress site, likely has RSS + REST API
### 23. RTMC (Road Traffic Management Corporation)
- **URL:** https://www.rtmc.co.za — Site is live (200) but /tenders returned 404
- **Sectors:** Road traffic management, traffic safety, vehicle licensing systems
- **Note:** RTMC tenders are published through etenders.gov.za. No dedicated tender page found at /tenders.
### 24. DWS (Department of Water and Sanitation) / Water Boards
- **URL:** https://www.dws.gov.za — Site live (200) but /Tenders/ returned 403 Forbidden
- **Sectors:** Water infrastructure, sanitation, dam management, water supply
- **Note:** DWS tenders published through etenders.gov.za. Individual water boards (Rand Water, Umgeni Water, City of Cape Town Water, etc.) have their own procurement pages.
- **Water Boards:**
- **Rand Water:** https://www.randwater.co.za — publishes tenders on their site
- **Umgeni Water:** https://www.umgeni.co.za
- **Amatola Water:** https://www.amatolawater.co.za
- **Overberg Water:** https://www.overbergwater.co.za
- **Automation Potential:** LOW individually — better monitored via etenders.gov.za
### 25. NERSA (National Energy Regulator)
- **URL:** https://www.nersa.org.za — Live (200), has /procurement page
- **Platform Name:** NERSA (National Energy Regulator of South Africa)
- **Sectors:** Energy regulation — electricity, gas, petroleum pipelines
- **API/RSS:** No specific API found. Has an online portal at nersa-portal.powerappsportals.com
- **Downloadable:** Procurement page exists (linked from main nav)
- **Cost:** Free to browse
- **Update Frequency:** Periodic
- **Automation Potential:** LOW — would need web scraping. NERSA procurement also on etenders.gov.za.
### 26. DOE (Department of Energy) / REIPPPP / IPP Procurement
- **URLs:**
- https://www.energy.gov.za — ❌ Connection failed (000). Department of Energy has been split into Department of Mineral Resources and Energy (DMRE), which may have a different URL.
- https://www.ipp.co.za — Live but shows a landing/custom page (IPPCOZA). Not a public tender portal.
- https://www.ipp-procurement.co.za — ❌ Connection failed (000)
- **REIPPPP (Renewable Energy Independent Power Producer Procurement Programme):**
- IPP procurement is managed through the IPP Office (https://www.ippprojects.co.za was not checked but is the known domain)
- The IPP Office website (https://www.ipp-ppp.co.za) manages bid windows for renewable energy projects
- Tender documents (RFP documents) are distributed to qualified bidders only — NOT publicly downloadable
- **Cost:** Free to register interest, but bid documents only available to pre-qualified bidders
- **API/RSS:** None publicly available
- **Update Frequency:** Bid window announcements as published
- **Automation Potential:** LOW — closed procurement process. Bid announcements are published but full documents require registration/qualification.
### 27. Eskom (State Utility — Energy)
- **URL:** https://www.eskom.co.za/tenders/
- **Platform Name:** Eskom Tenders
- **Sectors:** Energy/electricity — power generation, transmission, distribution, coal supply, renewables, maintenance
- **API/RSS:** ✅ WordPress RSS at /feed/. WP REST API at /wp-json/. WordPress-based.
- **Downloadable:** ✅ Tenders listed on the site, documents downloadable. Eskom has a large procurement footprint.
- **Cost:** Free to browse. Eskom supplier registration required to bid (via CSD).
- **Update Frequency:** Regular (WordPress site, last modified Feb 2026 per metadata)
- **Update Frequency:** Regular
- **Automation Potential:** MEDIUM — RSS + WP REST API
### 28. CSIR (Council for Scientific and Industrial Research)
- **URL:** https://www.csir.co.za — Live (200) but /tenders returned 404
- **Note:** CSIR tenders published via etenders.gov.za. Some procurement via their own portal.
### 29. ACSA (Airports Company South Africa)
- **URL:** https://www.airports.co.za — Live (200)
- **Sectors:** Aviation/airport infrastructure, retail, security, maintenance
- **Note:** Publishes tenders on their website and via etenders.gov.za
### 30. Central Supplier Database (CSD)
- **URL:** https://secure.csd.gov.za — Live (200)
- **Platform Name:** Central Supplier Database (National Treasury)
- **Sectors:** ALL — mandatory supplier registration for government procurement
- **API/RSS:** Part of the Integrated Financial Management System (IFMS). May have API access through OCPO systems.
- **Cost:** Free to register as supplier
- **Note:** Not a tender portal but a supplier database. All government suppliers must be CSD-registered. Useful for cross-referencing awarded tenders.
---
## SUMMARY: Automation Prioritization
### TIER 1 — Highest Priority (APIs / Structured Data Available)
1. **eTenders.gov.za** — FULL OCDS REST API with Swagger docs. Bulk CSV/JSON/Excel downloads. Creative Commons licensed. THIS IS THE PRIMARY SOURCE.
- API: https://ocds-api.etenders.gov.za/swagger/
- Data: https://data.etenders.gov.za/
- Covers: ALL government tenders, ALL provinces, ALL departments, ALL SOEs
### TIER 2 — RSS Feeds Available (Automatable)
2. **sa-tenders.co.za** — RSS feed + WordPress REST API. Freemium (R749/6mo or R1,199/yr for full access).
3. **armscor.co.za** — RSS feed + WordPress REST API. Defence sector.
4. **eskom.co.za** — RSS feed + WordPress REST API. Energy sector.
5. **health.gov.za** — RSS feed + WP REST API + TablePress tables. Health sector.
6. **cidb.org.za** — WP REST API + iCal feed + tender-management plugin. Construction sector.
7. **masterbuilders.org.za** — RSS feed + WP REST API. Construction industry.
### TIER 3 — Web Scraping Required
8. **SETAs** (12+ active sites) — Individual websites, no APIs. Better monitored via eTenders.gov.za.
9. **Universities** (7+ active sites) — Individual procurement pages, no APIs. Also on eTenders.gov.za.
10. **SANRAL** — WordPress site, likely has RSS but needs verification.
11. **NERSA** — Has procurement page, needs scraping.
12. **Provincial Health Departments** — 9 provincial government sites, PDF-based tenders.
### TIER 4 — Closed Networks / Not Applicable
13. **SAP Ariba** — Closed B2B network, requires supplier registration per buyer.
14. **Basware** — Closed e-procurement network.
15. **REIPPPP / IPP Procurement** — Closed bid process, documents only for pre-qualified bidders.
### DEFUNCT / NOT FOUND
16. **etenders.co.za** — HTTP 410 Gone
17. **tender247.co.za** — DNS failure
18. **tendersa.co.za** — DNS failure
19. **freetender.co.za** — DNS failure
20. **freetendersite.co.za** — DNS failure
21. **corridorprocurement.co.za** — DNS failure
22. **procurementportal.co.za** — DNS failure
23. **professionalprocurement.co.za** — DNS failure
24. **tradematic.com** — NOT a tender platform (crypto trading bot platform)
25. **quest.co.za** — NOT a tender platform (recruitment agency)
26. **publicworks.gov.za** — DNS failure (tenders via eTenders.gov.za)
27. **energy.gov.za** — DNS failure (department restructured)
---
## KEY FINDING
The **eTenders.gov.za OCDS API** is the single most important data source. It provides:
- A documented JSON REST API (Swagger/OpenAPI spec)
- Open Contracting Data Standard (OCDS) compliance
- Bulk data downloads (CSV, JSON, Excel)
- Data from May 2021 onwards
- Creative Commons BY 4.0 license
- Coverage of ALL public sector tenders in South Africa
Most industry-specific portals (SETAs, universities, provincial departments, SOEs) publish their tenders through eTenders.gov.za as mandated by Treasury regulations. The OCDS API should capture the vast majority of these.
Commercial aggregators like sa-tenders.co.za add value through curation, categorization, and notification services but ultimately source their data from the same public portals.

306
docs/sources.md Normal file
View File

@@ -0,0 +1,306 @@
# South African Tender Sources — Comprehensive Research Report
**Research date:** July 2026
**Methodology:** Direct HTTP probing of 120+ URLs, OCDS API testing, browser verification, subagent web research
**Purpose:** Identify all online sources where SA tenders are published, with technical details for automation
---
## EXECUTIVE SUMMARY
**120+ URLs probed.** 42 confirmed live tender portals. 7 defunct/dead. The rest are either closed networks or redirect to the central portal.
### 🏆 The Gold Standard: eTenders.gov.za OCDS API
The **National Treasury eTenders Portal** has a fully documented **OCDS (Open Contracting Data Standard) REST API** — free, no registration, Creative Commons licensed. It covers ALL government, SOE, and public entity tenders. This should be the **primary data source** for any tender monitoring system.
- **API Base:** `https://ocds-api.etenders.gov.za/api/OCDSReleases`
- **Swagger Docs:** `https://ocds-api.etenders.gov.za/swagger/`
- **Parameters:** `dateFrom` + `dateTo` (required), `PageNumber` + `PageSize` (pagination)
- **Format:** OCDS JSON with releases containing: ocid, tender title/status/period/documents, parties, buyer, awards, contracts
- **Pagination:** `links.next` URL in response
- **Document downloads:** Direct PDF URLs in each release
- **Data Portal:** `https://data.etenders.gov.za/` — bulk CSV/JSON/Excel + dashboards
- **License:** Creative Commons BY 4.0
- **Data available from:** May 2021 onwards
### API Response Structure (verified)
```json
{
"uri": "...",
"version": "1.1",
"publishedDate": "2025-06-30T...",
"publisher": {"name": "National Treasury"},
"license": "https://creativecommons.org/licenses/by/4.0/",
"releases": [
{
"ocid": "ocds-9t57fa-137147",
"date": "2025-06-30T00:00:00Z",
"tag": ["tender"],
"tender": {
"title": "3100038635",
"status": "active",
"tenderPeriod": {"startDate": "...", "endDate": "..."},
"documents": [{"url": "https://www.etenders.gov.za/home/Download?..."}]
},
"parties": [{"name": "SAB AND T CHARTERED ACCOUNTANTS", "roles": ["buyer"]}],
"buyer": {...},
"awards": [...],
"contracts": [...]
}
],
"links": {"next": "https://ocds-api.etenders.gov.za/api/OCDSReleases?PageNumber=2&PageSize=100&dateFrom=...&dateTo=..."}
}
```
---
## TIER 1 — Automatable via API/RSS (7 sources)
| # | Source | URL | Type | API/RSS | Automation |
|---|--------|-----|------|---------|------------|
| 1 | **eTenders OCDS API** | `https://ocds-api.etenders.gov.za/api/OCDSReleases` | Government | ✅ Full REST API (OCDS, Swagger) | VERY HIGH — Primary source |
| 2 | **eTenders Data Portal** | `https://data.etenders.gov.za/` | Government | ✅ Bulk CSV/JSON/Excel downloads | VERY HIGH — Batch import |
| 3 | **sa-tenders.co.za** | `https://sa-tenders.co.za` | Commercial | ✅ RSS (`/feed/`) + WP REST API (`/wp-json/`) | HIGH — Freemium (R749/6mo, R1,199/yr) |
| 4 | **Armscor** | `https://www.armscor.co.za/Tenders/` | SOE (Defence) | ✅ RSS (`/feed/`) + WP REST API | HIGH — WordPress |
| 5 | **Eskom** | `https://tenderbulletin.eskom.co.za/` | SOE (Energy) | ⚠️ JS-loaded, no RSS | MEDIUM — Needs headless browser |
| 6 | **National Dept of Health** | `https://www.health.gov.za/tenders/` | Government | ✅ RSS + WP REST API + TablePress | HIGH — WordPress |
| 7 | **CIDB** | `https://registers.cidb.org.za/PublicTenders/TenderSearch` + `https://www.cidb.org.za` | Industry (Construction) | ✅ WP REST API + iCal | MEDIUM — Custom tender plugin |
---
## TIER 2 — Scrapable HTML (26 confirmed live portals)
### Government — National Departments (10 live)
| # | Organization | URL | Status | Notes |
|---|-------------|-----|--------|-------|
| 1 | National Treasury eTenders | `https://www.etenders.gov.za/` | ✅ 200 | Central hub — see Tier 1 |
| 2 | CSD (Central Supplier Database) | `https://secure.csd.gov.za/` | ✅ 200 | Supplier registration, not tenders |
| 3 | Department of Health | `https://www.health.gov.za/tenders/` | ✅ 200 | WordPress + RSS |
| 4 | DWS (Water & Sanitation) | `https://www.dws.gov.za/tenders` | ✅ 200 | HTML scraping |
| 5 | SARS | `https://www.sars.gov.za/tenders` | ✅ 200 | HTML scraping |
| 6 | Stats SA | `https://www.statssa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 7 | DIRCO (Foreign Affairs) | `https://www.dirco.gov.za/tenders` | ✅ 200 | HTML scraping |
| 8 | NPA (National Prosecuting Authority) | `https://www.npa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 9 | Public Service Commission | `https://www.psc.gov.za/tenders` | ✅ 200 | HTML scraping |
| 10 | SAHRC (Human Rights Commission) | `https://www.sahrc.org.za/tenders` | ✅ 200 | HTML scraping |
| 11 | State Security Agency | `https://www.ssa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 12 | Brand South Africa | `https://www.brandsouthafrica.co.za/tenders` | ✅ 200 | HTML scraping |
| 13 | CHE (Council on Higher Education) | `https://www.che.ac.za/tenders` | ✅ 200 | HTML scraping |
| 14 | DTIC (Trade & Industry) | `https://www.thedtic.gov.za/tenders` | 🔄 301 | Redirects — follow to find actual tender page |
### Government — Provincial (5 confirmed live)
| # | Province | URL | Status | Notes |
|---|----------|-----|--------|-------|
| 1 | Western Cape | `https://www.westerncape.gov.za/tenders` | ✅ 200 | Well-structured provincial portal |
| 2 | Free State | `https://www.freestateonline.fs.gov.za/tenders` | ✅ 200 | SharePoint-based |
| 3 | Limpopo | `https://www.limpopo.gov.za/tenders` | ✅ 200 | HTML scraping |
| 4 | North West | `https://www.nwpg.gov.za/tenders` | ✅ 200 | HTML scraping |
| 5 | Eastern Cape Treasury | `https://www.ectreasury.gov.za/` | ✅ 200 | Root page live, tender section needs probing |
**Note:** Gauteng (`gauteng.gov.za`), KZN (`kzn.gov.za`), Eastern Cape (`ecprovince.gov.za`), Mpumalanga (`mpu.gov.za`), and Northern Cape (`northerncape.gov.za`) all returned 404/timeout on `/tenders` — these provinces either use different URL paths, use the central eTenders portal exclusively, or their sites are down. Their tenders ARE captured in the eTenders OCDS API.
### Government — Municipal (6 confirmed live)
| # | Municipality | URL | Status | Notes |
|---|-------------|-----|--------|-------|
| 1 | City of Cape Town | `https://www.capetown.gov.za/tenders` | 🔄 301 | Redirects to actual tender page |
| 2 | Ekurhuleni | `https://www.ekurhuleni.gov.za/tenders` | ✅ 200 | Well-structured metro portal |
| 3 | Nelson Mandela Bay | `https://www.nelsonmandelabay.gov.za/tenders` | ✅ 200 | Metro portal |
| 4 | George Municipality | `https://www.george.gov.za/tenders` | ✅ 200 | Local municipality |
| 5 | Midvaal | `https://www.midvaal.gov.za/tenders` | ✅ 200 | Local municipality |
| 6 | Mbombela (Nelspruit) | `https://www.mbombela.gov.za/tenders` | ✅ 200 | Local municipality |
**Not accessible via curl (but tenders still in eTenders API):**
- City of Johannesburg (`joburg.org.za`) — site blocks curl
- eThekwini/Durban (`durban.gov.za`) — site not responding
- City of Tshwane (`tshwane.gov.za`) — site not responding
- Buffalo City (`buffalocity.gov.za`) — 404 on /tenders
- Mangaung/Bloemfontein (`mangaung.co.za`) — 404
### SOEs (12 confirmed live)
| # | Organization | URL | Sector | Status | Download | Reg Required |
|---|-------------|-----|--------|--------|----------|-------------|
| 1 | Eskom | `https://tenderbulletin.eskom.co.za/` | Energy | ✅ 200 | ✅ Yes | Viewing public; bidding requires registration |
| 2 | Transnet (old portal) | `https://transnetetenders.azurewebsites.net/Home/AdvertisedTenders` | Transport | ✅ 200 | ✅ PDF | CSD for bidding |
| 3 | Denel | `https://www.denel.co.za/tenders` | Defence | ✅ 200 | Limited | Unknown |
| 4 | Armscor | `https://www.armscor.co.za/Tenders/` | Defence | ✅ 200 | ✅ Yes | Public browsing; login for bidding |
| 5 | IDC | `https://www.idc.co.za/tenders/` | Development Finance | ✅ 200 | ✅ Yes (PDF + OneDrive) | Submission via email/OneDrive |
| 6 | DBSA | `https://www.dbsa.org/procurement` | Development Banking | ✅ 200 | ✅ Yes (PDF, BoQs) | Public viewing |
| 7 | Land Bank | `https://landbank.co.za/Supply-Chain/Pages/SupplyChainManagement.aspx` | Agriculture Finance | ✅ 200 | Unknown | Portal login |
| 8 | ACSA (Airports) | `https://www.airports.co.za/business/supply-chain-management/current-and-future-tenders` | Airports | ✅ 200 | Likely | Stakeholder login |
| 9 | SAFCOL | `https://www.safcol.co.za/tenders` | Forestry | ✅ 200 | Unknown | Unknown |
| 10 | Sedibeng Water | `https://www.sedibengwater.co.za/tenders/` | Water | ✅ 200 | Unknown | Unknown |
| 11 | NERSA | `https://www.nersa.org.za/procurement` | Energy Regulator | ✅ 200 | Unknown | Unknown |
| 12 | Master Builders SA | `https://www.masterbuilders.org.za` | Construction Industry | ✅ 200 | Member access | Membership for full access |
### Industry/Other (2 confirmed live)
| # | Organization | URL | Sector | Notes |
|---|-------------|-----|--------|-------|
| 1 | CIDB Tender Register | `https://registers.cidb.org.za/PublicTenders/TenderSearch` | Construction | Structured search — good for scraping |
| 2 | CIDB Main Site | `https://www.cidb.org.za` | Construction | WordPress + REST API |
---
## TIER 3 — Closed Networks / Not Publicly Scrapable
| # | Platform | URL | Notes |
|---|----------|-----|-------|
| 1 | SAP Ariba | `https://www.sap.com/products/spend-management.html` | Closed B2B network. Used by Anglo American, Sasol, Standard Bank, etc. Requires supplier registration per buyer. APIs exist but only within network. |
| 2 | Basware | `https://www.basware.com/` | Closed e-procurement. Same model as Ariba. |
| 3 | REIPPPP / IPP Office | `https://www.ipp-ppp.co.za/` | Renewable energy procurement. Closed bid process — documents only for pre-qualified bidders. |
| 4 | Transnet eSupplier Portal (new) | `https://esupplierportal.transnet.net/` | New portal (Oct 2025). Not accessible via curl. Requires registration. |
| 5 | Rand Water Bids | `https://bids.randwater.co.za/` | Dedicated bidding platform. Requires registration. Not accessible via curl. |
| 6 | tenders.co.za | `https://www.tenders.co.za` | Cloudflare-protected (403). Would need headless browser. |
### Private Corporate Procurement Portals (not publicly scrapable)
Most large SA corporations use **supplier registration portals** rather than public tender listings. Tenders are sent to pre-registered suppliers. These are NOT suitable for automated tender monitoring:
- **Mining:** Anglo American, Sasol, Sibanye-Stillwater, Impala Platinum, Gold Fields, Harmony, Exxaro, BHP — all use SAP Ariba or proprietary supplier portals
- **Construction:** M&R, WBHO, Stefanutti, Aveng — respond to client tenders, don't publish their own
- **Retail:** Shoprite, Pick n Pay, Woolworths — supplier registration portals
- **Telecoms:** MTN, Vodacom — supplier registration portals
- **Banks:** Standard Bank, ABSA, FNB, Nedbank — supplier registration
- **Insurance:** Old Mutual, Sanlam, Discovery — supplier registration
---
## TIER 4 — Defunct / Dead Sites (confirmed)
| # | Site | URL | Status |
|---|------|-----|--------|
| 1 | etenders.co.za | `https://www.etenders.co.za` | HTTP 410 Gone |
| 2 | tender247.co.za | `https://www.tender247.co.za` | DNS failure |
| 3 | tendersa.co.za | `https://www.tendersa.co.za` | DNS failure |
| 4 | freetender.co.za | `https://www.freetender.co.za` | DNS failure |
| 5 | freetendersite.co.za | `https://www.freetendersite.co.za` | DNS failure |
| 6 | corridorprocurement.co.za | `https://www.corridorprocurement.co.za` | DNS failure |
| 7 | procurementportal.co.za | `https://www.procurementportal.co.za` | DNS failure |
| 8 | professionalprocurement.co.za | `https://www.professionalprocurement.co.za` | DNS failure |
| 9 | publicworks.gov.za | `https://www.publicworks.gov.za/` | DNS failure (tenders via eTenders.gov.za) |
| 10 | energy.gov.za | `https://www.energy.gov.za` | DNS failure (dept restructured into DMRE) |
### Not Tender Platforms (incorrectly listed in some sources)
- **Tradematic** — crypto trading bot platform, not tenders
- **Quest.co.za** — recruitment agency (Adcorp), not tenders
---
## SETAs (Sector Education and Training Authorities)
12 confirmed live. None have APIs — all need web scraping. **Better monitored via eTenders.gov.za** which captures all SETA tenders.
| SETA | URL | Sector | Status |
|------|-----|--------|--------|
| merSETA | `https://www.merseta.org.za` | Manufacturing & Engineering | ✅ Live |
| CHIETA | `https://www.chieta.org.za` | Chemical Industries | ✅ Live |
| MICT SETA | `https://www.mictseta.org.za` | Media & ICT | ✅ Live |
| INSETA | `https://www.inseta.org.za` | Insurance | ✅ Live |
| HWSETA | `https://www.hwseta.org.za` | Health & Welfare | ✅ Live |
| FoodBev SETA | `https://www.foodbev.org.za` | Food & Beverage | ✅ Live |
| Services SETA | `https://www.serviceseta.org.za` | Services | ✅ Live |
| MQA | `https://www.mqa.org.za` | Mining | ✅ Live |
| PSETA | `https://www.pseta.org.za` | Public Service | ✅ Live |
| BANKSETA | `https://www.bankseta.org.za` | Banking | ✅ Live |
| AgriSETA | `https://www.agriseta.co.za` | Agriculture | ✅ Live |
| UMALUSI | `https://www.umalusi.org.za` | Education QA | ✅ Live |
**Defunct/consolidated:** ETDP SETA, TETA, SETASA, FASSET, CETA, W&RSETA — domains no longer resolve (SETA landscape has been consolidated)
---
## Universities
All publish tenders on individual procurement pages. No APIs. Also captured in eTenders.gov.za.
| University | URL | Status |
|------------|-----|--------|
| UCT | `https://www.uct.ac.za` | ✅ Live |
| Wits | `https://www.wits.ac.za` | ✅ Live |
| Stellenbosch | `https://www.sun.ac.za` | ✅ Live |
| UKZN | `https://www.ukzn.ac.za` | ✅ Live |
| Nelson Mandela | `https://www.mandela.ac.za` | ✅ Live |
| Rhodes | `https://www.ru.ac.za` | ✅ Live |
| UP (Pretoria) | `https://www.up.ac.za` | ⚠️ 403 (blocks curl) |
| UJ (Johannesburg) | `https://www.uj.ac.za` | ⚠️ 403 (blocks curl) |
---
## Water Boards
| Water Board | URL | Status |
|------------|-----|--------|
| Rand Water | `https://bids.randwater.co.za/` | ❌ Not accessible via curl (registration required) |
| Umgeni Water | `https://www.umgeni.co.za/tenders/` | ❌ SSL issues |
| Amatola Water | `https://www.amatolawater.co.za/tenders/` | ❌ 500 error |
| Overberg Water | `https://www.overbergwater.co.za/tenders/` | ❌ 403 |
| Sedibeng Water | `https://www.sedibengwater.co.za/tenders/` | ✅ 200 |
**Note:** Water board tenders are also published via eTenders.gov.za.
---
## RECOMMENDED AUTOMATION ARCHITECTURE
### Phase 1: Core Data Pipeline (eTenders OCDS API)
```
eTenders OCDS API → Daily pull (dateFrom/dateTo) → Parse releases → Store in DB
↓
Tender monitoring system
```
- Pull daily using `dateFrom=<yesterday>&dateTo=<today>`
- Paginate using `links.next`
- Extract: ocid, title, status, tenderPeriod, documents (PDF URLs), buyer, parties
- Download tender PDFs from document URLs
- This single source covers **ALL government, SOE, and public entity tenders**
### Phase 2: Supplementary Sources (RSS/WordPress)
- sa-tenders.co.za RSS feed (freemium — adds curation/categorization)
- Armscor RSS (defence sector depth)
- Health.gov.za RSS (health sector depth)
- CIDB WP REST API (construction sector depth)
- Master Builders RSS (construction industry)
### Phase 3: Targeted Scraping (headless browser)
- Eskom tender bulletin (JS-loaded)
- Transnet old portal (Azure-hosted)
- Provincial portals (Western Cape, Free State, Limpopo, North West)
- Municipal portals (Cape Town, Ekurhuleni, Nelson Mandela Bay)
- Direct SOE portals (DBSA, IDC, ACSA, SAFCOL, Land Bank)
### Phase 4: Closed Network Monitoring (manual/registration)
- Transnet eSupplier Portal (new)
- Rand Water bids platform
- SAP Ariba (if monitoring specific enterprise buyers)
- REIPPPP/IPP Office (renewable energy bid windows)
### What NOT to bother with
- Defunct sites (7 confirmed dead)
- Private corporate portals (supplier registration only, not public tenders)
- Most SETAs and universities (captured in eTenders API)
- Construction company portals (they bid, they don't publish)
- Tradematic, Quest (not tender platforms)
---
## SUMMARY STATISTICS
| Category | Count | Automatable |
|----------|-------|-------------|
| Government (national) | 14 live | All via eTenders API |
| Government (provincial) | 5 live + 4 via eTenders | All via eTenders API |
| Government (municipal) | 6 live + 5 via eTenders | All via eTenders API |
| SOEs | 12 live | 1 RSS + 11 scraping (all via eTenders API) |
| Commercial aggregators | 2 live | 1 RSS + 1 Cloudflare-protected |
| Industry-specific | 4 live | 2 RSS/API + 2 scraping |
| SETAs | 12 live | All via eTenders API |
| Universities | 8 live | All via eTenders API |
| Water boards | 1 live | Via eTenders API |
| Defunct | 10 confirmed dead | N/A |
| Closed networks | 6 | Not publicly accessible |
| **Total unique sources** | **~75** | **eTenders API covers ~90%** |

10
requirements.txt Normal file
View File

@@ -0,0 +1,10 @@
httpx>=0.28.1,<1
beautifulsoup4>=4.12.3,<5
lxml>=5.3.0,<6
selectolax>=0.3.27,<1
playwright>=1.49.0,<2
rich>=13.9.0,<14
click>=8.1.7,<9
feedparser>=6.0.11,<7
python-dateutil>=2.9.0,<3
tqdm>=4.67.0,<5

0
src/__init__.py Normal file
View File

94
src/config.py Normal file
View File

@@ -0,0 +1,94 @@
"""Configuration for the SA Tender Monitor."""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
PROJECT_ROOT = Path(__file__).resolve().parent.parent
DATA_DIR = PROJECT_ROOT / "data"
DB_PATH = DATA_DIR / "tenders.db"
RAW_DIR = DATA_DIR / "raw"
CACHE_DIR = DATA_DIR / "cache"
DOWNLOADS_DIR = DATA_DIR / "downloads"
@dataclass(frozen=True)
class OcdsConfig:
"""eTenders OCDS API configuration."""
base_url: str = "https://ocds-api.etenders.gov.za/api/OCDSReleases"
swagger_url: str = "https://ocds-api.etenders.gov.za/swagger/"
page_size: int = 100
timeout: int = 60
rate_limit_ms: int = 500 # delay between API page requests
@dataclass(frozen=True)
class WebUiConfig:
"""eTenders web UI scraping configuration.
The web UI uses DataTables server-side processing via
/Home/PaginatedTenderOpportunities. This endpoint requires a browser
session (cookies/CSRF) so we use Playwright to interact with it.
"""
base_url: str = "https://www.etenders.gov.za"
opportunities_path: str = "/Home/opportunities?id=1"
datatables_endpoint: str = "/Home/PaginatedTenderOpportunities"
download_path: str = "/home/Download"
page_size: int = 10 # the web UI default
timeout_ms: int = 30_000
headless: bool = True
@dataclass(frozen=True)
class SupplementarySource:
"""A supplementary tender source (RSS or scrapeable HTML)."""
name: str
url: str
source_type: str # "rss" | "scrape" | "api"
sector: str = ""
notes: str = ""
SUPPLEMENTARY_SOURCES: tuple[SupplementarySource, ...] = (
SupplementarySource("armscor", "https://www.armscor.co.za/Tenders/", "rss", "Defence"),
SupplementarySource("health", "https://www.health.gov.za/tenders/", "rss", "Health"),
SupplementarySource("cidb", "https://registers.cidb.org.za/PublicTenders/TenderSearch", "scrape", "Construction"),
SupplementarySource("sa-tenders", "https://sa-tenders.co.za/feed/", "rss", "All-sector aggregator"),
SupplementarySource("eskom", "https://tenderbulletin.eskom.co.za/", "scrape", "Energy"),
SupplementarySource("transnet", "https://transnetetenders.azurewebsites.net/Home/AdvertisedTenders", "scrape", "Transport"),
SupplementarySource("dbsa", "https://www.dbsa.org/procurement", "scrape", "Development Banking"),
SupplementarySource("idc", "https://www.idc.co.za/tenders/", "scrape", "Development Finance"),
SupplementarySource("nersa", "https://www.nersa.org.za/procurement", "scrape", "Energy Regulator"),
SupplementarySource("safcol", "https://www.safcol.co.za/tenders", "scrape", "Forestry"),
SupplementarySource("sedibeng-water", "https://www.sedibengwater.co.za/tenders/", "scrape", "Water"),
SupplementarySource("master-builders", "https://www.masterbuilders.org.za", "rss", "Construction"),
)
@dataclass(frozen=True)
class Config:
"""Top-level configuration."""
ocds: OcdsConfig = field(default_factory=OcdsConfig)
web_ui: WebUiConfig = field(default_factory=WebUiConfig)
supplementary_sources: tuple[SupplementarySource, ...] = SUPPLEMENTARY_SOURCES
db_path: Path = DB_PATH
downloads_dir: Path = DOWNLOADS_DIR
raw_dir: Path = RAW_DIR
cache_dir: Path = CACHE_DIR
def get_config() -> Config:
return Config()
def ensure_dirs() -> None:
"""Create data directories if they don't exist."""
for d in (DATA_DIR, RAW_DIR, CACHE_DIR, DOWNLOADS_DIR):
d.mkdir(parents=True, exist_ok=True)

294
src/db.py Normal file
View File

@@ -0,0 +1,294 @@
"""SQLite database for storing tenders and documents."""
from __future__ import annotations
import json
import sqlite3
from datetime import datetime
from pathlib import Path
from typing import Any
from .models import Tender, TenderDocument
SCHEMA = """
CREATE TABLE IF NOT EXISTS tenders (
ocid TEXT PRIMARY KEY,
internal_id INTEGER,
title TEXT NOT NULL DEFAULT '',
description TEXT DEFAULT '',
status TEXT DEFAULT '',
tender_number TEXT DEFAULT '',
category TEXT DEFAULT '',
tender_type TEXT DEFAULT '',
province TEXT DEFAULT '',
department TEXT DEFAULT '',
date_published TEXT,
closing_date TEXT,
briefing_date TEXT,
delivery_location TEXT DEFAULT '',
conditions TEXT DEFAULT '',
has_briefing INTEGER DEFAULT 0,
briefing_compulsory INTEGER DEFAULT 0,
briefing_venue TEXT DEFAULT '',
buyer_name TEXT DEFAULT '',
buyer_contact TEXT DEFAULT '',
buyer_email TEXT DEFAULT '',
buyer_telephone TEXT DEFAULT '',
buyer_fax TEXT DEFAULT '',
source TEXT DEFAULT '',
doc_count INTEGER DEFAULT 0,
last_updated TEXT,
raw_data TEXT,
UNIQUE(ocid)
);
CREATE TABLE IF NOT EXISTS documents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tender_ocid TEXT NOT NULL,
document_id TEXT,
title TEXT DEFAULT '',
url TEXT DEFAULT '',
document_type TEXT DEFAULT '',
format TEXT DEFAULT '',
file_name TEXT DEFAULT '',
extension TEXT DEFAULT '',
support_document_id TEXT DEFAULT '',
date_published TEXT,
date_modified TEXT,
source TEXT DEFAULT '',
downloaded INTEGER DEFAULT 0,
local_path TEXT,
FOREIGN KEY (tender_ocid) REFERENCES tenders(ocid) ON DELETE CASCADE,
UNIQUE(tender_ocid, document_id)
);
CREATE INDEX IF NOT EXISTS idx_tenders_status ON tenders(status);
CREATE INDEX IF NOT EXISTS idx_tenders_closing ON tenders(closing_date);
CREATE INDEX IF NOT EXISTS idx_tenders_department ON tenders(department);
CREATE INDEX IF NOT EXISTS idx_tenders_province ON tenders(province);
CREATE INDEX IF NOT EXISTS idx_tenders_category ON tenders(category);
CREATE INDEX IF NOT EXISTS idx_docs_tender ON documents(tender_ocid);
"""
class Database:
"""SQLite-backed tender storage."""
def __init__(self, db_path: Path | str) -> None:
self.db_path = Path(db_path)
self.db_path.parent.mkdir(parents=True, exist_ok=True)
self._init_schema()
def _init_schema(self) -> None:
with self._conn() as conn:
conn.executescript(SCHEMA)
def _conn(self) -> sqlite3.Connection:
conn = sqlite3.connect(str(self.db_path))
conn.execute("PRAGMA foreign_keys = ON")
conn.execute("PRAGMA journal_mode = WAL")
return conn
def upsert_tender(self, tender: Tender) -> bool:
"""Insert or update a tender. Returns True if new, False if updated."""
buyer = tender.buyer
with self._conn() as conn:
existing = conn.execute(
"SELECT ocid FROM tenders WHERE ocid = ?", (tender.ocid,)
).fetchone()
conn.execute(
"""
INSERT INTO tenders (
ocid, internal_id, title, description, status, tender_number,
category, tender_type, province, department,
date_published, closing_date, briefing_date,
delivery_location, conditions,
has_briefing, briefing_compulsory, briefing_venue,
buyer_name, buyer_contact, buyer_email, buyer_telephone, buyer_fax,
source, doc_count, last_updated, raw_data
) VALUES (
:ocid, :internal_id, :title, :description, :status, :tender_number,
:category, :tender_type, :province, :department,
:date_published, :closing_date, :briefing_date,
:delivery_location, :conditions,
:has_briefing, :briefing_compulsory, :briefing_venue,
:buyer_name, :buyer_contact, :buyer_email, :buyer_telephone, :buyer_fax,
:source, :doc_count, :last_updated, :raw_data
)
ON CONFLICT(ocid) DO UPDATE SET
internal_id = :internal_id,
title = :title,
description = :description,
status = :status,
tender_number = :tender_number,
category = :category,
tender_type = :tender_type,
province = :province,
department = :department,
date_published = :date_published,
closing_date = :closing_date,
briefing_date = :briefing_date,
delivery_location = :delivery_location,
conditions = :conditions,
has_briefing = :has_briefing,
briefing_compulsory = :briefing_compulsory,
briefing_venue = :briefing_venue,
buyer_name = :buyer_name,
buyer_contact = :buyer_contact,
buyer_email = :buyer_email,
buyer_telephone = :buyer_telephone,
buyer_fax = :buyer_fax,
source = :source,
doc_count = :doc_count,
last_updated = :last_updated,
raw_data = :raw_data
""",
{
"ocid": tender.ocid,
"internal_id": tender.internal_id,
"title": tender.title,
"description": tender.description,
"status": tender.status,
"tender_number": tender.tender_number,
"category": tender.category,
"tender_type": tender.tender_type,
"province": tender.province,
"department": tender.department,
"date_published": tender.date_published.isoformat() if tender.date_published else None,
"closing_date": tender.closing_date.isoformat() if tender.closing_date else None,
"briefing_date": tender.briefing_date.isoformat() if tender.briefing_date else None,
"delivery_location": tender.delivery_location,
"conditions": tender.conditions,
"has_briefing": int(tender.has_briefing),
"briefing_compulsory": int(tender.briefing_compulsory),
"briefing_venue": tender.briefing_venue,
"buyer_name": buyer.name if buyer else "",
"buyer_contact": buyer.contact_person if buyer else "",
"buyer_email": buyer.email if buyer else "",
"buyer_telephone": buyer.telephone if buyer else "",
"buyer_fax": buyer.fax if buyer else "",
"source": tender.source,
"doc_count": len(tender.documents),
"last_updated": tender.last_updated.isoformat() if tender.last_updated else None,
"raw_data": json.dumps(tender.raw_data, default=str),
},
)
# Upsert documents
for doc in tender.documents:
conn.execute(
"""
INSERT INTO documents (
tender_ocid, document_id, title, url, document_type, format,
file_name, extension, support_document_id,
date_published, date_modified, source
) VALUES (
:tender_ocid, :document_id, :title, :url, :document_type, :format,
:file_name, :extension, :support_document_id,
:date_published, :date_modified, :source
)
ON CONFLICT(tender_ocid, document_id) DO UPDATE SET
title = :title,
url = :url,
document_type = :document_type,
format = :format,
file_name = :file_name,
extension = :extension,
date_modified = :date_modified,
source = :source
""",
{
"tender_ocid": tender.ocid,
"document_id": doc.document_id,
"title": doc.title,
"url": doc.download_url,
"document_type": doc.document_type,
"format": doc.format,
"file_name": doc.file_name,
"extension": doc.extension,
"support_document_id": doc.support_document_id,
"date_published": doc.date_published.isoformat() if doc.date_published else None,
"date_modified": doc.date_modified.isoformat() if doc.date_modified else None,
"source": doc.source,
},
)
conn.commit()
return existing is None
def get_tender(self, ocid: str) -> dict[str, Any] | None:
"""Get a tender by ocid with its documents."""
with self._conn() as conn:
conn.row_factory = sqlite3.Row
row = conn.execute(
"SELECT * FROM tenders WHERE ocid = ?", (ocid,)
).fetchone()
if not row:
return None
tender = dict(row)
docs = conn.execute(
"SELECT * FROM documents WHERE tender_ocid = ? ORDER BY date_modified DESC",
(ocid,),
).fetchall()
tender["documents"] = [dict(d) for d in docs]
return tender
def list_tenders(
self,
status: str | None = None,
department: str | None = None,
province: str | None = None,
category: str | None = None,
limit: int = 100,
offset: int = 0,
) -> list[dict[str, Any]]:
"""List tenders with optional filters."""
query = "SELECT * FROM tenders WHERE 1=1"
params: list[Any] = []
if status:
query += " AND status = ?"
params.append(status)
if department:
query += " AND department LIKE ?"
params.append(f"%{department}%")
if province:
query += " AND province = ?"
params.append(province)
if category:
query += " AND category LIKE ?"
params.append(f"%{category}%")
query += " ORDER BY date_published DESC LIMIT ? OFFSET ?"
params.extend([limit, offset])
with self._conn() as conn:
conn.row_factory = sqlite3.Row
rows = conn.execute(query, params).fetchall()
return [dict(r) for r in rows]
def stats(self) -> dict[str, Any]:
"""Return database statistics."""
with self._conn() as conn:
total = conn.execute("SELECT COUNT(*) FROM tenders").fetchone()[0]
active = conn.execute("SELECT COUNT(*) FROM tenders WHERE status = 'active'").fetchone()[0]
total_docs = conn.execute("SELECT COUNT(*) FROM documents").fetchone()[0]
depts = conn.execute("SELECT COUNT(DISTINCT department) FROM tenders WHERE department != ''").fetchone()[0]
provinces = conn.execute("SELECT COUNT(DISTINCT province) FROM tenders WHERE province != ''").fetchone()[0]
# Tenders with >1 document (from web UI enrichment)
multi_doc = conn.execute(
"SELECT COUNT(*) FROM tenders WHERE doc_count > 1"
).fetchone()[0]
return {
"total_tenders": total,
"active_tenders": active,
"total_documents": total_docs,
"departments": depts,
"provinces": provinces,
"multi_doc_tenders": multi_doc,
}

265
src/models.py Normal file
View File

@@ -0,0 +1,265 @@
"""Data models for tenders, documents, and buyers."""
from __future__ import annotations
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from typing import Any
def _parse_dt(value: str | None) -> datetime | None:
"""Parse an ISO datetime string, returning None on failure."""
if not value:
return None
try:
# OCDS dates look like "2025-06-30T00:00:00Z"
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
return dt
except (ValueError, TypeError):
return None
@dataclass
class TenderDocument:
"""A document attached to a tender."""
document_id: str # UUID or blob name
title: str = ""
url: str = ""
document_type: str = "" # "basic", "tenderNotice", etc.
format: str = "" # "pdf", "pptx", etc.
date_published: datetime | None = None
date_modified: datetime | None = None
source: str = "ocds" # "ocds" or "web_ui"
# Web UI specific fields
file_name: str = ""
extension: str = ""
support_document_id: str = "" # the UUID used in download URLs
def to_dict(self) -> dict[str, Any]:
d = asdict(self)
for k in ("date_published", "date_modified"):
if d[k]:
d[k] = d[k].isoformat()
return d
@property
def download_url(self) -> str:
"""Construct the download URL for this document."""
if self.url:
return self.url
if self.support_document_id:
ext = self.extension or ".pdf"
base = "https://www.etenders.gov.za/home/Download"
return f"{base}?blobName={self.support_document_id}{ext}&downloadedFileName={self.file_name}"
return ""
@classmethod
def from_ocds(cls, doc: dict) -> TenderDocument:
"""Create from an OCDS document object."""
return cls(
document_id=doc.get("id", ""),
title=doc.get("title", ""),
url=doc.get("url", ""),
document_type=doc.get("documentType", ""),
format=doc.get("format", ""),
date_published=_parse_dt(doc.get("datePublished")),
date_modified=_parse_dt(doc.get("dateModified")),
source="ocds",
)
@classmethod
def from_web_ui(cls, doc: dict) -> TenderDocument:
"""Create from a web UI supportDocument object."""
sid = doc.get("supportDocumentID", "")
ext = doc.get("extension", "")
fname = doc.get("fileName", "")
return cls(
document_id=sid,
title=fname,
document_type="supportDocument",
format=ext.lstrip("."),
date_modified=_parse_dt(doc.get("dateModified")),
source="web_ui",
file_name=fname,
extension=ext,
support_document_id=sid,
)
@dataclass
class TenderBuyer:
"""A buyer (organ of state) for a tender."""
name: str = ""
identifier: str = "" # org ID if available
address: str = ""
contact_person: str = ""
email: str = ""
telephone: str = ""
fax: str = ""
def to_dict(self) -> dict[str, Any]:
return asdict(self)
@dataclass
class Tender:
"""A tender (procurement opportunity)."""
# Primary key — OCDS ocid or generated ID
ocid: str = ""
internal_id: int | None = None # eTenders internal ID (from web UI)
# Core metadata
title: str = ""
description: str = ""
status: str = "" # active, complete, cancelled, etc.
tender_number: str = "" # e.g. "TFR/2026/07/0002/114548/RFI"
# Classification
category: str = ""
tender_type: str = "" # "Request for Information", "Bid", etc.
province: str = ""
department: str = "" # organ of state
# Dates
date_published: datetime | None = None
closing_date: datetime | None = None
briefing_date: datetime | None = None
# Delivery
delivery_location: str = ""
conditions: str = ""
# Briefing session
has_briefing: bool = False
briefing_compulsory: bool = False
briefing_venue: str = ""
# Relationships
buyer: TenderBuyer | None = None
documents: list[TenderDocument] = field(default_factory=list)
# Metadata
source: str = "etenders" # which portal this came from
last_updated: datetime | None = None
raw_data: dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict[str, Any]:
d = {
"ocid": self.ocid,
"internal_id": self.internal_id,
"title": self.title,
"description": self.description,
"status": self.status,
"tender_number": self.tender_number,
"category": self.category,
"tender_type": self.tender_type,
"province": self.province,
"department": self.department,
"date_published": self.date_published.isoformat() if self.date_published else None,
"closing_date": self.closing_date.isoformat() if self.closing_date else None,
"briefing_date": self.briefing_date.isoformat() if self.briefing_date else None,
"delivery_location": self.delivery_location,
"conditions": self.conditions,
"has_briefing": self.has_briefing,
"briefing_compulsory": self.briefing_compulsory,
"briefing_venue": self.briefing_venue,
"source": self.source,
"last_updated": self.last_updated.isoformat() if self.last_updated else None,
"buyer": self.buyer.to_dict() if self.buyer else None,
"documents": [doc.to_dict() for doc in self.documents],
"doc_count": len(self.documents),
}
return d
@classmethod
def from_ocds(cls, release: dict) -> Tender:
"""Create a Tender from an OCDS release object."""
tender_data = release.get("tender", {})
parties = release.get("parties", [])
# Extract buyer info from parties
buyer = None
buyer_ref = release.get("buyer")
if parties:
for party in parties:
if "buyer" in (party.get("roles") or []):
buyer = TenderBuyer(
name=party.get("name", ""),
identifier=str(party.get("identifier", {}).get("id", "")) if isinstance(party.get("identifier"), dict) else "",
)
break
if not buyer and buyer_ref:
buyer = TenderBuyer(name=buyer_ref.get("name", ""))
# Map OCDS documents
docs = [TenderDocument.from_ocds(d) for d in tender_data.get("documents", [])]
# Also check contracts for documents
for contract in release.get("contracts", []):
for cdoc in contract.get("documents", []):
docs.append(TenderDocument.from_ocds(cdoc))
tender_period = tender_data.get("tenderPeriod", {})
published = _parse_dt(release.get("date"))
return cls(
ocid=release.get("ocid", ""),
title=tender_data.get("title", ""),
description=tender_data.get("description", ""),
status=tender_data.get("status", ""),
date_published=published,
closing_date=_parse_dt(tender_period.get("endDate")),
buyer=buyer,
documents=docs,
source="etenders_ocds",
last_updated=datetime.now(timezone.utc),
raw_data=release,
)
@classmethod
def from_web_ui(cls, row: dict) -> Tender:
"""Create a Tender from a web UI DataTables row."""
support_docs = row.get("supportDocument", []) or []
# Map supporting documents
docs = [TenderDocument.from_web_ui(d) for d in support_docs]
# Build buyer info
buyer = TenderBuyer(
name=row.get("department", ""),
contact_person=row.get("contactPerson", ""),
email=row.get("email", ""),
telephone=row.get("telephone", ""),
fax=row.get("fax", ""),
)
return cls(
ocid=row.get("ocid") or "",
internal_id=row.get("id"),
title=row.get("description", ""),
description=row.get("description", ""),
status=row.get("status", ""),
tender_number=row.get("tender_No", ""),
category=row.get("category", ""),
tender_type=row.get("type", ""),
province=row.get("province", ""),
department=row.get("department", ""),
date_published=_parse_dt(row.get("date_Published")),
closing_date=_parse_dt(row.get("closing_Date")),
briefing_date=_parse_dt(row.get("compulsory_briefing_session")),
delivery_location=row.get("delivery", ""),
conditions=row.get("conditions", ""),
has_briefing=bool(row.get("briefingSession")),
briefing_compulsory=bool(row.get("briefingCompulsory")),
briefing_venue=row.get("briefingVenue") or "",
buyer=buyer,
documents=docs,
source="etenders_web_ui",
last_updated=datetime.now(timezone.utc),
raw_data=row,
)

106
src/ocds_client.py Normal file
View File

@@ -0,0 +1,106 @@
"""eTenders OCDS API client.
The OCDS API is the primary source for tender discovery. It returns
OCDS-compliant JSON releases with tender metadata, buyer info, and at least
one document (the main advert). Supporting documents are NOT in the API —
they must be fetched via the web UI scraper.
API docs: https://ocds-api.etenders.gov.za/swagger/
Endpoint: https://ocds-api.etenders.gov.za/api/OCDSReleases
Params: dateFrom (required), dateTo (required), PageNumber, PageSize
"""
from __future__ import annotations
import time
from datetime import datetime, timedelta, timezone
from typing import Any, Iterator
import httpx
from .config import OcdsConfig
from .models import Tender
class OcdsClient:
"""Client for the eTenders OCDS API."""
def __init__(self, config: OcdsConfig | None = None) -> None:
self.config = config or OcdsConfig()
self._client = httpx.Client(
base_url=self.config.base_url,
timeout=self.config.timeout,
headers={"Accept": "application/json"},
)
def __enter__(self) -> OcdsClient:
return self
def __exit__(self, *args: Any) -> None:
self.close()
def close(self) -> None:
self._client.close()
def fetch_releases(
self,
date_from: str,
date_to: str,
) -> Iterator[list[dict[str, Any]]]:
"""Fetch OCDS releases for a date range, yielding pages.
Args:
date_from: Start date (YYYY-MM-DD or ISO 8601)
date_to: End date (YYYY-MM-DD or ISO 8601)
Yields:
Lists of OCDS release dicts (one list per page).
"""
page_number = 1
while True:
params = {
"dateFrom": date_from,
"dateTo": date_to,
"PageNumber": page_number,
"PageSize": self.config.page_size,
}
response = self._client.get("", params=params)
response.raise_for_status()
data = response.json()
releases = data.get("releases", [])
if not releases:
break
yield releases
# Check for next page
next_link = data.get("links", {}).get("next", "")
if not next_link:
break
page_number += 1
time.sleep(self.config.rate_limit_ms / 1000)
def fetch_tenders(
self,
date_from: str,
date_to: str,
) -> Iterator[Tender]:
"""Fetch and parse tenders for a date range.
Convenience wrapper around fetch_releases that yields Tender objects.
"""
for releases in self.fetch_releases(date_from, date_to):
for release in releases:
yield Tender.from_ocds(release)
@staticmethod
def date_range(days: int = 7) -> tuple[str, str]:
"""Generate a date range for the last N days (up to today)."""
today = datetime.now(timezone.utc)
start = today - timedelta(days=days)
return start.strftime("%Y-%m-%d"), today.strftime("%Y-%m-%d")

254
src/pipeline.py Normal file
View File

@@ -0,0 +1,254 @@
"""Main pipeline: OCDS API → web UI enrichment → database storage."""
from __future__ import annotations
import sys
import time
from datetime import datetime, timedelta, timezone
import click
from rich.console import Console
from rich.table import Table as RichTable
from .config import get_config, ensure_dirs
from .db import Database
from .models import Tender
from .ocds_client import OcdsClient
console = Console()
def run_pipeline(
date_from: str,
date_to: str,
enrich_with_web_ui: bool = True,
db_path: str | None = None,
) -> dict:
"""Run the full tender pipeline.
1. Pull tenders from OCDS API (discovery + metadata)
2. Optionally enrich with web UI scraping (full document lists)
3. Store everything in SQLite
Returns:
Summary stats dict.
"""
config = get_config()
ensure_dirs()
db = Database(db_path or str(config.db_path))
# ── Step 1: OCDS API ──────────────────────────────────────────────────
console.print(f"\n[bold cyan]Step 1: Pulling tenders from OCDS API[/]")
console.print(f" Date range: {date_from} → {date_to}")
ocds_tenders: list[Tender] = []
total_releases = 0
with OcdsClient(config.ocds) as client:
for page_releases in client.fetch_releases(date_from, date_to):
total_releases += len(page_releases)
for release in page_releases:
tender = Tender.from_ocds(release)
ocds_tenders.append(tender)
console.print(f" Fetched {total_releases} releases...")
console.print(f" [green]Total: {len(ocds_tenders)} tenders from OCDS API[/]")
# ── Step 2: Web UI enrichment (optional) ───────────────────────────────
web_ui_rows = []
matched = 0
if enrich_with_web_ui and ocds_tenders:
console.print(f"\n[bold cyan]Step 2: Enriching with web UI document lists[/]")
console.print(f" This uses Playwright to scrape the eTenders web UI")
console.print(f" to get the full supportDocument[] array per tender.")
console.print(f" [dim](The OCDS API only returns 1 doc per tender)[/]")
try:
from .web_scraper import WebUiScraper
with WebUiScraper(config.web_ui) as scraper:
web_ui_rows = scraper.fetch_tender_rows(status=1)
console.print(f" [green]Fetched {len(web_ui_rows)} rows from web UI[/]")
# Match and enrich
matches = scraper.match_ocds_to_web_ui(ocds_tenders, web_ui_rows)
for tender in ocds_tenders:
row = matches.get(tender.ocid)
if row:
scraper.enrich_tender_documents(tender, row)
matched += 1
console.print(f" [green]Matched & enriched: {matched}/{len(ocds_tenders)}[/]")
except ImportError:
console.print(f" [yellow]Playwright not installed. Skipping web UI enrichment.[/]")
console.print(f" [dim]Install with: pip install playwright && playwright install chromium[/]")
except Exception as e:
console.print(f" [yellow]Web UI enrichment failed: {e}[/]")
console.print(f" [dim]OCDS tenders will be stored without supporting documents.[/]")
# ── Step 3: Store in database ─────────────────────────────────────────
console.print(f"\n[bold cyan]Step 3: Storing in database[/]")
new_count = 0
updated_count = 0
for tender in ocds_tenders:
is_new = db.upsert_tender(tender)
if is_new:
new_count += 1
else:
updated_count += 1
console.print(f" [green]New: {new_count}[/] | [blue]Updated: {updated_count}[/]")
# ── Summary ────────────────────────────────────────────────────────────
stats = db.stats()
console.print(f"\n[bold green]✓ Pipeline complete![/]")
console.print(f" Total tenders in DB: {stats['total_tenders']}")
console.print(f" Active tenders: {stats['active_tenders']}")
console.print(f" Total documents: {stats['total_documents']}")
console.print(f" Multi-doc tenders: {stats['multi_doc_tenders']}")
console.print(f" Departments: {stats['departments']}")
console.print(f" Provinces: {stats['provinces']}")
return stats
def show_stats() -> None:
"""Show database statistics."""
config = get_config()
db = Database(str(config.db_path))
stats = db.stats()
table = RichTable(title="Tender Database Statistics")
table.add_column("Metric", style="cyan")
table.add_column("Count", style="green", justify="right")
table.add_row("Total Tenders", str(stats["total_tenders"]))
table.add_row("Active Tenders", str(stats["active_tenders"]))
table.add_row("Total Documents", str(stats["total_documents"]))
table.add_row("Multi-doc Tenders", str(stats["multi_doc_tenders"]))
table.add_row("Departments", str(stats["departments"]))
table.add_row("Provinces", str(stats["provinces"]))
console.print(table)
def list_tenders(
status: str | None = None,
department: str | None = None,
province: str | None = None,
category: str | None = None,
limit: int = 20,
) -> None:
"""List tenders from the database."""
config = get_config()
db = Database(str(config.db_path))
tenders = db.list_tenders(
status=status, department=department, province=province,
category=category, limit=limit,
)
if not tenders:
console.print("[yellow]No tenders found.[/]")
return
table = RichTable(title=f"Tenders ({len(tenders)} shown)")
table.add_column("OCID", style="dim")
table.add_column("Title", max_width=50)
table.add_column("Status")
table.add_column("Department", max_width=20)
table.add_column("Closing")
table.add_column("Docs", justify="right")
for t in tenders:
table.add_row(
t["ocid"][:20] + "..." if len(t["ocid"]) > 20 else t["ocid"],
t["title"][:50] + "..." if len(t["title"]) > 50 else t["title"],
t["status"],
t["department"][:20] if t["department"] else "",
t["closing_date"][:10] if t["closing_date"] else "",
str(t["doc_count"]),
)
console.print(table)
@click.group()
def cli() -> None:
"""SA Tender Monitor — pull and monitor SA government tenders."""
@cli.command()
@click.option("--days", default=7, help="Pull tenders from the last N days")
@click.option("--from", "from_date", help="Start date (YYYY-MM-DD)")
@click.option("--to", "to_date", help="End date (YYYY-MM-DD)")
@click.option("--no-web-ui", is_flag=True, help="Skip web UI document enrichment")
@click.option("--db", help="Database path (default: data/tenders.db)")
def pull(days: int, from_date: str | None, to_date: str | None, no_web_ui: bool, db: str | None) -> None:
"""Pull tenders from the eTenders portal."""
if from_date and to_date:
date_from, date_to = from_date, to_date
else:
today = datetime.now(timezone.utc)
start = today - timedelta(days=days)
date_from = start.strftime("%Y-%m-%d")
date_to = today.strftime("%Y-%m-%d")
run_pipeline(date_from, date_to, enrich_with_web_ui=not no_web_ui, db_path=db)
@cli.command()
def stats() -> None:
"""Show database statistics."""
show_stats()
@cli.command("list")
@click.option("--status", help="Filter by status (active, complete, etc.)")
@click.option("--department", help="Filter by department")
@click.option("--province", help="Filter by province")
@click.option("--category", help="Filter by category")
@click.option("--limit", default=20, help="Number of tenders to show")
def list_cmd(status: str | None, department: str | None, province: str | None,
category: str | None, limit: int) -> None:
"""List tenders from the database."""
list_tenders(status, department, province, category, limit)
@cli.command()
@click.argument("ocid")
def show(ocid: str) -> None:
"""Show details for a specific tender by OCID."""
config = get_config()
db = Database(str(config.db_path))
tender = db.get_tender(ocid)
if not tender:
console.print(f"[yellow]Tender {ocid} not found.[/]")
return
console.print(f"\n[bold cyan]Tender: {tender['title']}[/]")
console.print(f" OCID: {tender['ocid']}")
console.print(f" Status: {tender['status']}")
console.print(f" Department: {tender['department']}")
console.print(f" Province: {tender['province']}")
console.print(f" Published: {tender['date_published']}")
console.print(f" Closing: {tender['closing_date']}")
console.print(f" Buyer: {tender['buyer_name']}")
console.print(f" Contact: {tender['buyer_contact']} ({tender['buyer_email']})")
console.print(f" Documents: {tender['doc_count']}")
for doc in tender.get("documents", []):
console.print(f"\n [dim]📄 {doc['title']}[/]")
console.print(f" [dim]Type: {doc['document_type']} | Format: {doc['format']}[/]")
console.print(f" [dim]URL: {doc['url'][:100]}...[/]")
console.print(f" [dim]Source: {doc['source']}[/]")
if __name__ == "__main__":
cli()

284
src/web_scraper.py Normal file
View File

@@ -0,0 +1,284 @@
"""eTenders web UI scraper for fetching supporting documents.
The OCDS API only returns one document per tender (the main advert). The
web UI's internal DataTables endpoint returns each tender row with a
``supportDocument[]`` array containing ALL attached files. This module uses
Playwright to load the opportunities page, intercept the DataTables AJAX
response, and extract the full document list per tender.
The DataTables endpoint (/Home/PaginatedTenderOpportunities) requires a
browser session — it returns empty when called via curl/httpx without
proper cookies/session state. Playwright handles this transparently.
"""
from __future__ import annotations
import json
import time
from typing import Any
from .config import WebUiConfig
from .models import Tender, TenderDocument
# JavaScript injected into the page to intercept DataTables AJAX responses
# and extract the raw JSON data (which includes supportDocument[]).
INTERCEPT_JS = """
(async () => {
// Intercept the next DataTables AJAX response
return new Promise((resolve) => {
// Override XMLHttpRequest to capture the DataTables response
const origOpen = XMLHttpRequest.prototype.open;
const origSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.open = function(method, url) {
this._url = url;
return origOpen.apply(this, arguments);
};
XMLHttpRequest.prototype.send = function() {
const xhr = this;
const origOnReady = xhr.onreadystatechange;
xhr.onreadystatechange = function() {
if (xhr.readyState === 4 && xhr._url &&
xhr._url.includes('PaginatedTenderOpportunities')) {
try {
const data = JSON.parse(xhr.responseText);
window.__tenderData = data;
} catch(e) {
window.__tenderData = null;
}
}
if (origOnReady) origOnReady.apply(xhr, arguments);
};
return origSend.apply(xhr, arguments);
};
// Wait for the data to arrive, with a timeout
let attempts = 0;
const checkInterval = setInterval(() => {
attempts++;
if (window.__tenderData || attempts > 100) {
clearInterval(checkInterval);
resolve(JSON.stringify(window.__tenderData || {}));
}
}, 200);
});
})()
"""
# Alternative: directly read the DataTable API data (if table is already loaded)
READ_TABLE_DATA_JS = """
(() => {
if (typeof table === 'undefined' || !table) {
return JSON.stringify({error: 'table not initialized'});
}
const data = table.rows().data();
const rows = [];
for (let i = 0; i < data.length; i++) {
const row = data[i];
if (row) rows.push(row);
}
return JSON.stringify({rows: rows, count: rows.length});
})()
"""
class WebUiScraper:
"""Scrapes the eTenders web UI for full tender document lists.
Uses Playwright to load the opportunities page, which triggers the
DataTables AJAX call to /Home/PaginatedTenderOpportunities. The response
contains each tender row with a supportDocument[] array holding all
attached files — something the OCDS API doesn't provide.
"""
def __init__(self, config: WebUiConfig | None = None) -> None:
self.config = config or WebUiConfig()
self._playwright = None
self._browser = None
self._page = None
def start(self) -> None:
"""Launch the browser and open the opportunities page."""
from playwright.sync_api import sync_playwright
self._playwright = sync_playwright().start()
self._browser = self._playwright.chromium.launch(
headless=self.config.headless
)
context = self._browser.new_context(
user_agent=(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
)
)
self._page = context.new_page()
def stop(self) -> None:
"""Close the browser."""
if self._browser:
self._browser.close()
if self._playwright:
self._playwright.stop()
self._page = None
self._browser = None
self._playwright = None
def __enter__(self) -> WebUiScraper:
self.start()
return self
def __exit__(self, *args: Any) -> None:
self.stop()
def _load_page(self) -> None:
"""Navigate to the opportunities page and wait for DataTables to load."""
url = f"{self.config.base_url}{self.config.opportunities_path}"
self._page.goto(url, wait_until="networkidle", timeout=self.config.timeout_ms)
# Wait for the DataTable to be populated
self._page.wait_for_function(
"typeof table !== 'undefined' && table && table.rows().count() > 0",
timeout=self.config.timeout_ms,
)
def _get_table_rows(self) -> list[dict[str, Any]]:
"""Extract all tender row data from the loaded DataTable."""
# Execute JavaScript to read all row data directly from the DataTable API
result = self._page.evaluate(READ_TABLE_DATA_JS)
data = json.loads(result) if isinstance(result, str) else result
if "error" in data:
raise RuntimeError(f"DataTable not available: {data['error']}")
return data.get("rows", [])
def fetch_tender_rows(self, status: int = 1) -> list[dict[str, Any]]:
"""Fetch tender rows from the web UI.
Args:
status: 1=Advertised, 2=Awarded, 3=Closed, 4=Cancelled
Returns:
List of raw tender row dicts, each containing a supportDocument[] array.
"""
if not self._page:
raise RuntimeError("Scraper not started. Call start() or use context manager.")
self._load_page()
# If a different status is requested, click the corresponding tab
if status != 1:
tab_selectors = {2: "a:has-text('Awarded')", 3: "a:has-text('Closed')", 4: "a:has-text('Cancelled')"}
selector = tab_selectors.get(status)
if selector:
self._page.click(selector)
self._page.wait_for_timeout(2000) # Wait for DataTables to reload
rows = self._get_table_rows()
# If there are more pages, paginate through them
all_rows = list(rows)
total_pages = self._page.evaluate(
"() => typeof table !== 'undefined' && table ? table.page.info().pages : 1"
)
current_page = 1
while current_page < total_pages:
self._page.evaluate("table.page('next').draw(false)")
self._page.wait_for_timeout(1500) # Wait for AJAX to complete
page_rows = self._get_table_rows()
all_rows.extend(page_rows)
current_page += 1
return all_rows
def enrich_tender_documents(self, tender: Tender, web_ui_row: dict) -> Tender:
"""Enrich a Tender with supporting documents from a web UI row.
The OCDS API only has 1 document (the main advert). The web UI row
has the full supportDocument[] array. This merges them, adding any
documents from the web UI that aren't already in the tender.
"""
existing_doc_ids = {d.document_id for d in tender.documents}
for sd in web_ui_row.get("supportDocument", []) or []:
doc = TenderDocument.from_web_ui(sd)
if doc.document_id not in existing_doc_ids:
tender.documents.append(doc)
existing_doc_ids.add(doc.document_id)
# Also update metadata from the web UI that's not in OCDS
if web_ui_row.get("tender_No") and not tender.tender_number:
tender.tender_number = web_ui_row["tender_No"]
if web_ui_row.get("category") and not tender.category:
tender.category = web_ui_row["category"]
if web_ui_row.get("province") and not tender.province:
tender.province = web_ui_row["province"]
if web_ui_row.get("department") and not tender.department:
tender.department = web_ui_row["department"]
if web_ui_row.get("type") and not tender.tender_type:
tender.tender_type = web_ui_row["type"]
if web_ui_row.get("delivery") and not tender.delivery_location:
tender.delivery_location = web_ui_row["delivery"]
if web_ui_row.get("contactPerson") and tender.buyer:
tender.buyer.contact_person = web_ui_row["contactPerson"]
if web_ui_row.get("email") and tender.buyer:
tender.buyer.email = web_ui_row["email"]
if web_ui_row.get("telephone") and tender.buyer:
tender.buyer.telephone = web_ui_row["telephone"]
return tender
def match_ocds_to_web_ui(
self, ocds_tenders: list[Tender], web_ui_rows: list[dict]
) -> dict[str, dict[str, Any]]:
"""Build a lookup index matching OCDS tenders to web UI rows.
Matches on tender number, title similarity, or internal ID.
Returns a dict mapping ocid -> web_ui_row.
"""
# Build index by tender_No and title
by_tender_no: dict[str, dict] = {}
by_title: dict[str, dict] = {}
by_internal_id: dict[int, dict] = {}
for row in web_ui_rows:
tno = row.get("tender_No", "")
if tno:
by_tender_no[tno] = row
title = row.get("description", "")
if title:
by_title[title.strip().lower()[:100]] = row
iid = row.get("id")
if iid:
by_internal_id[iid] = row
matches: dict[str, dict] = {}
for tender in ocds_tenders:
# Try matching by tender number (if we can extract it from OCDS title)
row = None
# OCDS title often is the tender number
if tender.title and tender.title in by_tender_no:
row = by_tender_no[tender.title]
# Try title match
if not row and tender.description:
key = tender.description.strip().lower()[:100]
row = by_title.get(key)
if row:
matches[tender.ocid] = row
return matches

0
tests/__init__.py Normal file
View File

232
tests/test_ocds_client.py Normal file
View File

@@ -0,0 +1,232 @@
"""Tests for the OCDS API client and data models."""
import json
from datetime import datetime, timezone
import pytest
from src.models import Tender, TenderDocument, TenderBuyer
from src.ocds_client import OcdsClient
# ── Fixtures ────────────────────────────────────────────────────────────────
SAMPLE_OCDS_RELEASE = {
"ocid": "ocds-9t57fa-123931",
"id": "123931",
"date": "2025-06-02T00:00:00Z",
"tag": ["tender"],
"initiationType": "tender",
"tender": {
"title": "RFQ2025/135",
"status": "complete",
"tenderPeriod": {
"startDate": "2025-06-02T00:00:00Z",
"endDate": "2025-06-03T10:00:00Z",
},
"documents": [
{
"id": "5f928b6e-0be2-47ec-a7fa-39b1bf6d8e0a",
"documentType": "basic",
"title": "21 BATTALION CLC ADVERT.pdf",
"description": "21 BATTALION CLC ADVERT.pdf",
"url": "https://www.etenders.gov.za/home/Download?blobName=5f928b6e-0be2-47ec-a7fa-39b1bf6d8e0a.pdf&downloadedFileName=21%20BATT",
"datePublished": "2025-06-02T00:00:00Z",
"dateModified": "2025-06-02T00:00:00Z",
"format": "pdf",
"language": "en",
}
],
},
"parties": [
{
"name": "Test Department",
"roles": ["buyer"],
"identifier": {"id": "TEST-001"},
}
],
"buyer": {"name": "Test Department"},
}
SAMPLE_WEB_UI_ROW = {
"id": 160907,
"tender_No": "TFR/2026/07/0002/114548/RFI",
"description": "NOTICE TO THE MARKET: INVITATION FOR INTERESTED PARTIES",
"status": "Published",
"category": "Services: Professional",
"type": "Request for Information",
"province": "Western Cape",
"department": "Transnet SOC Ltd",
"date_Published": "2026-07-02T00:00:00",
"closing_Date": "2026-09-03T23:00:00",
"contactPerson": "TRIM Commercial",
"email": "noticeandcomments@transnet.net",
"telephone": "031-308-8256",
"supportDocument": [
{
"supportDocumentID": "2b32232e-777c-41d1-86e1-8dd2bd0eea6e",
"extension": ".pptx",
"fileName": "Notice and Comment advert.pptx",
"dateModified": "2026-07-02T05:57:24.3088558",
"active": True,
},
{
"supportDocumentID": "40efd46e-c7c5-4342-a0f6-f4d1692adbbd",
"extension": ".pdf",
"fileName": "Notice and Comments Advertisement Form.pdf",
"dateModified": "2026-07-02T05:57:24.7810594",
"active": True,
},
],
}
# ── Model tests ─────────────────────────────────────────────────────────────
class TestTenderDocument:
def test_from_ocds(self):
doc_data = SAMPLE_OCDS_RELEASE["tender"]["documents"][0]
doc = TenderDocument.from_ocds(doc_data)
assert doc.document_id == "5f928b6e-0be2-47ec-a7fa-39b1bf6d8e0a"
assert doc.title == "21 BATTALION CLC ADVERT.pdf"
assert doc.format == "pdf"
assert doc.source == "ocds"
assert doc.url.startswith("https://www.etenders.gov.za/home/Download")
assert doc.download_url == doc.url
def test_from_web_ui(self):
sd = SAMPLE_WEB_UI_ROW["supportDocument"][0]
doc = TenderDocument.from_web_ui(sd)
assert doc.document_id == "2b32232e-777c-41d1-86e1-8dd2bd0eea6e"
assert doc.title == "Notice and Comment advert.pptx"
assert doc.format == "pptx"
assert doc.source == "web_ui"
assert doc.extension == ".pptx"
assert doc.support_document_id == "2b32232e-777c-41d1-86e1-8dd2bd0eea6e"
def test_download_url_construction(self):
"""Web UI documents construct their own download URL from supportDocumentID."""
sd = SAMPLE_WEB_UI_ROW["supportDocument"][0]
doc = TenderDocument.from_web_ui(sd)
expected = (
"https://www.etenders.gov.za/home/Download"
"?blobName=2b32232e-777c-41d1-86e1-8dd2bd0eea6e.pptx"
"&downloadedFileName=Notice and Comment advert.pptx"
)
assert doc.download_url == expected
class TestTender:
def test_from_ocds(self):
tender = Tender.from_ocds(SAMPLE_OCDS_RELEASE)
assert tender.ocid == "ocds-9t57fa-123931"
assert tender.title == "RFQ2025/135"
assert tender.status == "complete"
assert tender.source == "etenders_ocds"
assert len(tender.documents) == 1
assert tender.documents[0].title == "21 BATTALION CLC ADVERT.pdf"
assert tender.buyer is not None
assert tender.buyer.name == "Test Department"
assert tender.date_published is not None
assert tender.closing_date is not None
def test_from_web_ui(self):
tender = Tender.from_web_ui(SAMPLE_WEB_UI_ROW)
assert tender.internal_id == 160907
assert tender.tender_number == "TFR/2026/07/0002/114548/RFI"
assert tender.category == "Services: Professional"
assert tender.province == "Western Cape"
assert tender.department == "Transnet SOC Ltd"
assert tender.tender_type == "Request for Information"
assert tender.source == "etenders_web_ui"
assert len(tender.documents) == 2
assert tender.documents[0].file_name == "Notice and Comment advert.pptx"
assert tender.documents[1].file_name == "Notice and Comments Advertisement Form.pdf"
assert tender.buyer is not None
assert tender.buyer.email == "noticeandcomments@transnet.net"
def test_ocds_to_web_ui_doc_count_difference(self):
"""The key test: OCDS gives 1 doc, web UI gives 2 docs for the same tender."""
ocds_tender = Tender.from_ocds(SAMPLE_OCDS_RELEASE)
web_ui_tender = Tender.from_web_ui(SAMPLE_WEB_UI_ROW)
assert len(ocds_tender.documents) == 1
assert len(web_ui_tender.documents) == 2
def test_to_dict_serialization(self):
tender = Tender.from_ocds(SAMPLE_OCDS_RELEASE)
d = tender.to_dict()
assert d["ocid"] == "ocds-9t57fa-123931"
assert d["title"] == "RFQ2025/135"
assert d["doc_count"] == 1
assert len(d["documents"]) == 1
assert d["buyer"]["name"] == "Test Department"
# ── OCDS client tests (mocked) ───────────────────────────────────────────────
class TestOcdsClient:
def test_date_range(self):
"""Date range helper returns valid YYYY-MM-DD strings."""
date_from, date_to = OcdsClient.date_range(days=7)
assert date_from < date_to
# Should be valid date strings
datetime.strptime(date_from, "%Y-%m-%d")
datetime.strptime(date_to, "%Y-%m-%d")
# ── Web UI enrichment tests ──────────────────────────────────────────────────
class TestWebUiEnrichment:
def test_enrichment_merges_documents(self):
"""Enriching an OCDS tender with web UI data adds the missing documents."""
from src.web_scraper import WebUiScraper
ocds_tender = Tender.from_ocds(SAMPLE_OCDS_RELEASE)
assert len(ocds_tender.documents) == 1 # Only the advert
scraper = WebUiScraper()
# Simulate enrichment with web UI data (different tender, same concept)
web_ui_doc = {
"supportDocumentID": "new-doc-001",
"extension": ".pdf",
"fileName": "Additional Document.pdf",
"dateModified": "2025-06-01T00:00:00",
}
ocds_tender = scraper.enrich_tender_documents(
ocds_tender, {"supportDocument": [web_ui_doc]}
)
# Should now have 2 documents: the OCDS one + the web UI one
assert len(ocds_tender.documents) == 2
assert ocds_tender.documents[0].source == "ocds"
assert ocds_tender.documents[1].source == "web_ui"
assert ocds_tender.documents[1].file_name == "Additional Document.pdf"
def test_enrichment_does_not_duplicate(self):
"""If a document already exists (same ID), it's not added again."""
from src.web_scraper import WebUiScraper
ocds_tender = Tender.from_ocds(SAMPLE_OCDS_RELEASE)
assert len(ocds_tender.documents) == 1
scraper = WebUiScraper()
# Add the same document ID as the OCDS one
web_ui_doc = {
"supportDocumentID": "5f928b6e-0be2-47ec-a7fa-39b1bf6d8e0a",
"extension": ".pdf",
"fileName": "21 BATTALION CLC ADVERT.pdf",
"dateModified": "2025-06-02T00:00:00",
}
ocds_tender = scraper.enrich_tender_documents(
ocds_tender, {"supportDocument": [web_ui_doc]}
)
# Should still be 1 (no duplicate)
assert len(ocds_tender.documents) == 1