jjsm01/sa-tender-monitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
sa-tender-monitor/docs/sources.md
jjsm01 2b39c0aa28 feat: initial SA Tender Monitor project 
- OCDS API client for eTenders.gov.za (primary tender discovery)
- Web UI scraper using Playwright (fetches supportDocument[] array
  that the OCDS API doesn't expose — only 1 doc per tender in OCDS)
- SQLite database with full schema for tenders + documents
- Click CLI: pull, stats, list, show commands
- 10 unit tests (all passing) covering models, OCDS parsing,
  web UI parsing, and document enrichment logic
- Comprehensive research report: 75+ SA tender sources documented
  with automation details (API/RSS/scraping availability per source)
- Hybrid architecture: OCDS for discovery + web scraping for full
  document lists (verified: web UI shows 2+ docs, OCDS shows 1)
2026-07-03 09:09:39 +00:00

306 lines
17 KiB
Markdown
Raw Permalink Blame History

# South African Tender Sources — Comprehensive Research Report
**Research date:** July 2026
**Methodology:** Direct HTTP probing of 120+ URLs, OCDS API testing, browser verification, subagent web research
**Purpose:** Identify all online sources where SA tenders are published, with technical details for automation
---
## EXECUTIVE SUMMARY
**120+ URLs probed.** 42 confirmed live tender portals. 7 defunct/dead. The rest are either closed networks or redirect to the central portal.
### 🏆 The Gold Standard: eTenders.gov.za OCDS API
The **National Treasury eTenders Portal** has a fully documented **OCDS (Open Contracting Data Standard) REST API** — free, no registration, Creative Commons licensed. It covers ALL government, SOE, and public entity tenders. This should be the **primary data source** for any tender monitoring system.
- **API Base:** `https://ocds-api.etenders.gov.za/api/OCDSReleases`
- **Swagger Docs:** `https://ocds-api.etenders.gov.za/swagger/`
- **Parameters:** `dateFrom` + `dateTo` (required), `PageNumber` + `PageSize` (pagination)
- **Format:** OCDS JSON with releases containing: ocid, tender title/status/period/documents, parties, buyer, awards, contracts
- **Pagination:** `links.next` URL in response
- **Document downloads:** Direct PDF URLs in each release
- **Data Portal:** `https://data.etenders.gov.za/` — bulk CSV/JSON/Excel + dashboards
- **License:** Creative Commons BY 4.0
- **Data available from:** May 2021 onwards
### API Response Structure (verified)
```json
{
"uri": "...",
"version": "1.1",
"publishedDate": "2025-06-30T...",
"publisher": {"name": "National Treasury"},
"license": "https://creativecommons.org/licenses/by/4.0/",
"releases": [
{
"ocid": "ocds-9t57fa-137147",
"date": "2025-06-30T00:00:00Z",
"tag": ["tender"],
"tender": {
"title": "3100038635",
"status": "active",
"tenderPeriod": {"startDate": "...", "endDate": "..."},
"documents": [{"url": "https://www.etenders.gov.za/home/Download?..."}]
},
"parties": [{"name": "SAB AND T CHARTERED ACCOUNTANTS", "roles": ["buyer"]}],
"buyer": {...},
"awards": [...],
"contracts": [...]
}
],
"links": {"next": "https://ocds-api.etenders.gov.za/api/OCDSReleases?PageNumber=2&PageSize=100&dateFrom=...&dateTo=..."}
}
```
---
## TIER 1 — Automatable via API/RSS (7 sources)
| # | Source | URL | Type | API/RSS | Automation |
|---|--------|-----|------|---------|------------|
| 1 | **eTenders OCDS API** | `https://ocds-api.etenders.gov.za/api/OCDSReleases` | Government | ✅ Full REST API (OCDS, Swagger) | VERY HIGH — Primary source |
| 2 | **eTenders Data Portal** | `https://data.etenders.gov.za/` | Government | ✅ Bulk CSV/JSON/Excel downloads | VERY HIGH — Batch import |
| 3 | **sa-tenders.co.za** | `https://sa-tenders.co.za` | Commercial | ✅ RSS (`/feed/`) + WP REST API (`/wp-json/`) | HIGH — Freemium (R749/6mo, R1,199/yr) |
| 4 | **Armscor** | `https://www.armscor.co.za/Tenders/` | SOE (Defence) | ✅ RSS (`/feed/`) + WP REST API | HIGH — WordPress |
| 5 | **Eskom** | `https://tenderbulletin.eskom.co.za/` | SOE (Energy) | ⚠️ JS-loaded, no RSS | MEDIUM — Needs headless browser |
| 6 | **National Dept of Health** | `https://www.health.gov.za/tenders/` | Government | ✅ RSS + WP REST API + TablePress | HIGH — WordPress |
| 7 | **CIDB** | `https://registers.cidb.org.za/PublicTenders/TenderSearch` + `https://www.cidb.org.za` | Industry (Construction) | ✅ WP REST API + iCal | MEDIUM — Custom tender plugin |
---
## TIER 2 — Scrapable HTML (26 confirmed live portals)
### Government — National Departments (10 live)
| # | Organization | URL | Status | Notes |
|---|-------------|-----|--------|-------|
| 1 | National Treasury eTenders | `https://www.etenders.gov.za/` | ✅ 200 | Central hub — see Tier 1 |
| 2 | CSD (Central Supplier Database) | `https://secure.csd.gov.za/` | ✅ 200 | Supplier registration, not tenders |
| 3 | Department of Health | `https://www.health.gov.za/tenders/` | ✅ 200 | WordPress + RSS |
| 4 | DWS (Water & Sanitation) | `https://www.dws.gov.za/tenders` | ✅ 200 | HTML scraping |
| 5 | SARS | `https://www.sars.gov.za/tenders` | ✅ 200 | HTML scraping |
| 6 | Stats SA | `https://www.statssa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 7 | DIRCO (Foreign Affairs) | `https://www.dirco.gov.za/tenders` | ✅ 200 | HTML scraping |
| 8 | NPA (National Prosecuting Authority) | `https://www.npa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 9 | Public Service Commission | `https://www.psc.gov.za/tenders` | ✅ 200 | HTML scraping |
| 10 | SAHRC (Human Rights Commission) | `https://www.sahrc.org.za/tenders` | ✅ 200 | HTML scraping |
| 11 | State Security Agency | `https://www.ssa.gov.za/tenders` | ✅ 200 | HTML scraping |
| 12 | Brand South Africa | `https://www.brandsouthafrica.co.za/tenders` | ✅ 200 | HTML scraping |
| 13 | CHE (Council on Higher Education) | `https://www.che.ac.za/tenders` | ✅ 200 | HTML scraping |
| 14 | DTIC (Trade & Industry) | `https://www.thedtic.gov.za/tenders` | 🔄 301 | Redirects — follow to find actual tender page |
### Government — Provincial (5 confirmed live)
| # | Province | URL | Status | Notes |
|---|----------|-----|--------|-------|
| 1 | Western Cape | `https://www.westerncape.gov.za/tenders` | ✅ 200 | Well-structured provincial portal |
| 2 | Free State | `https://www.freestateonline.fs.gov.za/tenders` | ✅ 200 | SharePoint-based |
| 3 | Limpopo | `https://www.limpopo.gov.za/tenders` | ✅ 200 | HTML scraping |
| 4 | North West | `https://www.nwpg.gov.za/tenders` | ✅ 200 | HTML scraping |
| 5 | Eastern Cape Treasury | `https://www.ectreasury.gov.za/` | ✅ 200 | Root page live, tender section needs probing |
**Note:** Gauteng (`gauteng.gov.za`), KZN (`kzn.gov.za`), Eastern Cape (`ecprovince.gov.za`), Mpumalanga (`mpu.gov.za`), and Northern Cape (`northerncape.gov.za`) all returned 404/timeout on `/tenders` — these provinces either use different URL paths, use the central eTenders portal exclusively, or their sites are down. Their tenders ARE captured in the eTenders OCDS API.
### Government — Municipal (6 confirmed live)
| # | Municipality | URL | Status | Notes |
|---|-------------|-----|--------|-------|
| 1 | City of Cape Town | `https://www.capetown.gov.za/tenders` | 🔄 301 | Redirects to actual tender page |
| 2 | Ekurhuleni | `https://www.ekurhuleni.gov.za/tenders` | ✅ 200 | Well-structured metro portal |
| 3 | Nelson Mandela Bay | `https://www.nelsonmandelabay.gov.za/tenders` | ✅ 200 | Metro portal |
| 4 | George Municipality | `https://www.george.gov.za/tenders` | ✅ 200 | Local municipality |
| 5 | Midvaal | `https://www.midvaal.gov.za/tenders` | ✅ 200 | Local municipality |
| 6 | Mbombela (Nelspruit) | `https://www.mbombela.gov.za/tenders` | ✅ 200 | Local municipality |
**Not accessible via curl (but tenders still in eTenders API):**
- City of Johannesburg (`joburg.org.za`) — site blocks curl
- eThekwini/Durban (`durban.gov.za`) — site not responding
- City of Tshwane (`tshwane.gov.za`) — site not responding
- Buffalo City (`buffalocity.gov.za`) — 404 on /tenders
- Mangaung/Bloemfontein (`mangaung.co.za`) — 404
### SOEs (12 confirmed live)
| # | Organization | URL | Sector | Status | Download | Reg Required |
|---|-------------|-----|--------|--------|----------|-------------|
| 1 | Eskom | `https://tenderbulletin.eskom.co.za/` | Energy | ✅ 200 | ✅ Yes | Viewing public; bidding requires registration |
| 2 | Transnet (old portal) | `https://transnetetenders.azurewebsites.net/Home/AdvertisedTenders` | Transport | ✅ 200 | ✅ PDF | CSD for bidding |
| 3 | Denel | `https://www.denel.co.za/tenders` | Defence | ✅ 200 | Limited | Unknown |
| 4 | Armscor | `https://www.armscor.co.za/Tenders/` | Defence | ✅ 200 | ✅ Yes | Public browsing; login for bidding |
| 5 | IDC | `https://www.idc.co.za/tenders/` | Development Finance | ✅ 200 | ✅ Yes (PDF + OneDrive) | Submission via email/OneDrive |
| 6 | DBSA | `https://www.dbsa.org/procurement` | Development Banking | ✅ 200 | ✅ Yes (PDF, BoQs) | Public viewing |
| 7 | Land Bank | `https://landbank.co.za/Supply-Chain/Pages/SupplyChainManagement.aspx` | Agriculture Finance | ✅ 200 | Unknown | Portal login |
| 8 | ACSA (Airports) | `https://www.airports.co.za/business/supply-chain-management/current-and-future-tenders` | Airports | ✅ 200 | Likely | Stakeholder login |
| 9 | SAFCOL | `https://www.safcol.co.za/tenders` | Forestry | ✅ 200 | Unknown | Unknown |
| 10 | Sedibeng Water | `https://www.sedibengwater.co.za/tenders/` | Water | ✅ 200 | Unknown | Unknown |
| 11 | NERSA | `https://www.nersa.org.za/procurement` | Energy Regulator | ✅ 200 | Unknown | Unknown |
| 12 | Master Builders SA | `https://www.masterbuilders.org.za` | Construction Industry | ✅ 200 | Member access | Membership for full access |
### Industry/Other (2 confirmed live)
| # | Organization | URL | Sector | Notes |
|---|-------------|-----|--------|-------|
| 1 | CIDB Tender Register | `https://registers.cidb.org.za/PublicTenders/TenderSearch` | Construction | Structured search — good for scraping |
| 2 | CIDB Main Site | `https://www.cidb.org.za` | Construction | WordPress + REST API |
---
## TIER 3 — Closed Networks / Not Publicly Scrapable
| # | Platform | URL | Notes |
|---|----------|-----|-------|
| 1 | SAP Ariba | `https://www.sap.com/products/spend-management.html` | Closed B2B network. Used by Anglo American, Sasol, Standard Bank, etc. Requires supplier registration per buyer. APIs exist but only within network. |
| 2 | Basware | `https://www.basware.com/` | Closed e-procurement. Same model as Ariba. |
| 3 | REIPPPP / IPP Office | `https://www.ipp-ppp.co.za/` | Renewable energy procurement. Closed bid process — documents only for pre-qualified bidders. |
| 4 | Transnet eSupplier Portal (new) | `https://esupplierportal.transnet.net/` | New portal (Oct 2025). Not accessible via curl. Requires registration. |
| 5 | Rand Water Bids | `https://bids.randwater.co.za/` | Dedicated bidding platform. Requires registration. Not accessible via curl. |
| 6 | tenders.co.za | `https://www.tenders.co.za` | Cloudflare-protected (403). Would need headless browser. |
### Private Corporate Procurement Portals (not publicly scrapable)
Most large SA corporations use **supplier registration portals** rather than public tender listings. Tenders are sent to pre-registered suppliers. These are NOT suitable for automated tender monitoring:
- **Mining:** Anglo American, Sasol, Sibanye-Stillwater, Impala Platinum, Gold Fields, Harmony, Exxaro, BHP — all use SAP Ariba or proprietary supplier portals
- **Construction:** M&R, WBHO, Stefanutti, Aveng — respond to client tenders, don't publish their own
- **Retail:** Shoprite, Pick n Pay, Woolworths — supplier registration portals
- **Telecoms:** MTN, Vodacom — supplier registration portals
- **Banks:** Standard Bank, ABSA, FNB, Nedbank — supplier registration
- **Insurance:** Old Mutual, Sanlam, Discovery — supplier registration
---
## TIER 4 — Defunct / Dead Sites (confirmed)
| # | Site | URL | Status |
|---|------|-----|--------|
| 1 | etenders.co.za | `https://www.etenders.co.za` | HTTP 410 Gone |
| 2 | tender247.co.za | `https://www.tender247.co.za` | DNS failure |
| 3 | tendersa.co.za | `https://www.tendersa.co.za` | DNS failure |
| 4 | freetender.co.za | `https://www.freetender.co.za` | DNS failure |
| 5 | freetendersite.co.za | `https://www.freetendersite.co.za` | DNS failure |
| 6 | corridorprocurement.co.za | `https://www.corridorprocurement.co.za` | DNS failure |
| 7 | procurementportal.co.za | `https://www.procurementportal.co.za` | DNS failure |
| 8 | professionalprocurement.co.za | `https://www.professionalprocurement.co.za` | DNS failure |
| 9 | publicworks.gov.za | `https://www.publicworks.gov.za/` | DNS failure (tenders via eTenders.gov.za) |
| 10 | energy.gov.za | `https://www.energy.gov.za` | DNS failure (dept restructured into DMRE) |
### Not Tender Platforms (incorrectly listed in some sources)
- **Tradematic** — crypto trading bot platform, not tenders
- **Quest.co.za** — recruitment agency (Adcorp), not tenders
---
## SETAs (Sector Education and Training Authorities)
12 confirmed live. None have APIs — all need web scraping. **Better monitored via eTenders.gov.za** which captures all SETA tenders.
| SETA | URL | Sector | Status |
|------|-----|--------|--------|
| merSETA | `https://www.merseta.org.za` | Manufacturing & Engineering | ✅ Live |
| CHIETA | `https://www.chieta.org.za` | Chemical Industries | ✅ Live |
| MICT SETA | `https://www.mictseta.org.za` | Media & ICT | ✅ Live |
| INSETA | `https://www.inseta.org.za` | Insurance | ✅ Live |
| HWSETA | `https://www.hwseta.org.za` | Health & Welfare | ✅ Live |
| FoodBev SETA | `https://www.foodbev.org.za` | Food & Beverage | ✅ Live |
| Services SETA | `https://www.serviceseta.org.za` | Services | ✅ Live |
| MQA | `https://www.mqa.org.za` | Mining | ✅ Live |
| PSETA | `https://www.pseta.org.za` | Public Service | ✅ Live |
| BANKSETA | `https://www.bankseta.org.za` | Banking | ✅ Live |
| AgriSETA | `https://www.agriseta.co.za` | Agriculture | ✅ Live |
| UMALUSI | `https://www.umalusi.org.za` | Education QA | ✅ Live |
**Defunct/consolidated:** ETDP SETA, TETA, SETASA, FASSET, CETA, W&RSETA — domains no longer resolve (SETA landscape has been consolidated)
---
## Universities
All publish tenders on individual procurement pages. No APIs. Also captured in eTenders.gov.za.
| University | URL | Status |
|------------|-----|--------|
| UCT | `https://www.uct.ac.za` | ✅ Live |
| Wits | `https://www.wits.ac.za` | ✅ Live |
| Stellenbosch | `https://www.sun.ac.za` | ✅ Live |
| UKZN | `https://www.ukzn.ac.za` | ✅ Live |
| Nelson Mandela | `https://www.mandela.ac.za` | ✅ Live |
| Rhodes | `https://www.ru.ac.za` | ✅ Live |
| UP (Pretoria) | `https://www.up.ac.za` | ⚠️ 403 (blocks curl) |
| UJ (Johannesburg) | `https://www.uj.ac.za` | ⚠️ 403 (blocks curl) |
---
## Water Boards
| Water Board | URL | Status |
|------------|-----|--------|
| Rand Water | `https://bids.randwater.co.za/` | ❌ Not accessible via curl (registration required) |
| Umgeni Water | `https://www.umgeni.co.za/tenders/` | ❌ SSL issues |
| Amatola Water | `https://www.amatolawater.co.za/tenders/` | ❌ 500 error |
| Overberg Water | `https://www.overbergwater.co.za/tenders/` | ❌ 403 |
| Sedibeng Water | `https://www.sedibengwater.co.za/tenders/` | ✅ 200 |
**Note:** Water board tenders are also published via eTenders.gov.za.
---
## RECOMMENDED AUTOMATION ARCHITECTURE
### Phase 1: Core Data Pipeline (eTenders OCDS API)
```
eTenders OCDS API → Daily pull (dateFrom/dateTo) → Parse releases → Store in DB
↓
Tender monitoring system
```
- Pull daily using `dateFrom=<yesterday>&dateTo=<today>`
- Paginate using `links.next`
- Extract: ocid, title, status, tenderPeriod, documents (PDF URLs), buyer, parties
- Download tender PDFs from document URLs
- This single source covers **ALL government, SOE, and public entity tenders**
### Phase 2: Supplementary Sources (RSS/WordPress)
- sa-tenders.co.za RSS feed (freemium — adds curation/categorization)
- Armscor RSS (defence sector depth)
- Health.gov.za RSS (health sector depth)
- CIDB WP REST API (construction sector depth)
- Master Builders RSS (construction industry)
### Phase 3: Targeted Scraping (headless browser)
- Eskom tender bulletin (JS-loaded)
- Transnet old portal (Azure-hosted)
- Provincial portals (Western Cape, Free State, Limpopo, North West)
- Municipal portals (Cape Town, Ekurhuleni, Nelson Mandela Bay)
- Direct SOE portals (DBSA, IDC, ACSA, SAFCOL, Land Bank)
### Phase 4: Closed Network Monitoring (manual/registration)
- Transnet eSupplier Portal (new)
- Rand Water bids platform
- SAP Ariba (if monitoring specific enterprise buyers)
- REIPPPP/IPP Office (renewable energy bid windows)
### What NOT to bother with
- Defunct sites (7 confirmed dead)
- Private corporate portals (supplier registration only, not public tenders)
- Most SETAs and universities (captured in eTenders API)
- Construction company portals (they bid, they don't publish)
- Tradematic, Quest (not tender platforms)
---
## SUMMARY STATISTICS
| Category | Count | Automatable |
|----------|-------|-------------|
| Government (national) | 14 live | All via eTenders API |
| Government (provincial) | 5 live + 4 via eTenders | All via eTenders API |
| Government (municipal) | 6 live + 5 via eTenders | All via eTenders API |
| SOEs | 12 live | 1 RSS + 11 scraping (all via eTenders API) |
| Commercial aggregators | 2 live | 1 RSS + 1 Cloudflare-protected |
| Industry-specific | 4 live | 2 RSS/API + 2 scraping |
| SETAs | 12 live | All via eTenders API |
| Universities | 8 live | All via eTenders API |
| Water boards | 1 live | Via eTenders API |
| Defunct | 10 confirmed dead | N/A |
| Closed networks | 6 | Not publicly accessible |
| **Total unique sources** | **~75** | **eTenders API covers ~90%** |
Reference in New Issue View Git Blame Copy Permalink