Replace the JS variable polling approach with Playwright response
interception — listen for DataTables AJAX responses at the network
layer instead of waiting for the JS variable to populate.
- Use 'domcontentloaded' instead of 'networkidle' (site keeps
long-poll connections open, preventing networkidle from firing)
- Capture /PaginatedTenderOpportunities responses via page.on('response')
- Paginate through all pages using table.page('next').draw()
- Verified live: 500 rows fetched, 173 multi-doc tenders found
- Full pipeline tested: 149 OCDS tenders → 141 matched & enriched
with web UI docs → 34 tenders now have >1 document in DB
- OCDS API client for eTenders.gov.za (primary tender discovery)
- Web UI scraper using Playwright (fetches supportDocument[] array
that the OCDS API doesn't expose — only 1 doc per tender in OCDS)
- SQLite database with full schema for tenders + documents
- Click CLI: pull, stats, list, show commands
- 10 unit tests (all passing) covering models, OCDS parsing,
web UI parsing, and document enrichment logic
- Comprehensive research report: 75+ SA tender sources documented
with automation details (API/RSS/scraping availability per source)
- Hybrid architecture: OCDS for discovery + web scraping for full
document lists (verified: web UI shows 2+ docs, OCDS shows 1)
