REPOSITORY CÔNG KHAI /Python
high-performance-hybrid-web-scraper
Hybrid web scraper with FastAPI, Playwright, pybind11, and a C++ cpr scraping engine
README
High-Performance Hybrid Web Scraper
A hybrid scraping MVP that combines a FastAPI backend, a C++17 HTTP engine exposed through pybind11, and Playwright-based browser workflows for dynamic pages.
The project is designed around a simple flow:
Web UI
-> FastAPI Backend
-> C++ cpr Swarm for fast static HTTP scraping
-> Playwright Deep Scrape for rendered HTML
-> Playwright API Interceptor for XHR/fetch JSON capture
-> Playwright + BeautifulSoup/lxml SSR DOM Parser
-> Structured JSON response
-> Browser UI display and exportFeatures
- Fast C++ scraping path using
cpr, worker threads, pybind11, and GIL release. - Browser Scout with domain-based in-memory cookie/User-Agent cache.
- Graceful fallback when browser scouting is unavailable.
- Deep Scrape mode for JavaScript-rendered pages.
- API Interceptor mode for capturing JSON returned by XHR/fetch calls.
- SSR DOM Parser mode using Playwright, BeautifulSoup4, and lxml.
- Optional proxy list passed through the frontend and backend.
- Per-request delay with jitter to avoid accidental request bursts.
- Client-side TXT and HTML export from the web UI.
- Cross-platform build scripts for Windows and Unix-like systems.
Project Structure
high-performance-hybrid-web-scraper/
├── backend/
│ ├── main.py
│ ├── models.py
│ ├── scout.py
│ ├── scraper_service.py
│ └── native/
├── cpp_core/
│ ├── CMakeLists.txt
│ ├── bindings.cpp
│ ├── scraper_core.cpp
│ └── scraper_core.hpp
├── frontend/
│ └── index.html
├── scripts/
│ └── copy_native_module.py
├── build.bat
├── build.sh
├── run.bat
├── run.sh
├── requirements.txt
└── README.mdRequirements
- Python 3.10 or newer
- CMake 3.24 or newer
- A C++17 compiler
- Git
- Network access during the first build
Windows:
- Install Visual Studio Build Tools with the "Desktop development with C++" workload.
- Make sure
cmakeis available in PATH.
Linux/macOS:
- Install a C++17 compiler such as GCC, Clang, or Apple Clang.
- Install CMake from your package manager or from the official installer.
Local Setup
Clone the repository:
git clone <your-repo-url>
cd high-performance-hybrid-web-scraperBuild everything:
chmod +x build.sh run.sh
./build.shWindows:
build.batThe build script will:
- Create
.venv. - Install Python dependencies.
- Install Playwright Chromium.
- Resolve the pybind11 CMake path using
python -m pybind11 --cmakedir. - Configure CMake.
- Build the
cpp_scrapernative module. - Copy the native module into
backend/native.
Run the app:
./run.shWindows:
run.batOpen the UI:
http://localhost:8000API docs:
http://localhost:8000/docsScraping Modes
C++ Swarm
Default mode. The backend scouts the first URL for browser headers, then passes URLs, cookies, User-Agent, proxies, and delay settings to the C++ module.
{
"urls": ["https://example.com"],
"proxies": [],
"deep_scrape": false,
"intercept_mode": false,
"ssr_parse_mode": false,
"delay_ms": 1000
}Deep Scrape
Uses Playwright to load pages and return fully rendered HTML.
{
"urls": ["https://example.com"],
"deep_scrape": true,
"proxies": [],
"delay_ms": 1000
}API Interceptor
Uses Playwright to load a page and capture JSON responses from XHR/fetch requests.
If
intercept_keyword is empty, all JSON XHR/fetch responses are captured.{
"urls": ["https://example.com"],
"intercept_mode": true,
"intercept_keyword": "products",
"proxies": [],
"delay_ms": 1000
}SSR DOM Parser
Uses Playwright to load the DOM and BeautifulSoup/lxml to extract matching elements.
If
css_selector is empty, the raw HTML is returned.{
"urls": ["https://example.com"],
"ssr_parse_mode": true,
"css_selector": "a",
"delay_ms": 1000
}API Endpoints
Health:
curl http://localhost:8000/healthScrape:
curl -X POST http://localhost:8000/scrape \
-H "Content-Type: application/json" \
-d '{"urls":["https://example.com"],"delay_ms":1000}'Typical response:
{
"mode": "cpp_swarm",
"scout_source": "cache",
"scout_error": null,
"results": [
{
"url": "https://example.com",
"status_code": 200,
"html_snippet": "<!doctype html>...",
"elapsed_ms": 184,
"thread_id": "12345",
"worker_id": 0,
"success": true,
"error": ""
}
],
"meta": {
"count": 1,
"worker_count": 1,
"proxy_count": 0,
"delay_ms": 1000,
"engine": "cpp17-cpr-http-swarm"
}
}Native Module Notes
The C++ module is imported by Python as
cpp_scraper.The binding releases the Python GIL while native scraping is running:
py::gil_scoped_release release;If the native function signature changes, rebuild the module:
./build.shWindows:
build.batTroubleshooting
ModuleNotFoundError: cpp_scraper
Run the build script again and check that a compiled module exists in
backend/native.Windows cannot replace cpp_scraper.pyd
Stop any running FastAPI/Uvicorn or Python process that imported the native module, then run
build.bat again.CMake cannot find pybind11
Activate the virtual environment and run:
python -m pybind11 --cmakedirThe build scripts already pass the printed path to CMake as
-Dpybind11_DIR.Playwright browser missing
Run:
python -m playwright install chromiumCMake cannot fetch cpr
The first build downloads cpr through CMake FetchContent. Check network access, then rerun the build.
Security and Repository Hygiene
The repository intentionally ignores:
.venv- Playwright browser binaries
- CMake build directories
- compiled native extensions
- environment files
- logs
- archives
Do not commit private proxy lists, API keys,
.env files, or scraped private data.