Data Tools
Scraping, ETL, notebooks. Production-grade where we'd ship it, dev-only where we wouldn't.
Scraping
Apify
freemiumManaged scraping infra + open-source SDK.
— Hosted or self-run. Their SDK alone is worth a look.
cheerio
OSSjQuery-style server-side HTML parsing.
— We use it for every non-SPA crawl in the KB pipeline.
linkedom
OSSSpec-compliant DOM for worker/Node environments.
— When you need a real DOM and cheerio's jQuery style isn't enough.
Playwright
OSSHeadless browser for when fetch+parse isn't enough.
— Ships its own browsers; no driver drama. Our default for SPA scraping.
Puppeteer
OSSPlaywright's older sibling. Still fine, still maintained.
— Reach for this when the team already knows it; Playwright when starting fresh.
Scrapy
OSSPython-native crawling framework with built-in queue + middlewares.
— The right tool at 10K+ pages. Scales without becoming your project.
ETL
Dagster
freemiumType-first data pipelines with asset lineage.
— More opinionated than Prefect. Pick it when asset thinking fits your data.
DuckDB
OSSLocal analytical SQL on files.
— We use it for CSV wrangling in consulting engagements. One binary, no server.
Prefect
freemiumPython workflow orchestration with UI.
— What VORLUX's orchestrator would look like if we used off-the-shelf scheduling.
