It is a (rather messy) node.js codebase. Two rendering engines, including a hacked puppeteer package with stealth mode for better success rate. A big set of proxy providers under the hood. Bootstrapped.
Quite curious, I have been scraping some websites for my girlfriend with nodejs/puppeteer and put the content on an .epub file (she likes to read on her e-reader) and it can be quite annoying to bypass some anti-scraping techniques.
I use Clickhouse to store close to 1TB of API analytics data (which would be 10TB in MongoDB, Clickhouse has insane compression ) and it's a wonderful and stable SQL-first alternative to DuckDB - which is a very exciting piece of software, but is indeed too young to embed into boring production. The last time I checked DuckDB npm package, it used callbacks instead of awaits..
I can understand how the older callback API for node.js might form a negative impression, but it's really not indicative of the maturity of the core db engine at all. And remember: the vast majority of users use the Python API.
Even better news is that, as of a couple of months ago, there is now this package (which I wrote at MotherDuck and we have open sourced) which provides typed promise wrappers for the Duckdb API: https://www.npmjs.com/package/duckdb-async. This is an independent npm package for now, but was developed in close coordination with the DuckDb core team.
scrapeninja.net /scrape-js endpoint scrapes company pages of g2 without big troubles (with "us"/"eu" proxy geo in their online sandbox: https://scrapeninja.net/scraper-sandbox ).
They also have /scrape which is much faster because it does not bootstrap real browser, and bypasses CloudFlare TLS fingeprint check: https://pixeljets.com/blog/bypass-cloudflare/
If you are a minimalist, and are using VS Code, try https://marketplace.visualstudio.com/items?itemName=humao.re... which is a pure text syntax to describe API requests, and execute them right from the editor window. I now have api.http text file in every API-first project I am building and I love it.
I like this one because it's easy to keep API workflows with my projects. The scripting ability here is phenomenal. However only really useful if you code in VS Code.
Jetbrains also provides a similar, albeit slightly incompatible syntax for the same thing.
In the end, I think hurl [0] is nicer, because it’s open source and it’s a cli tool (and VS code also has a syntax highlighting plugin for it), making it editor independent.
Not OP but you can store all your routes in one file or multiple, it's up to you.
Personally what I do is I script out full API workflows in different files. So one file might login, then POST to add an object, then GET that object off an endpoint, then patch that endpoint, then trigger the GET object again.
Another workflow might login, upload an image, get that image, etc. For me the scripting is what makes this appealing.
But you could setup one file that documents and tests all your endpoints similar to Postman.
For me, it is always a pain to write and test cheerio code unless I was doing it on the previous week. The syntax of cheerio is somewhat similar to jQuery, but this is still node.js, and not "real" DOM.
I was suffering every time I was googling for "Cheerio quick examples", so I have built a cheerio sandbox to quickly test cheerio syntax against various test inputs. This is already helpful for myself and saves me up to 15-30 minutes on every simple scraper I am writing, I think, just because I have working selectors samples at hand and I can quickly test my new selectors.
Thank you for what you guys do. Altinity blog and videos are an outstanding source of practical in-depth knowledge on the subject, so much needed for Clickhouse recognition.
It is a (rather messy) node.js codebase. Two rendering engines, including a hacked puppeteer package with stealth mode for better success rate. A big set of proxy providers under the hood. Bootstrapped.
reply