Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
marginalia_nu
3 months ago
|
parent
|
context
|
favorite
| on:
Marginalia – A search engine that prioritizes non-...
I do a big recrawl around once every 10 weeks, but I have a RSS crawler that does daily visits and fetches new items. But that should be very low volume traffic.
stackghost
3 months ago
[–]
Interesting, does your RSS crawler auto-detect feeds? That's a neat solution I'd never considered.
marginalia_nu
3 months ago
|
parent
[–]
The big crawler tries to find feeds, and the RSS crawler acts on what the big one has found.
Consider applying for YC's Summer 2025 batch! Applications are open till May 13
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
