I also made a YQL table for HackerNews a while back which should be as polite to the site as possible.

http://developer.yahoo.com/yql/console/?q=select%20*%20from%...

YQL implements a bunch of caching layers so it should tread as lightly as possible.

Disclaimer: I do work for Yahoo!

Just curious, especially after the deal with Microsoft: Has Yahoo made any long term commitments to support this? Would love to use this for some projects.

Yes. It's been explicitly stated on our blog and on the YQL page that it's not going away.

Specifically we've promised a minimum of 6mo notice if we ever did decide to shut it down.

is this condoned by PG?

He's aware of it because he responded to an AskHN question about it: http://news.ycombinator.com/item?id=1702399

He didn't say not to do it :)

For now, the answer seems to be yes.

I'm inclined to agree. Remember, pg doesn't provide an RSS feed for his essays.

http://www.paulgraham.com/rss.html

Any words on why there is no API for HN?

It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.

I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.

Time that could be spent working on some other, higher-marginal-utility thing, I would think.