http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this. There... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

binux on Nov 17, 2014 | parent | context | favorite | on: Show HN: A Python Spider System with Web UI

http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this.

There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some JavaScript running before/after page loaded to simulate a mouse click.

pknerd on Nov 17, 2014 [–]

But will it not be slow? Assuming downloading css/images etc?

binux on Nov 17, 2014 | [–]

Images not downloaded default. Both the fetcher and the phantomjs proxy is totally async.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact