[Fixed]-Scrapy getting blank pages of javascript


You’re getting this because the website is served up via “Incapsula”, which uses a Javascript test before allowing you to continue onto the page. Since Scrapy doesn’t process Javascript, you aren’t getting any further then that.

To see what’s going on, open up Chrome, go to Privacy, and disable cookies and Javascript.

The site is actively trying to block Crawling, which is a clear indication that they don’t want to be crawled.

Leave a comment