Crawlers

Mar 26

Python is Ugly but Damn She’s Beautiful

Remember the Python crawler NotSleepy built to suck up all your internets and find your affiliate IDs? Well we kept massaging the code and finally slapped that thing down on a fat pipe. WOW. The stats are rocking now. How about double time! Latest Stats: 35.6 ...

Continue reading Python is Ugly but Damn She's Beautiful in >>

Jan 07

Crazy Python Crawler

Code, Crawlers, Python 4 Comments »

Someone emailed me doubting my crawler could operate at the speeds I posted last week so here is a video I took this morning. I should have waited a few minutes after launching it before starting the video as it really starts cranking once all the threads get rocking ...

Continue reading Crazy Python Crawler in >>

Jan 03

Big SEO’s with Crawlers: Lets See Your Stats

Code, Crawlers, Python 8 Comments »

OK I'm just ecstatic with my new crawler, I think nobody but Google has one better than me, and I'm ready for a good old fashion show-and-tell. Multi-threaded programming is a bear to deal with and I've written several crawlers in different languages. For years I've been plagued with ...

Continue reading Big SEO's with Crawlers: Lets See Your Stats in >>