Home > Code, Crawlers > Python is Ugly but Damn She’s Beautiful

Python is Ugly but Damn She’s Beautiful

Remember the Python crawler NotSleepy built to suck up all your internets and find your affiliate IDs? Well we kept massaging the code and finally slapped that thing down on a fat pipe. WOW. The stats are rocking now. How about double time!

Latest Stats:
35.6 URLs per second
3.073 Million URLs per day!

Whats most promising is that the new fat pipe is still the bottleneck which means that if anybody really wants to party, all we need to do is lay down some greenbacks and a OC-12 will show us mass terabyte pleasure.

Categories: Code, Crawlers Tags:
  1. April 4th, 2008 at 16:28 | #1

    Is your servers name COOTER? hahaha no wonder we get along.

  2. Andy
    April 21st, 2008 at 19:10 | #2

    Bring it on!