Home > Code, Crawlers, Python > Crazy Python Crawler

Crazy Python Crawler

January 7th, 2008 tony Leave a comment Go to comments

Someone emailed me doubting my crawler could operate at the speeds I posted last week so here is a video I took this morning. I should have waited a few minutes after launching it before starting the video as it really starts cranking once all the threads get rocking and you can see that near the end of the video. Also notice my streaming internet radio going in and out thanks to no available bandwidth left on my 5Mbps line.

You can also hear a ticking sound. That is my new 1TB drive. It makes these weird ticking noises even when its not in use. REally sounds like the arm hitting something its not supposed to hit. Hope its not defective.

Video link

Categories: Code, Crawlers, Python Tags:
  1. January 11th, 2008 at 08:11 | #1

    Wow. Although, that ticking sound is never good. I would replace that drive immediately.

  2. roy
    June 20th, 2008 at 23:19 | #2

    Hi please send that script to me.

  3. Michael
    June 30th, 2008 at 16:31 | #3

    Hello, are you willing to post your source code so that we can all learn from your technique. I would really like to take a look at it. Are you recognizing robots.txt or parsing meta data at all? Thanks!

  4. tony
    June 30th, 2008 at 21:13 | #4

    Hi Michael
    Unfortunately this crawler was developed for a client and as a result I have to obey the terms of my contract with them which does not allow me to release the code.