Comments on: Ruby Could Replace my Python Crawler Pretty Soon http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/ It's Just Links Wed, 14 Sep 2011 13:47:04 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: tony http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98897 tony Wed, 30 Jul 2008 13:58:15 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98897 @Curtis: Here is those polish guy's test code. I haven't run it yet: http://pastie.org/private/jtqfdbloc83wqqnk525mzw @Curtis: Here is those polish guy’s test code. I haven’t run it yet: http://pastie.org/private/jtqfdbloc83wqqnk525mzw

]]> By: tony http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98895 tony Wed, 30 Jul 2008 13:54:38 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98895 @Michael You are correct. @Michael You are correct.

]]> By: tony http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98894 tony Wed, 30 Jul 2008 13:53:44 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98894 @Phil If your crawler is network bound, then you need more pipe. @Phil If your crawler is network bound, then you need more pipe.

]]> By: Phil http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98703 Phil Tue, 29 Jul 2008 22:21:56 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98703 I don't get it. Crawlers are network-bound; the speed of your implementation language has virtually no importance. I don’t get it. Crawlers are network-bound; the speed of your implementation language has virtually no importance.

]]> By: Curtis http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98608 Curtis Tue, 29 Jul 2008 14:58:35 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98608 It's kind of hard to evaluate metrics without seeing the actual code. A lot of micro benchmarks aren't indicative of real world performance. Tim Bray's widefinder project might be a good reference (both your and his are IO bound). In the end, programmer proficiency is probably the most important factor in speed. ;-) It’s kind of hard to evaluate metrics without seeing the actual code. A lot of micro benchmarks aren’t indicative of real world performance.

Tim Bray’s widefinder project might be a good reference (both your and his are IO bound). In the end, programmer proficiency is probably the most important factor in speed. ;-)

]]> By: Michael Campbell http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/comment-page-1/#comment-98586 Michael Campbell Tue, 29 Jul 2008 13:05:48 +0000 http://www.tonyspencer.com/2008/07/28/ruby-could-replace-my-python-crawler-pretty-soon/#comment-98586 You're presuming that your bottleneck was the threading. You’re presuming that your bottleneck was the threading.

]]>