tony spencer » Tony It's Just Links Sat, 19 Oct 2013 14:31:31 +0000 hourly 1 Ruby on Rails Scalability – Is it a Problem? Sun, 26 Nov 2006 22:17:39 +0000 Tony Nick Wilson caught my eye on his new podcasting blog when he stated/asked “Ruby on Rails doesn’t scale?”. Its no secret that we’ve been shifting the NotSleepy shop from a PHP on the front, Java on the back setup to Ruby on Rails for the past few months. In fact we started a new RoR training side business with David Black, and have been building a new suite of tools from scratch in Rails. The experience has been nothing short of revolutionary for me and has been more liberating than switching from briefs to boxers in my teens. So when someone claims I’m not going to be able to scale this thing I felt my heart sink.

Unfortunately Nick doesn’t go into detail as to what the specific bottleneck was for CrazyEgg but after speaking with a few friends and reading DHH’s entry on scaling Rails, I’m thinking its an issue of an architect that doesn’t know how to scale his app or has coded the system poorly. David Hansson’s point is that if you design your system to not maintain state at the app level and keep in line with shared nothing at the app level, then you can easily scale infinitely by adding more app nodes much like LiveJournal, eBay and even Google do. David concedes that Rails can only handle half as many requests per second as PHP and therefore you are roughly going to have to spend $500 a month for 2.6 million requests/day on Rails versus $250 on PHP. If you’ve built a Rails app you know this is absolutely an inconsequential cost compared to the additional cost of man hours to build, maintain, and enhance a PHP app.

There aren’t a lot of mega traffic sites running on Rails yet but CrazyEgg certainly isn’t touching the traffic 43things brings in daily. Also, a friend of mine runs SoulCast and while I don’t know his daily stats I imagine he does as many pages views a day as CrazyEgg and I’ve never seen the site lag.


Holy crap what a small internet world it is! Ok, after blogging this I was just sending a few emails reaching out to some folks I hung out with at Pubcon Vegas last week and I didn’t get a card from Neil Patel so in the process of hunting down his email I discovered that he is CTO for the company that built CrazyEgg and now I feel like an ass. :) Sorry Neil. I’m absolutely sure you are a skillfull architect and look forward to hearing more details.

Update 01/26/2007

I didn’t want to create a new post so I decided to append to the bottom of this post. David on scalability in the middle of a very long thread I just discovered. Great synopis and worth your time if you are still on the fence:

 I've said it before, but it bears repeating: There's nothing interesting about how Ruby on Rails scales. We've gone the easy route and merely followed what makes Yahoo!, LiveJournal, and other high-profile LAMP stacks scale high and mighty.

Take state out of the application servers and push it to database/memcached/shared network drive (that's the whole Shared Nothing thang). Use load balancers between your tiers, so you have load balancers -> web servers -> load balancers -> app servers -> load balancers -> database/memcached/shared network drive servers. (Past the entry point, load balancers can just be software, like haproxy).

In a setup like that, you can add almost any number of web and app servers without changing a thing.

Scaling the database is the "hard part", but still a solved problem. Once you get beyond what can be easily managed by a descent master-slave setup (and that'll probably take millions and millions of pageviews per day), you start doing partitioning.

Users 1-100K on cluster A, 100K-200K on cluster B, and so on. But again, this is nothing new. LiveJournal scales like that. I hear eBay too. And probably everyone else that has to deal with huge numbers.

So the scaling part is solved. What's left is judging whether the economics of it are sensible to you. And that's really a performance issue, not a scalability one.

If your app server costs $500 per month (like our dual xeons does) and can drive 30 requests/second on Rails and 60 requests/second on Java/PHP/.NET/whatever (these are totally arbitrary numbers pulled out of my...), then you're faced with the cost of $500 for 2.6 million requests/day on the Rails setup and $250 for the same on the other one.

Now. How much is productivity worth to you? Let's just take a $60K/year programmer. That's $5K/month. If you need to handle 5 million requests/day, your programmer needs to be 10% more productive on Rails to make it even. If he's 15% more productive, you're up $250. And this is not even considering the joy and happiness programmers derive from working with more productive tools (nor that people have claimed to be many times more productive).

Of course, the silly math above hinges on the assumption that the whatever stack is twice as fast as Rails. That's a very big if. And totally dependent on the application, the people, and so on. Some have found Rails to be as fast or faster than comparable "best-of-breed J2EE stacks" -- see

The point is that the cost per request is plummeting, but the cost of programming is not. Thus, we have to find ways to trade efficiency in the runtime for efficiency in the "thought time" in order to make the development of applications cheaper. I believed we've long since entered an age where simplicity of development and maintenance is where the real value lies.
David Heinemeier Hansson
Tuesday, July 12, 2005
]]> 6
Cross Linking Sites Still Burns Mon, 20 Nov 2006 19:15:58 +0000 Tony cross linking idiot
Its been widely known that cross linking your own sites will result in a penalty in Google. Just before leaving PubCon Vegas one of my bots notified me that something was seriously wrong with 2 of my sites. Yesterday when I got home I started trying to figure out what happened. DUH! Somehow in the heat of trying to do too many things at one time I had a brain lapse and decided to link to a page on siteB from siteA. They weren’t just on the same C-class, they were on the same IP. Boom! Both domains rendered useless.


]]> 1
Killer Idea for KEXP and Other Listener Supported Radio Pledge Drives Wed, 25 Oct 2006 02:42:35 +0000 Tony kexp.gifI’m a big fan of listener supported radio stations like KEXP in Seattle and KCRW from Santa Monica, CA and I’ve given money to both but I really hate the long winded pleas for money. So this weekend in the mountains I came up with an idea that will rock the radio pledge drives. Stations like KEXP have a huge internet audience. The problem with the old school pledge drive is that it goes on FOREVER. Give me the chance to make it stop on demand I will gladly donate on day 1.

Heres the idea:

  • During the pledge drive you promote the URL
  • When I, the listener from across the ether goes to that URL I can easily register an account and donate money
  • As soon as I donate the minimum amount I am given a login and password that allows me access to a private streaming feed that brings me the usual awesome streaming indie rock that I love. I don’t have to wait for 3 week long pledge drive to end.

Pay to Play!

I’m sure your bottom line will more than double and I can keep listening to commercial free radio.

]]> 1
Firefox 2.0 is not ready for public consumption Mon, 23 Oct 2006 16:57:03 +0000 Tony beachball.jpg On the heels of Matt Cutt’s review of the Firefox release candidate 2 I decided to give it whirl. I installed the soon to be version 2.0 on my Mac and have experienced many spinning beach ball of death while browsing. The browser locks up on everyday sites and I no option but to kill the process. I guess Matt isn’t on a Mac as he reports no crashes so my advice to the Mac users is to wait for the official release. I also didn’t see much improvement. I don’t really know if I like the fact that it implements SessionSaver by default. Isn’t that a memory hog? On the other hand Firefox doesn’t appear to be using near as much memory as before.

]]> 2
Borat is Mahir… I Kiss You! Thu, 12 Oct 2006 04:30:30 +0000 Tony I keep seeing the ads for the movie Borat and I couldn’t remember the name of the Turkish guy’s website that it reminded me of from the late 90’s. Finally wikipedia hooked me up and I found it. Mahir! Yeah they aren’t the same guy but Borat must have been derived from Mahir.



]]> 10
Cool Flickr set feature Mon, 09 Oct 2006 03:48:15 +0000 Tony I was always annoyed at how Flickr couldn’t keep my photos in order of date taken when I created a new set. After some messing around I discovered that they have added this feature. Go Flickr! The screenshot below should show you how to change the ordering. Hey Flickr… this should be the default behavior :)


]]> 0
Parallels… Another reason to love my Mac Mon, 09 Oct 2006 03:12:02 +0000 Tony I went to the Raleigh ruby on rails group hack night last week with David Black and had a good time chatting with local programmers. I spotted someone with a MacBook Pro and headed over to ask for some advice on how to get lighttpd running properly. Not only did Jared of No Fluff, Just Stuff help me with a better choice of RoR web server (he suggested Mongrel and after spending a weekend with it I whole heartedly agree) but he also sold me on the whole Parallels thing.


Once I switched to Mac I really never had any desire to waste any of my precious disk space on Windows. I use Remote Desktop to take care of my few Windows needs. However RDP often fails to work properly and is slow. So when Jared showed me how freaking fast and reliable and how little disk space a Windows XP install took on Parallels I was convinced. Its crazy. Boot XP in 30 seconds, in FULL SCREEN mode and it feels like you are on the fastest PC available. Not only that but you can boot just about any OS such as SusE or Fedora in Parallels.

Important one: It functions way better if you follow this tip from Jared (reply to my question about two mouse pointers and slow response):

Regarding the double cursor..

Load the Windows VM, then make it a partial window (via [alt]+[Return]).

Then, from the menu bar, go to VM/Install Parallels tools. I think that takes care of it. They enhance various OS level integrations.

]]> 1
Danah Boyd does freestyle social networking talk at UNC Fri, 15 Sep 2006 04:33:42 +0000 Tony I skipped out on the office today so I could catch Danah Boyd’s talk on social networking ala MySpace on the campus of UNC. Danah is a candid speaker and a damn good one as well. She put on as the shy type before the talk kicked off but as soon as Fred Stutzman introduced her she hit the ground running in a very entertaining and comfortable talk about the explosion of MySpace and the general teen culture on the internet.


Danah let it slip that she had collected info on nearly all Friendster and MySpace users which I interprete to mean that she wrote code to scrape all profiles. Cool! I can relate! So it was you that was the cause for Friendster to be so slow back in the JSP days. :) (P.S. to both Danah and Fred: the 100 million figure on MySpace is bogus. I know people generating MANY believeable profiles on MySpace with bots and I think they account for a pretty large chunk. MySpace is pretty inept to dealing with bots. Open up a thousand threads and let her rip!). But most of the talk was non-geeky and focused on how teens and adults interact in the virtual world. Here are some of my notes:

As a result of the 60’s backlash society has sought to constrain teens
and keep in a constant state of activity. Myspace is an outlet in a world in which teens are mostly occupied with school related activities.

These are just rough notes. I’ll clean them up this weekend.

Properties of MySpace
persistance – conversation continues when you aren’t around
searchability -
replicability – publish your conversation you had with your friend
invisible audiences -

Diliemna for teens – be cool but not get in trouble with adult peers

battlecry – social network by jerry fallwell

privacy problem with FB feeds
yes you can get that data by refreshing someones profile daily or through a script but not likely

exposure – don’t tell the world when i join a threesome group. friends with that geeky person
but don’t want all your friends to be notified

invasive – information that people are not ready deal with. cobot collected data in rooms.

cognitive caps to how much info you can deal with. following all that info is not reciprocal.
just b/c i follow someone carefully doesn’t mean they are there for you. what are the social
costs of having this info?

]]> 0
Roll your own Ruby on Rails Training Thu, 31 Aug 2006 18:12:45 +0000 Tony I’ve been wanting to dive into Ruby on Rails for nearly a year now since I saw that amazing 15 minute demonstration on the official site. I’ve known for a long time now that there is something really wrong about the repetitiveness of the code we’ve had to write in Java/JSP everytime we update an object model or change an interface.

We’re taking on quite a few new projects here that involve heavy development so I decided it was time to take the plunge. Immediately I started looking at the few available quick immersion courses offered around the country to see if we could get up to speed in a hurry. Bummer! Everything was booked.

So then I got the hairbrained idea that if I was going to spend the time locating someone to come do private session in house I may as well start a Ruby on Rails training series here in Raleigh with open registration. After a lot of calling around (thanks Nathaniel!) I was fortunate enough to establish a relationship with David A. Black who is highly regarded in the industry as an excellent trainer with 13 years experience as a professor, an acclaimed author of a new book Ruby for Rails which received a 10/10 rating on Slashdot, as well as a dedicated founding co-director of Ruby Central. So registration is officially open for the Introduction to Ruby on Rails course being offered by Raleigh on Rails October 2nd – 5th!

]]> 3
Virante’s super clever use of on page optimization Wed, 23 Aug 2006 19:01:05 +0000 Tony Russ IM’d me today to show me something cool. Very cool. Check out the clever use of on-page optimization to get rankings for a non-competitive phrase: five seo excuses

Pay particular attention to the subdomains and what they spell out. Very slick. I’m sure your phone will be ringing off the hook over at Virante now Russ.

]]> 2