Comments on: What Happens When Web-Scale Computing Becomes a Commodity?

By: Greg Borenstein

Greg Borenstein — Sun, 23 Dec 2007 15:17:13 +0000

Thomas,
I totally agree with you about how much expertise is required to manage this kind of scale and that’s one of the major reasons I’m excited about Amazon and — one has to imagine — other companies going into this field: they manage a lot of the complexity for you. I haven’t actually used simpleDB yet, but in all of my s3 and ec2 experience, the services pretty much just work. All I think about is the surface area of their APIs and how those relate to my application. Part of that is because those services have beautiful ruby wrappings (for s3) and great pre-assembled Ruby on Rails-ready images (for ec2) that are a joy to work with — a great example of the open source abstraction and frameworks that I was talking about in regards to this kind of thing.
It’s not so much that the knowledge becomes a commodity as that it gets embodied in working software that is available to any serious developer (my definition of anyone :).

By: Thomas

Thomas — Sat, 22 Dec 2007 19:04:41 +0000

Sorry, just now remembered to come back and read the response to my comments. Guess they sounded a bit inflammatory. Wasn’t meant to sound that way. Sorry! ;~)
Anyway, I understand your point, Greg, that you didn’t really mean to say it was meant for “anyone,” but I think my initial reaction was that your description did seem to imply that the day is near when “anyone” could scale out like that.
I do have to disagree somewhat with Marcus’ point about scaling knowledge becoming a “commodity.” Sure, things might be moving in that direction, but I think it’s a long ways off still.
My reaction, though, really does hinge on the issue of managing this kind of scale. I remember clearly when IBM started talking about autonomous systems management back in the 90s. A lot of promises have been made over and over in that area and very little has happened. Having spent a good portion of my career in that market and been involved at least indirectly with a number of the open source efforts, I think it’s still a long way off from being ready to take advantage of this kind of scaling. That said, I really hope I’m wrong.
Either way, I do get the excitement that all this brings and in that sense I agree with your sentiments. Things ARE changing and quickly. I’ve been preaching about the wonders of EC2 and similar efforts since I first read about them and will continue to do so until I see the being accepted as a standard way of doing business.

By: Randy Bias

Randy Bias — Wed, 19 Dec 2007 22:54:35 +0000

Greg, I shared some thoughts on your great post in a recent article in my blog here. Regards, --Randy

By: Greg Borenstein

Greg Borenstein — Tue, 18 Dec 2007 09:09:31 +0000

Good points, Marcus.
Yeah, we’re looking at SimpleDB, though more in the six month timescale than for the immediate future. We already slosh around a lot of data and we’re about to increase that amount by a few orders of magnitude. And we’re beginning to run into some feasibility limits when trying to make big improvements to our data model that involve munging a lot of data. We think something like SimpleDB might be able to ease the pain and cost somewhat, but it’s gonna take major planning for us to get there.
It’s worth noting that we’re not in this situation because we have 500+ million users, but because our way of providing value is just super data intensive (something like looking for a needle in a haystack). Maybe I should have defined web-scale above in terms of moving around millions records (and their corresponding resources) than having millions of users…

By: Marcus

Marcus — Tue, 18 Dec 2007 08:53:50 +0000

Thomas, it’s now easier to launch a web application on a dedicated box than it was 3 years ago and this trend will only continue. Scaling expertise is itself becoming a commodity and it will continue to become even more so.
And yeah, cloud computing will help. Greg, are you two considering using Simple.DB? I dunno, getting results in XML? I think it looks messy. Then again, frameworks will abstract the pain away (at the cost of performance).
But as speed gets cheaper, abstraction gets cheaper and more prevalent.

By: Greg Borenstein

Greg Borenstein — Mon, 17 Dec 2007 22:15:08 +0000

Thomas,
First up, I’m not talking about “anyone”. I’m not picturing the Yahoo Pipes audience, but your average competent, intelligent web developer who does more than just Drupal customization and e-commerce integration.
Second, I’m not talking about building apps with eBay- and Google-sized audiences. Maybe I should have been more clear about that. I’m talking about applications that need access to that scale of computing — the ability to grep for some content across a serious majority of all web pages or store a gazillion records in some db storage system — as one-off actions in order to provide some service that has value to a normal scale audience.
Finally, while a lot of your criticisms of the feasibility of operating at this scale are true today, they won’t remain so. How long do you think it will be before existing frameworks and communities build tools and best practices that abstract a lot of these problems? Or for new ones to sprout up that are native to this computing environment but still easy to use for our target smart single developer? One of the great things about software is that not everyone has to be as smart as Linus Torvalds or David Hannemeier Hansson for a large number of people to be able to build valuable things on top of the systems they create.

By: Thomas

Thomas — Mon, 17 Dec 2007 20:54:08 +0000

I’m sorry, but I have to call bullsh*t on this. The mere existence of these services is far from breaking down the barrier necessary for *anyone* to scale to amazon/ebay/google size services. There are at least, as I see it, 2 things still missing that won’t be “commoditized.”
First, is the know how. Building a scalable application (really, application infrastructure) is not the same as building a basic app. And if anything, the general trends have shown that there is a great wide void of knowledge in the market about how to do this successfully.
Second, is the management of this architecture. Ask any company running a large number of EC2 instances how they are managing it. Most likely, they are using a number of custom built tools. If they’re lucky, they managed to shoehorn some of the existing systems management tools into doing what they need. Either way, there’s a lot that goes into making this work.
Now, I’m not going to try and say that widespread availability of these services isn’t going to change things drastically, but it’s certainly not going to suddenly open a flood gate for massive, google-sized applications.