Virtualization and Grid Computing

Instead of a single allocation, EC2 announced you can run several different kinds of instances.

See the EC2 home page for details:

$0.10 - Small Instance (Default)

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

In many cases it may be more cost effective to still get the small instance but just get a lot of them, this will be interesting for our workspace EC2 adapter and contextualization users (and us!). Once we make the small alterations to accomodate requesting these types, it will be just as easy to get 100 x small instance as 25 x large instance, or whatever combination, because deployment configurations can be coordinated on the fly. What would be best for what situation would have to be examined closely. An extra large instance for the virtual cluster head node(s) or storage/transfer node(s) could be extremely useful for the typical grid-cluster bottlenecks.

Quoting from workspace news:

The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources.

The 100 node cluster was composed of a headnode and workernodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology.

[UPDATE, related: http://www.gridvm.org/virtual-cluster-appliances.html]

[UPDATE, see: One-click clusters, VWS TP1.3.3]

Previously, in S3 re-pricing commentary, I wrote about the good news that Amazon’s EC2 service was hitting capacity limits.

Sun has built it, but will they come? talks about Sun’s lackluster sales with its utility computing effort.

I’m wondering why there is this disparity. In my opinion, there are two major differences between Sun and Amazon’s offerings:

1. With Sun’s offering you need to port your program to Solaris.
2. Sun’s costs a dollar an hour, Amazon’s costs 10 cents an hour.

I think the porting problem is a much bigger limitation and this bodes well for the workspace concept in grid computing. There is a similar problem with the big grids in that they usually expect scientists to port their code to a homogenous platform — this is sometimes a near-impossible proposition.

You can of course read new accounts of using VMs for server consolidation on a daily basis. Here’s a particularly dramatic example.

This article relays (HP CEO) Mark Hurd’s comments in a recent keynote:

He said that HP had more than 700 datamarts, 87 datacenters, 20 petabytes of non-shared storage, and were running on 5,000 applications. Since his arrival in 2004, the company has reduced the number of datacenters to three and consolidated 9,000 servers “by virtualizing and blading half the infrastructure.”

Noting that “virtualization is a big deal to us,” Hurd also said HP has cut the number of apps its uses to 1,400, and can now “provide real-time enterprise financial information, supply chain information, and customer information across the enterprise.”

What is most impressive to me is how HP’s internal users survived this transition. That’s quite a lot of infrastructure change over the course of three years, especially considering the application changes. It’s unwise to speculate without more information, but I bet it was not easy.

Nicholas Carr has commentary on Amazon’s S3 service here: Amazon re-prices S3.

One of the interesting things is the statement by Jeff Bezos that EC2 is “completely capacity-constrained” right now.

This is good news for workspaces and utility computing in general: people are buying into the VM-as-compute-capsule model in practice (workspaces are a lot like EC2, the biggest difference being the need for your own resources).

rPath Brings Amazon Web Services to Customers

It will work like this: software developers use rBuilder to build an Amazon Machine Image (AMI) that is stored using the Amazon Simple Storage Service (Amazon S3). Then, with a single click, rBuilder and rBuilder Online users can boot their software appliances on Amazon EC2.

If you’re currently in the Amazon Elastic Compute Cloud (Amazon EC2) beta community, you and your peers can now benefit from a larger pool of pre-built Amazon Machine Images (AMIs) by sharing your AMIs with others. An AMI is a packaged environment that includes all the necessary bits to set up and boot Amazon EC2 instances. The more AMIs are shared, the faster the community can innovate. After you’ve shared your image, advertise it to other developers in the Resource Center. We hope you’ll contribute yours!

Links:

Introduction to Sharing AMIs

Public AMIs

Amazon’s EC2 service runs on Xen. So ultimately you can grab ready to go VM images from other sources and use them on EC2 (only when you’re not remotely managing them on your own compute resources with the Virtual Workspace service, of course :-)).

There are good sources for many specialized images or base images (for customizing yourself) such as rBuilder and the VMware marketplace (VMware VMs can be converted to Xen images in most cases with an image copy and a little diligence).

rBuilder can deliver the built images directly to Amazon’s S3 service (where the EC2 images live as well), so I am hoping all of their appliances will be made available via the sharing options now available in the EC2 commandline interface.