Virtualization and Grid Computing

There’s been a lot of talk about the dangers of getting locked in to cloud platforms, developing an application that is only suited to one platform.

Here’s a, let’s say… “embellished” example: Gangsta cloud wars could pivot on the traffic-driving power of Google and Microsoft/Yahoo.

When you’re using VMs like Xen (e.g. on EC2), if you design things for it you “should be able to” move without a ton of hassle (research. plan.). The workspace project has been working on portability and usability (see The first one-click STAR production cluster) and one of the things we can do now is use the same VM image on a regular cluster (such as on the Teraport cloud) and EC2. The contextualization software can be configured to sense if it is on EC2 or not (and will bootstrap accordingly). It “would be nice” if such things were standardized but this is not a real problem right now (IMHO).

About something more “strongly typed” like Google’s AppEngine. Application migration might be a bit harder, but not if the APIs are well known and repeatable. Google’s SDK is even Apache 2 licensed.

To that point, have a look at Announcing AppDrop.com (host Google App Engine projects on EC2). It’s not there yet (database is a flat file) but, hey, it was developed in a few days. Cool. Read more at http://appdrop.com.

The long term idea is not that this would solve all your problems magically but that such things are possible, and if there’s a real market for choices, it seems like more work on things of this nature are also inevitable.

I’m no datacenter business expert, but the biggest problem right now seems to be that few people will be able to compete on costs/efficiencies of scale with Google/Amazon/Microsoft/eBay. (<predictions…>) It feels like it would naturally approach the straight web hosting business, though. Let’s say a standard, open source cloud computing infrastructure emerges (such as Apache httpd in the analogy). There will be various levels of players as far as the capital they have and certainly better and worse companies to choose from (including those that differentiate on service etc). But if you’re really sweating the savings an enormous company could provide with such efficiencies vs. a normal size company/datacenter, you’re probably at the point where you could save a whole lot more by buying your own computers.(</predictions…>)

Miscellaneous point about lock-in: something user-facing that ties you to a provider does not seem like a wise idea (e.g. Google’s Users API).

Emerging Filesystems is a LinuxWorld article that Chris Samuel just put online (see his blog entry about the article).

It has a lot of great overview information, benchmarks, and interesting insight into ChunkFS, NILFS, btrfs, ext4, Reiser4, and ZFS.

I’ve been interested in how people are tackling the fsck problem which is one of the things he discusses. In my mind, it is already pain enough regardless of where it’s heading. The problem is also relevant to virtualization and grid computing. If you’re persisting and reusing VMs (not using the copy+throwaway model), the filesystems inside the VM are going to fsck eventually. Perhaps you have some timeout code in the meta-client layer that will only tolerate a certain wait for the VM to boot and report on its status somehow (perhaps using workspace contextualization technology). Because of the possible fsck delay you wouldn’t want to set this timeout very low (where it’s useful) unless you disabled filesystem checking which is not a great idea. A fsck friendly filesystem might become a popular VM choice? I’ll surely use a stable one for my local VMs and laptop.

Wow, quoting from Virtualization to aid Intel in saving up to $1.8B through data center consolidation:

• Intel Corp. is reporting today that it is consolidating its 130 data centers worldwide to just 8 global hubs
• 90,000 employees, 137 terabytes of WAN traffic, 93,000 servers in house and supporting those is very, very challenging
• Intel will save between $1.4 and $1.8 billion over 7 years by replacing older technology with new multi-core Xeon processors, along with using techniques such as virtualization

The article points to more details in this video.

Instead of a single allocation, EC2 announced you can run several different kinds of instances.

See the EC2 home page for details:

$0.10 - Small Instance (Default)

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

In many cases it may be more cost effective to still get the small instance but just get a lot of them, this will be interesting for our workspace EC2 adapter and contextualization users (and us!). Once we make the small alterations to accomodate requesting these types, it will be just as easy to get 100 x small instance as 25 x large instance, or whatever combination, because deployment configurations can be coordinated on the fly. What would be best for what situation would have to be examined closely. An extra large instance for the virtual cluster head node(s) or storage/transfer node(s) could be extremely useful for the typical grid-cluster bottlenecks.

You’ve surely heard already that Citrix acquired XenSource.

The Xen guys I’ve met I’ve liked, I’m happy for them. Xen as an open source project and XenSource as its steward are supposed to live on through this. This could really happen given Ian Pratt’s (and Xen/XenSource as a whole) staunch take on avoiding “business infection” so far w/ the XenSource launch.

It’s been busy lately, attended the first dev.Globus All Hands Meeting and TeraGrid ‘07 right here in Madison.

At TG07, Kate gave a talk which is online. The paper she presented discusses among other things contextualization, the structure and mechanisms by which an appliance/workspace is “told” what it needs in order to adapt to its deployed environment. This is not just adaptation to site specific services but also to other appliances that may be deployed with it such as in a virtual cluster deployment.

Amidst the bustle we implemented a new backend to the Workspace Service, to Amazon’s Elastic Compute Cloud (EC2). We’ve deployed it to the University of Chicago’s Teraport cluster and will currently pay for usage by selected collaborators.

Besides being somewhat fun to implement (including getting the Globus and Amazon Secure Message stacks on the same wavelength), I think it’s going to be interesting.

Because grid resources are cautiously approaching the pioneering switch to virtualizing resources [1], even in part, it is going to be interesting and educational to see what people will be able to accomplish with workspaces when a large pool of resources is actually available on tap — today.

Because the same deployment protocols can be used for both native and EC2 resources, there are of course capacity overflow use cases. In the right situations, VMs are a good mechanism for providers to dynamically reach more consumers as the need arises.

For a feature list and description, see What is the EC2 backend?

——-

[1] and some would say inevitable switch, even with the performance costs. Consider also that ‘virtualizing resources’ may mean physical node re-imaging, cf. Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid.

From Quantum Scoop: The Holy Grail of particle physics may already have been found which was linked on Slashdot yesterday.

The current rumor, which comes in time for the summer conference circuit, may be different. It claims an experiment at the Tevatron has found a peak twice as high as the previous rumors’ bumps. And unlike the other rumors, this one includes details: the new particle’s mass, for instance, which fits within theoretical bounds on the standard model Higgs. Some versions include a decay chain, which describes what the new particle turned into as the experiment progressed, and which may be consistent with the standard model’s predictions.

Massimo Re Ferre’ asks: Will Microsoft sunset VMware? (responses on the VMTN forum)

One of the parts to note is the claim in his VMware / Microsoft comparison that VMware is trying to “change the rules” by participating in the creation of a new IT platform rather than focusing on mere server consolidation. From that section:

The next frontier would be Virtual Appliances which is a very different way to develop and deploy applications compared to what we are doing today.
[…]
This is a fascinating scenario and as you can imagine it involves more than just developing a hypervisor with a management interface: it involves creating a new culture on how we deal with IT, taking all the pieces apart and rebuild our datacenters in a much more efficient way.