Virtualization and Grid Computing

Quoting from workspace news:

The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources.

The 100 node cluster was composed of a headnode and workernodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology.

[UPDATE, related: http://www.gridvm.org/virtual-cluster-appliances.html]

[UPDATE, see: One-click clusters, VWS TP1.3.3]

Previously, in S3 re-pricing commentary, I wrote about the good news that Amazon’s EC2 service was hitting capacity limits.

Sun has built it, but will they come? talks about Sun’s lackluster sales with its utility computing effort.

I’m wondering why there is this disparity. In my opinion, there are two major differences between Sun and Amazon’s offerings:

1. With Sun’s offering you need to port your program to Solaris.
2. Sun’s costs a dollar an hour, Amazon’s costs 10 cents an hour.

I think the porting problem is a much bigger limitation and this bodes well for the workspace concept in grid computing. There is a similar problem with the big grids in that they usually expect scientists to port their code to a homogenous platform — this is sometimes a near-impossible proposition.

[[I’m back from vacation. Look for more blog content and those search engines in the coming weeks.]]

Meanwhile, I came across this note:

One of the world’s biggest science experiments is proceeding on schedule, and grid computing will play a big role in it.

CERN, the European Organization for Nuclear Research, told delegates attending the 140th meeting of the CERN Council last month that the world’s highest energy particle accelerator, the Large Hadron Collider (LHC), will be set to go in 2007.

Massive in scale, grid techniques are the only financially viable way to deal with the collider’s petabytes of data (roughly 15 PB per year is anticipated) and the resulting analysis opportunities. It is going to be interesting to follow its launch, especially with respect to the computing side and what, if any, adjustments are going to be necessary.

• You can read in detail about the computing efforts at the LHC Computing Project (LCG) site (a lot of their software is based on Globus Toolkit components under the covers).
• In 2005, the grid surpassed 100 sites (today, 177 are listed).
• There are some interesting visualizations on the LCG monitoring page.
• A good read is the September 2006 Report of the LHCC Comprehensive Review of LCG (10 pages).
• There’s a nice video about the accelerator featured by SEED magazine: Lords of the Ring.

The kernel inclusion of KVM makes me think about some virtualization adoption issues.

This much younger technology has made it into the kernel faster than Xen, is that important? Enabling virtualization support by using a vanilla Linux kernel configuration is important for grid computing mostly because it helps with resource provider adoption of virtualization capabilities which is probably the biggest impediment to VMs + grid computing right now.

But even though Xen has not made it into the mainline kernel (it’s been discussed for several years now, and has gone through some twists and turns), distribution packaging support for it is good because of the very strong demand for Xen.

Personally I think Xen is easy enough to install and deal with if you are already familiar with basic kernel configuration/installation. Its weakness in this area is when you need to diverge from the specific kernel versions the patches are geared to work against. It’s best to keep it as high up in the kernel dev chain as possible.

But overall, if you’ve already decided to support virtualization, the VMM’s features, cost, license, performance, and management options are much bigger factors than this. To me, it’s just a minor deployment detail — unless you have applications whose support contracts would be invalidated by switching kernels.

I would be interested to find out how much the lack of a native kernel option is really affecting administrators, especially if it is the one thing stopping them from running Xen — either because of support contract issues or “pain” issues (real or perceived).

THE ANNUAL TERAGRID CONFERENCE, TERAGRID ‘07: BROADENING PARTICIPATION IN TERAGRID, invites all interested individuals and organizations to participate. Attendees will include scientists and engineers, faculty, post docs, graduate and undergraduate students, high school teachers, representatives from federal agencies, grid computing industry representatives, and staff from TeraGrid resource providers and partners.

Submissions should address the development of grid computing capabilities and the applications of the TeraGrid to research and education, in particular:

* Scientific impacts that are the results of work on the TeraGrid and with TeraGrid partners
* Technology development, capabilities, and services
* Grid education/training and grids in support of education
* Education, outreach, and training

Full papers are due January 12, 2007.

http://www.union.wisc.edu/teragrid07/

[[ UPDATE: paper deadline has been extended to February 8th ]]