Virtualization and Grid Computing

We are pleased to announce GridShib SAML Tools v0.3.0, the final release in the v0.3.0 development cycle:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/readme.html

http://gridshib.globus.org/download.html#saml-tools

The GridShib SAML Tools are a suite of standalone client tools that issue SAML assertions and optionally bind these assertions to X.509 proxy certificates. To try out the software before downloading, visit our online demo:

https://computer.ncsa.uiuc.edu/gst-demo/

The GridShib SAML Tools require only Java 1.4 (or later) and Ant 1.6 (or later). Proxy certificates issued by the SAML Tools are compatible with GridShib for Globus Toolkit v0.6.0 Alpha (or later).

There have been significant changes in this version of the GridShib SAML Tools since the previous release:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/CHANGES.txt

Important new features of GridShib SAML Tools v0.3.0 include:

• enhanced command-line interface
• new command-line options for the SAML Assertion Issuer Tool, including the option to output a DER-encoded ASN.1 structure
• new X.509 Binding Tool, to bind arbitrary content to a non-critical extension of an X.509 proxy certificate
• new SAML Security Info Tool, for examining the contents of X.509-bound SAML tokens
• expanded Java API, for producing and consuming SAML assertions and X.509 proxy certificates
• support for the TeraGrid Science Gateway Use Case

This development cycle was largely driven by the TeraGrid Science Gateway Use Case:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/teragrid/readme.html

Science Gateways use the SAML Tools to enable auditing, incident response, and access control in Globus-based grids.

To learn more about this and other use cases, visit the “About GridShib” page:

http://gridshib.globus.org/about.html

While the GridShib SAML Tools produce X.509-bound SAML tokens, the complementary software component GridShib for Globus Toolkit consumes them. The latter is scheduled for release later this month or early next. See the roadmap on the GridShib home page for the latest updates.

Tom Scavo
For the GridShib Team

Amazon SQS is a distributed message queue system with a simple, robust API and real infrastructure to back it. And their prices just dropped significantly from a penny per 100 requests to a penny per 10,000:

Dear Amazon SQS Developers,

We wanted to let you know about some changes we are making to Amazon SQS, based on customer feedback and watching the way customers are using the service. One thing we’ve heard consistently is that customers want to be able to use SQS along with our other services (e.g. Amazon EC2, Amazon S3), but need SQS to be less expensive for this to be more feasible. We looked at our architecture and feature set, and found a way to make a few, targeted changes, by deprecating a few infrequently used requests, which allow us to operate the service much more efficiently. Simultaneously, we are introducing a new pricing structure that replaces the previous per-messages-sent charge ($0.10/1,000 messages) with a new per-request fee ($0.01/10,000 requests, including all Amazon SQS operations). The net result is that the new pricing will result in significantly lower charges for most developers being billed for SQS.

I’m hoping we’ll look back in five years and reminisce about how they charged so much for EC2 as well :-) (I do think it’s a good price now unless you are looking to continually use many, many computers).

http://www.utexas.edu/oncampus/2007/11/15/tacc-feature/

The Texas Advanced Computing Center (TACC) recently announced its partnership with the World Community Grid. It will assist the project by running World Community Grid software on its employee PCs, installing the client on the new Stampede cluster –helping scientists scale their research for the World Community Grid – and allowing other large TACC clusters to run Grid computations when there are idle processors.
[…]
“We look forward to working with IBM to explore how researchers can most effectively utilize both TACC advanced systems and the World Community Grid to address problems with deep impact to society as well as science.”

Instead of a single allocation, EC2 announced you can run several different kinds of instances.

See the EC2 home page for details:

$0.10 - Small Instance (Default)

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

In many cases it may be more cost effective to still get the small instance but just get a lot of them, this will be interesting for our workspace EC2 adapter and contextualization users (and us!). Once we make the small alterations to accomodate requesting these types, it will be just as easy to get 100 x small instance as 25 x large instance, or whatever combination, because deployment configurations can be coordinated on the fly. What would be best for what situation would have to be examined closely. An extra large instance for the virtual cluster head node(s) or storage/transfer node(s) could be extremely useful for the typical grid-cluster bottlenecks.

Quoting from workspace news:

The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources.

The 100 node cluster was composed of a headnode and workernodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology.

[UPDATE, related: http://www.gridvm.org/virtual-cluster-appliances.html]

[UPDATE, see: One-click clusters, VWS TP1.3.3]

Previously, in S3 re-pricing commentary, I wrote about the good news that Amazon’s EC2 service was hitting capacity limits.

Sun has built it, but will they come? talks about Sun’s lackluster sales with its utility computing effort.

I’m wondering why there is this disparity. In my opinion, there are two major differences between Sun and Amazon’s offerings:

1. With Sun’s offering you need to port your program to Solaris.
2. Sun’s costs a dollar an hour, Amazon’s costs 10 cents an hour.

I think the porting problem is a much bigger limitation and this bodes well for the workspace concept in grid computing. There is a similar problem with the big grids in that they usually expect scientists to port their code to a homogenous platform — this is sometimes a near-impossible proposition.

[[I’m back from vacation. Look for more blog content and those search engines in the coming weeks.]]

Meanwhile, I came across this note:

One of the world’s biggest science experiments is proceeding on schedule, and grid computing will play a big role in it.

CERN, the European Organization for Nuclear Research, told delegates attending the 140th meeting of the CERN Council last month that the world’s highest energy particle accelerator, the Large Hadron Collider (LHC), will be set to go in 2007.

Massive in scale, grid techniques are the only financially viable way to deal with the collider’s petabytes of data (roughly 15 PB per year is anticipated) and the resulting analysis opportunities. It is going to be interesting to follow its launch, especially with respect to the computing side and what, if any, adjustments are going to be necessary.

• You can read in detail about the computing efforts at the LHC Computing Project (LCG) site (a lot of their software is based on Globus Toolkit components under the covers).
• In 2005, the grid surpassed 100 sites (today, 177 are listed).
• There are some interesting visualizations on the LCG monitoring page.
• A good read is the September 2006 Report of the LHCC Comprehensive Review of LCG (10 pages).
• There’s a nice video about the accelerator featured by SEED magazine: Lords of the Ring.

The kernel inclusion of KVM makes me think about some virtualization adoption issues.

This much younger technology has made it into the kernel faster than Xen, is that important? Enabling virtualization support by using a vanilla Linux kernel configuration is important for grid computing mostly because it helps with resource provider adoption of virtualization capabilities which is probably the biggest impediment to VMs + grid computing right now.

But even though Xen has not made it into the mainline kernel (it’s been discussed for several years now, and has gone through some twists and turns), distribution packaging support for it is good because of the very strong demand for Xen.

Personally I think Xen is easy enough to install and deal with if you are already familiar with basic kernel configuration/installation. Its weakness in this area is when you need to diverge from the specific kernel versions the patches are geared to work against. It’s best to keep it as high up in the kernel dev chain as possible.

But overall, if you’ve already decided to support virtualization, the VMM’s features, cost, license, performance, and management options are much bigger factors than this. To me, it’s just a minor deployment detail — unless you have applications whose support contracts would be invalidated by switching kernels.

I would be interested to find out how much the lack of a native kernel option is really affecting administrators, especially if it is the one thing stopping them from running Xen — either because of support contract issues or “pain” issues (real or perceived).