Virtualization and Grid Computing

If you’re feeling adventurous, there’s a workspace service pre-release out (click on the pic):

Tom Scavo announces some great news:

Today, it is with great pleasure that the GridShib Project announces the immediate release of GridShib for Globus Toolkit v0.6.0. This release culminates a 20-month effort to bring SAML-based attribute push to X.509-based Grids.

GridShib for Globus Toolkit (GT) is an implementation of a Grid Service Provider, an entity much like a SAML Service Provider but for Grids. A Grid Service Provider consumes X.509-bound SAML tokens, a new type of security token that enables attributed-based authorization in X.509-based Grids.

Most everything you need to know about GridShib for GT is on this web page:

http://gridshib.globus.org/docs/gridshib-gt-0.6.0/readme.html

On this readme page, you will find more detailed information about the GridShib for GT software as well as links to downloads and documentation.

A major advance in this version of GridShib for GT is support for the TeraGrid Science Gateway use case where an intermediary makes a grid request on behalf of a browser user. The Gateway binds a SAML token to an X.509 proxy certificate and makes a request to a gridshib-enabled web service. On the service side, GridShib for GT consumes the SAML token and makes an access control decision based on the security information in the token.

As a SAML-consuming software component, GridShib for GT complements the previously released GridShib SAML Tools and GridShib Certification Authority (CA), which are SAML-producing software components. These three components together enable attribute-based authorization in X.509-based Grids. See the Quick Start for step-by-step instructions that show how to use GridShib for GT v0.6, GridShib SAML Tools v0.3, and GridShib CA v0.5.1 together on Windows and UNIX systems:

http://gridshib.globus.org/docs/gridshib/quick-start.html

For links to all GridShib software downloads and additional documentation, visit the GridShib Downloads page:

http://gridshib.globus.org/download.html

Funding for GridShib software has been provided by the NSF NMI program and the NSF TeraGrid program.

Tom Scavo
For the entire GridShib Team

Virtual Workspaces Tutorial at Open Source Grid Cluster (May 12-16, 2008)

There will be a Virtual Workspaces tutorial at the Open Source Grid Cluster conference in Oakland, CA. The conference is May 12-16, 2008. The Virtualization and Cloud Computing with Globus session is on Wednesday, May 14th, from 4:30-6:00 pm. We hope to see you there!

Quoting from the summary:

One of the primary obstacles users face in grid computing is that Grids provide access to many diverse resources, their applications often require a very specific, customized environment. This disconnect can lead to resource underutilization, user frustration, and much wasted effort spent on bridging the gap between applications and resources. Virtual Workspaces describe the environment required for the execution of an application that can be dynamically deployed across a variety of resources creating a working and consistent platform for grid applications.

This tutorial will introduce the Globus Toolkit workspace service that implements workspaces as Xen virtual machines and enables authorized grid clients to dynamically deploy them and manage their resources. Further, we will describe and demonstrate the workspace “cloudkit” that provides a user-friendly interface on top of the workspace service allowing authorized users to easily provision and run VMs on the available community clouds. Finally, we will describe how the process of contextualization can be used to provide on-demand functioning clusters and give examples of its use by applications.

There’s been a lot of talk about the dangers of getting locked in to cloud platforms, developing an application that is only suited to one platform.

Here’s a, let’s say… “embellished” example: Gangsta cloud wars could pivot on the traffic-driving power of Google and Microsoft/Yahoo.

When you’re using VMs like Xen (e.g. on EC2), if you design things for it you “should be able to” move without a ton of hassle (research. plan.). The workspace project has been working on portability and usability (see The first one-click STAR production cluster) and one of the things we can do now is use the same VM image on a regular cluster (such as on the Teraport cloud) and EC2. The contextualization software can be configured to sense if it is on EC2 or not (and will bootstrap accordingly). It “would be nice” if such things were standardized but this is not a real problem right now (IMHO).

About something more “strongly typed” like Google’s AppEngine. Application migration might be a bit harder, but not if the APIs are well known and repeatable. Google’s SDK is even Apache 2 licensed.

To that point, have a look at Announcing AppDrop.com (host Google App Engine projects on EC2). It’s not there yet (database is a flat file) but, hey, it was developed in a few days. Cool. Read more at http://appdrop.com.

The long term idea is not that this would solve all your problems magically but that such things are possible, and if there’s a real market for choices, it seems like more work on things of this nature are also inevitable.

I’m no datacenter business expert, but the biggest problem right now seems to be that few people will be able to compete on costs/efficiencies of scale with Google/Amazon/Microsoft/eBay. (<predictions…>) It feels like it would naturally approach the straight web hosting business, though. Let’s say a standard, open source cloud computing infrastructure emerges (such as Apache httpd in the analogy). There will be various levels of players as far as the capital they have and certainly better and worse companies to choose from (including those that differentiate on service etc). But if you’re really sweating the savings an enormous company could provide with such efficiencies vs. a normal size company/datacenter, you’re probably at the point where you could save a whole lot more by buying your own computers.(</predictions…>)

Miscellaneous point about lock-in: something user-facing that ties you to a provider does not seem like a wise idea (e.g. Google’s Users API).

Old news, but here’s an interesting website: Online Home for the TeraGrid Planning Process. In particular, the Position Papers section.

This month’s OGF newsletter has an article about the Cloud Systems BoF.

If you scroll down to the bottom of the latter link, there are slides and PDFs to view.

The mailing list URL in the newsletter is currently broken, this is the right one: http://www.ogf.org/mailman/listinfo/clouds-bof

Globus Toolkit 4.0.7 now available:

On behalf of the Globus Toolkit development team I am pleased to announce that a new incremental release of GT4 is now available for download. GT4.0.7 is recommended for all users. It was released because of bug 5910, a potential RFT data corruption bug. The bug affected only GT4.0.6, and users of GT4.0.6 can apply the update package from http://www.globus.org/toolkit/advisories.html. New users are encouraged to start with the 4.0.7 release, as other bugs were also fixed as listed in the release notes.

Relevant 4.0.7 links:

• Release notes: http://www.globus.org/toolkit/releasenotes/4.0.7/
• Software: http://www.globus.org/toolkit/downloads/4.0.7/
• Documentation: http://www.globus.org/toolkit/docs/4.0/

Thanks for your support of Globus software!

Cheers,

Charles

If you’re on the workspace-announce list, you will have already seen the “Science Cloud Available at the University of Chicago” email.

Built with the workspace service, we’ve made some nice client enhancements to get to “cloud simplicity” and it’s up and running on 16 nodes and already serving guests. See the the documentation for command samples, the idea is to make it as simple as possible. On the service side, Nimbus uses TP1.3.1 with some very small additions (mostly this differs because of a new authorization plugin). Building cloud computing solutions is the main business of the workspace service.

Have a look!

[UPDATE: using TP1.3.3.1 now which enables one-click clusters]

We are pleased to announce GridShib SAML Tools v0.3.0, the final release in the v0.3.0 development cycle:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/readme.html

http://gridshib.globus.org/download.html#saml-tools

The GridShib SAML Tools are a suite of standalone client tools that issue SAML assertions and optionally bind these assertions to X.509 proxy certificates. To try out the software before downloading, visit our online demo:

https://computer.ncsa.uiuc.edu/gst-demo/

The GridShib SAML Tools require only Java 1.4 (or later) and Ant 1.6 (or later). Proxy certificates issued by the SAML Tools are compatible with GridShib for Globus Toolkit v0.6.0 Alpha (or later).

There have been significant changes in this version of the GridShib SAML Tools since the previous release:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/CHANGES.txt

Important new features of GridShib SAML Tools v0.3.0 include:

• enhanced command-line interface
• new command-line options for the SAML Assertion Issuer Tool, including the option to output a DER-encoded ASN.1 structure
• new X.509 Binding Tool, to bind arbitrary content to a non-critical extension of an X.509 proxy certificate
• new SAML Security Info Tool, for examining the contents of X.509-bound SAML tokens
• expanded Java API, for producing and consuming SAML assertions and X.509 proxy certificates
• support for the TeraGrid Science Gateway Use Case

This development cycle was largely driven by the TeraGrid Science Gateway Use Case:

http://gridshib.globus.org/docs/gridshib-saml-tools-0.3.0/teragrid/readme.html

Science Gateways use the SAML Tools to enable auditing, incident response, and access control in Globus-based grids.

To learn more about this and other use cases, visit the “About GridShib” page:

http://gridshib.globus.org/about.html

While the GridShib SAML Tools produce X.509-bound SAML tokens, the complementary software component GridShib for Globus Toolkit consumes them. The latter is scheduled for release later this month or early next. See the roadmap on the GridShib home page for the latest updates.

Tom Scavo
For the GridShib Team

Some cool new features:

On behalf of the workspace team, I am happy to announce the TP 1.3.1 release of the Workspace Service. You can download the new release from: http://workspace.globus.org/downloads/index.html

The main new feature in this release is the implementation of the workspace pilot which provides non-invasive adaptations to batch schedulers (such as PBS) enabling sites to run virtual machines alongside jobs. The details of this approach are described in: http://workspace.globus.org/papers/workspace-pilot-paper-submitted.pdf

In addition, the release also contains the ensemble service that allows clients to create ensembles of heterogeneous virtual machines to be deployed and managed together, improvements to the client, and several bug fixes. The complete changelog can be found at: http://workspace.globus.org/vm/TP1.3.1/index.html#changelog

We welcome comments, feedback, and bug reports. Information about the project, software downloads, documentation and instructions on how to join the workspace-user mailing list for support questions can be found at: http://workspace.globus.org

Happy Valentine’s Day!

As you can read there, the main new feature is the pilot infrastructure. The paper Kate refers to in the announcement is a relatively short read and lays out the ideas (and a practical evaluation) in an organized way. But briefy: the pilot is a program the service will submit to a local site resource manager in order to obtain time on the VMM nodes. When not allocated to the workspace service, these nodes will be used for jobs as normal. Those jobs run in normal system accounts in Xen domain 0 with no guest VMs running.

Importantly, the approach leaves the site resource manager in full control of the nodes and requires no modifications to the site resource manager. Save perhaps possible configuration changes you might like to make. For example, you can mark particular nodes as able to accomodate guest VMs: the workspace service supports sending pilot requests to particular LRM queues, or providing a particular node property etc. This allows you to really organize not just when but where VMs can run.

Several extra safeguards have been added to make sure the node is returned from VM hosting mode at the proper time, including support for:

• the workspace service being down or malfunctioning
• LRM preemption (including deliberate LRM job cancellation)
• node reboot/shutdown

Also included is a one-command “kill 9″ facility for administrators as a “worst case scenario” contingency.

So as a buzzword experiment, I want to put in a particular keyword here and see how the search engine hits work out :-). I think you know what it may be…

Cloud computing

Go make a cloud!

And with the workspace pilot, you won’t have to switch over all at once. Take it for a test run and tell us about it on workspace-user.

We’ve got some exciting stuff in the pipeline for the next few months, too (see the last release announcement and the self-configuring 100 node VM cluster news). I am really happy with where the project is going and has been recently.

- Tim

Amazon SQS is a distributed message queue system with a simple, robust API and real infrastructure to back it. And their prices just dropped significantly from a penny per 100 requests to a penny per 10,000:

Dear Amazon SQS Developers,

We wanted to let you know about some changes we are making to Amazon SQS, based on customer feedback and watching the way customers are using the service. One thing we’ve heard consistently is that customers want to be able to use SQS along with our other services (e.g. Amazon EC2, Amazon S3), but need SQS to be less expensive for this to be more feasible. We looked at our architecture and feature set, and found a way to make a few, targeted changes, by deprecating a few infrequently used requests, which allow us to operate the service much more efficiently. Simultaneously, we are introducing a new pricing structure that replaces the previous per-messages-sent charge ($0.10/1,000 messages) with a new per-request fee ($0.01/10,000 requests, including all Amazon SQS operations). The net result is that the new pricing will result in significantly lower charges for most developers being billed for SQS.

I’m hoping we’ll look back in five years and reminisce about how they charged so much for EC2 as well :-) (I do think it’s a good price now unless you are looking to continually use many, many computers).

Globus Toolkit 4.0.6 has been released!

On behalf of the Globus Toolkit development team I am pleased to announce that a new incremental release of GT4 is now available for download. Users who wish to receive the latest bug fixes are encouraged to install this release. The list of bugs fixed by this release is available in the release notes below.

Relevant 4.0.6 links:

• Release notes: http://www.globus.org/toolkit/releasenotes/4.0.6/
• Software: http://www.globus.org/toolkit/downloads/4.0.6/
• Documentation: http://www.globus.org/toolkit/docs/4.0/

Thanks for your support of Globus software!

Cheers,
Charles