Virtualization and Grid Computing

Tom Scavo announces some great news:

Today, it is with great pleasure that the GridShib Project announces the immediate release of GridShib for Globus Toolkit v0.6.0. This release culminates a 20-month effort to bring SAML-based attribute push to X.509-based Grids.

GridShib for Globus Toolkit (GT) is an implementation of a Grid Service Provider, an entity much like a SAML Service Provider but for Grids. A Grid Service Provider consumes X.509-bound SAML tokens, a new type of security token that enables attributed-based authorization in X.509-based Grids.

Most everything you need to know about GridShib for GT is on this web page:

http://gridshib.globus.org/docs/gridshib-gt-0.6.0/readme.html

On this readme page, you will find more detailed information about the GridShib for GT software as well as links to downloads and documentation.

A major advance in this version of GridShib for GT is support for the TeraGrid Science Gateway use case where an intermediary makes a grid request on behalf of a browser user. The Gateway binds a SAML token to an X.509 proxy certificate and makes a request to a gridshib-enabled web service. On the service side, GridShib for GT consumes the SAML token and makes an access control decision based on the security information in the token.

As a SAML-consuming software component, GridShib for GT complements the previously released GridShib SAML Tools and GridShib Certification Authority (CA), which are SAML-producing software components. These three components together enable attribute-based authorization in X.509-based Grids. See the Quick Start for step-by-step instructions that show how to use GridShib for GT v0.6, GridShib SAML Tools v0.3, and GridShib CA v0.5.1 together on Windows and UNIX systems:

http://gridshib.globus.org/docs/gridshib/quick-start.html

For links to all GridShib software downloads and additional documentation, visit the GridShib Downloads page:

http://gridshib.globus.org/download.html

Funding for GridShib software has been provided by the NSF NMI program and the NSF TeraGrid program.

Tom Scavo
For the entire GridShib Team

http://groups.google.com/group/cloud-computing

Virtual Workspaces Tutorial at Open Source Grid Cluster (May 12-16, 2008)

There will be a Virtual Workspaces tutorial at the Open Source Grid Cluster conference in Oakland, CA. The conference is May 12-16, 2008. The Virtualization and Cloud Computing with Globus session is on Wednesday, May 14th, from 4:30-6:00 pm. We hope to see you there!

Quoting from the summary:

One of the primary obstacles users face in grid computing is that Grids provide access to many diverse resources, their applications often require a very specific, customized environment. This disconnect can lead to resource underutilization, user frustration, and much wasted effort spent on bridging the gap between applications and resources. Virtual Workspaces describe the environment required for the execution of an application that can be dynamically deployed across a variety of resources creating a working and consistent platform for grid applications.

This tutorial will introduce the Globus Toolkit workspace service that implements workspaces as Xen virtual machines and enables authorized grid clients to dynamically deploy them and manage their resources. Further, we will describe and demonstrate the workspace “cloudkit” that provides a user-friendly interface on top of the workspace service allowing authorized users to easily provision and run VMs on the available community clouds. Finally, we will describe how the process of contextualization can be used to provide on-demand functioning clusters and give examples of its use by applications.

Interesting #13 here (well, they’re all interesting): 25 radical network research projects you should know about.

This points us to Cloud Control with Distributed Rate Limiting which is a paper about distributed bandwidth management.

From the conclusion:

As cloud-based services transition from marketing vaporware to real, deployed systems, the demands on traditional Web-hosting and Internet service providers are likely to shift dramatically. In particular, current models of resource provisioning and accounting lack the flexibility to effectively support the dynamic composition and rapidly shifting load enabled by the software as a service paradigm. We have identified one key aspect of this problem, namely the need to rate limit network traffc in a distributed fashion, and provided two novel algorithms to address this pressing need.

Check out the summary at networkworld but also here is an excerpt from a UCSD post about it:

If half your company’s bandwidth is allocated to your mirror in New York, and it’s the middle of the night there, and your sites in London and Tokyo are slammed, that New York bandwidth is going to waste. UC San Diego computer scientists have designed, implemented, and evaluated a new bandwidth management system for cloud-based applications capable of solving this problem.

The UCSD algorithm enables distributed rate limiters to work together to enforce global bandwidth rate limits, and dynamically shift bandwidth allocations across multiple sites or networks, according to current network demand.”

There’s been a lot of talk about the dangers of getting locked in to cloud platforms, developing an application that is only suited to one platform.

Here’s a, let’s say… “embellished” example: Gangsta cloud wars could pivot on the traffic-driving power of Google and Microsoft/Yahoo.

When you’re using VMs like Xen (e.g. on EC2), if you design things for it you “should be able to” move without a ton of hassle (research. plan.). The workspace project has been working on portability and usability (see The first one-click STAR production cluster) and one of the things we can do now is use the same VM image on a regular cluster (such as on the Teraport cloud) and EC2. The contextualization software can be configured to sense if it is on EC2 or not (and will bootstrap accordingly). It “would be nice” if such things were standardized but this is not a real problem right now (IMHO).

About something more “strongly typed” like Google’s AppEngine. Application migration might be a bit harder, but not if the APIs are well known and repeatable. Google’s SDK is even Apache 2 licensed.

To that point, have a look at Announcing AppDrop.com (host Google App Engine projects on EC2). It’s not there yet (database is a flat file) but, hey, it was developed in a few days. Cool. Read more at http://appdrop.com.

The long term idea is not that this would solve all your problems magically but that such things are possible, and if there’s a real market for choices, it seems like more work on things of this nature are also inevitable.

I’m no datacenter business expert, but the biggest problem right now seems to be that few people will be able to compete on costs/efficiencies of scale with Google/Amazon/Microsoft/eBay. (<predictions…>) It feels like it would naturally approach the straight web hosting business, though. Let’s say a standard, open source cloud computing infrastructure emerges (such as Apache httpd in the analogy). There will be various levels of players as far as the capital they have and certainly better and worse companies to choose from (including those that differentiate on service etc). But if you’re really sweating the savings an enormous company could provide with such efficiencies vs. a normal size company/datacenter, you’re probably at the point where you could save a whole lot more by buying your own computers.(</predictions…>)

Miscellaneous point about lock-in: something user-facing that ties you to a provider does not seem like a wise idea (e.g. Google’s Users API).

EC2 announced future support for adding raw, persistent block devices to VMs, a few non-Amazon people are even testing it already.

• Ability to create volumes between 1GB and 1TB
• Ability to create any FS on them after the first mount
• Mounted from same availability zone as the EC2 instance
• Snapshots to S3 (awesome)

See Werner Vogels and this RightScale post.

Old news, but here’s an interesting website: Online Home for the TeraGrid Planning Process. In particular, the Position Papers section.

They’ve taken the application level approach (Python currently).

And unlike Sun’s attempt (which also needed porting of app to a platform instead of the looser requirements of EC2 style), there is an interesting entry incentive:

“It’s free to get started. Every Google App Engine application can use up to 500MB of persistent storage and enough bandwidth and CPU for 5 million monthly page views.”

http://code.google.com/appengine/

http://code.google.com/appengine/docs/whatisgoogleappengine.html

http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html

http://appgallery.appspot.com/

http://groups.google.com/group/google-appengine