Virtualization and Grid Computing

Cool to see the mainstream press (Newsweek) mention Nimbus: Number Crunching Made Easy.

Just ran across an “interesting” definition of Nimbus:

“Client-side cloud-computing interface to Globus-enabled TeraPort cluster at U of C”

… in these slides: http://eucalyptus.cs.ucsb.edu/documents/eucalyptus-slides-wolski-cloud_expo_apr08.ppt

There is a Nimbus client (which can be replaced by an Amazon’s EC2 client), true. But most of Nimbus is server side software.

Teraport (and other Science Clouds) are not “globus enabled” but rather host the server-side components of Nimbus. Nimbus converts a set of hypervisors into what some will call an “IaaS cloud” or “open source EC2″ (Nimbus was released before EC2 but EC2 protocol support has been added due to customer demand).

The remote messaging modules (EC2 and a separate WS system) are hosted in a container that is based on Axis — some Globus Toolkit components also use that. This is a thin layer that provides marshalling/unmarshalling and security, converting messages to a common format for use with the framework independent “meat” of the Nimbus service (it could be hosted in another container).

See the FAQ, publications, and news if you would like to learn more about Nimbus.

Press Release: Nimbus and Cloud Computing Meet STAR Production Demands

We’ve been running self-configuring 100+ node clusters on EC2 since 2007, but I would be remiss if I did not link to this announcement.

The Nimbus TP2.2 release provides a standalone context broker that can be used across Nimbus and EC2 clouds and continues our work on EC2 compatibility with the introduction of EC2 metadata server. In addition, the release contains new documentation and bug fixes.

See the changelog for all the details.

It was a good feeling to pause for a moment and put a user quotes page together for Nimbus. We’ve worked hard to make Nimbus usable and useful — but the best is yet to come!

Besides the good stuff added to Nimbus, this release also introduces something called the AutoContainer which allows you to set up a Globus Java web services environment, from scratch and with security working, within about a minute (requires Linux/OSX and Java 1.5+).

The main new features provided in this release are tools facilitating the deployment, configuration and management of clouds. We also updated our implementation to match the current Amazon EC2 deployment. In addition, the release contains new documentation and bug fixes.

You can download the new release from:
http://workspace.globus.org/downloads/index.html

The full changelog can be found here:
http://workspace.globus.org/vm/changelog.html#TP2.1

All this Windows cloud news afoot. Make sure to read this post on the GoGrid blog:

Windows in the Cloud? Been there, done that!

GoGrid has already almost a year of proven experience providing Windows Server 2003/2008 to end users… we are also a Microsoft Gold Certified Partner.

;-)

Quoting from http://gist.github.com/7263

#!/usr/bin/env ruby
# gcalc  / 26aug2008 chneukirchen / public domain
require ‘open-uri’
q = ARGV.join(” “).gsub(/./m) { “%%%02X” % $&[0] }
html = open(”http://www.google.com/search?q=#{q}”).read
puts (html[%r{(.*?)}, 1] || ‘not a calculation’).
gsub(%r{240| }, “”).gsub(”×”, “x”).
gsub(%r{(.*?)}, ‘^\1 ‘) 

(tip via HubbuH)

An interesting project I ran across, it started in 2004.

From http://vcl.ncsu.edu/:

The Virtual Computing Lab (VCL) is a remote access service that allows you to reserve a computer with a desired set of applications for yourself, and remotely access it over the Internet.

You can use all your favorite applications such as Matlab, Maple, SAS, Solidworks, and many others. Linux, Solaris and numerous Windows environments are now available to all NC State students and faculty.

Leasing custom environments to “public-ish” users via PXE or similar technology was happening in other places in 2004, but I never saw anything at this scale.

It is clear that some kind of reconfiguration/resetting happens:

What rights do I have on the VCL machine?

On custom Windows and Linux environments you have adminstrative and root level rights. Since the VCL system reloads each expired reservations with a clean environment, there is no threat of any residual data being left on a machine for the next user.

On Linux and Solaris Lab machine environments, you only have user level rights. The same premissions as you would experience at the console of a walk-in lab.

I wonder when they added the VM support mentioned at http://vcl.ncsu.edu/help/general-information/how-it-works:

The management nodes each control a subset of the VCL resources. These can be blades, virtual machines or lab machines. Currently, a set of individual blades or virtual machines can only be managed by a single management node. Typically there are anywhere from 80-120 physical computer nodes (blades) under one management node. Again the physical computer nodes can either be running a bare metal environment or a Virtual Machine hypervisor.

Here are deployment stats captured on Aug 25, 2008:

• Total blades online: 438
• Total blades offline: 87
• Active Reservations: 49

Cool.

It’s thrilling to organize things better.

Quoting from the Nimbus features page:

There are currently two supported remote protocol sets:

• WSRF based: protocol implementation in longstanding use by previous workspace services and clients including the cloud-client.
• EC2 based: clients written for EC2 can be used with Nimbus installations. For more information, see What is the EC2 frontend?

These protocols happen to both be Web Services based and both run in the Apache Axis based GT Java container. But neither thing is a necessity:

• There is nothing specific to web services based remote protocols in the workspace service implementation, the messaging system just needs to be able to speak to Java based libraries.
• Workspace service dependencies have nothing to do with what container it is running in, they are more “straight Java” style dependencies like Spring, ehcache, backport-util-concurrent, and JDBC (currently using the embedded Derby database).

See the announcement: new strong internal interfaces and a new remote protocol implementation (compatible with EC2 clients) that can run alongside the WSRF based ones.

Check out the changelog and the new FAQ.

Go Ask ALICE, the iSGTW image of the week. (Funny headline, see Go_Ask_Alice).

Check out some screenshots here of Nimbus resources invovled in supporting this experiment. It’s a small part of things as you can see from the scope of the grid but exciting nonetheless. The AliEn based virtual cluster is now “one-click” and can be launched anywhere running a workspace cloud setup.