Virtualization and Grid Computing

Cool to see the mainstream press (Newsweek) mention Nimbus: Number Crunching Made Easy.

Just ran across an “interesting” definition of Nimbus:

“Client-side cloud-computing interface to Globus-enabled TeraPort cluster at U of C”

… in these slides: http://eucalyptus.cs.ucsb.edu/documents/eucalyptus-slides-wolski-cloud_expo_apr08.ppt

There is a Nimbus client (which can be replaced by an Amazon’s EC2 client), true. But most of Nimbus is server side software.

Teraport (and other Science Clouds) are not “globus enabled” but rather host the server-side components of Nimbus. Nimbus converts a set of hypervisors into what some will call an “IaaS cloud” or “open source EC2″ (Nimbus was released before EC2 but EC2 protocol support has been added due to customer demand).

The remote messaging modules (EC2 and a separate WS system) are hosted in a container that is based on Axis — some Globus Toolkit components also use that. This is a thin layer that provides marshalling/unmarshalling and security, converting messages to a common format for use with the framework independent “meat” of the Nimbus service (it could be hosted in another container).

See the FAQ, publications, and news if you would like to learn more about Nimbus.

Press Release: Nimbus and Cloud Computing Meet STAR Production Demands

We’ve been running self-configuring 100+ node clusters on EC2 since 2007, but I would be remiss if I did not link to this announcement.

The Nimbus TP2.2 release provides a standalone context broker that can be used across Nimbus and EC2 clouds and continues our work on EC2 compatibility with the introduction of EC2 metadata server. In addition, the release contains new documentation and bug fixes.

See the changelog for all the details.

It was a good feeling to pause for a moment and put a user quotes page together for Nimbus. We’ve worked hard to make Nimbus usable and useful — but the best is yet to come!

Besides the good stuff added to Nimbus, this release also introduces something called the AutoContainer which allows you to set up a Globus Java web services environment, from scratch and with security working, within about a minute (requires Linux/OSX and Java 1.5+).

The main new features provided in this release are tools facilitating the deployment, configuration and management of clouds. We also updated our implementation to match the current Amazon EC2 deployment. In addition, the release contains new documentation and bug fixes.

You can download the new release from:
http://workspace.globus.org/downloads/index.html

The full changelog can be found here:
http://workspace.globus.org/vm/changelog.html#TP2.1

It’s thrilling to organize things better.

Quoting from the Nimbus features page:

There are currently two supported remote protocol sets:

• WSRF based: protocol implementation in longstanding use by previous workspace services and clients including the cloud-client.
• EC2 based: clients written for EC2 can be used with Nimbus installations. For more information, see What is the EC2 frontend?

These protocols happen to both be Web Services based and both run in the Apache Axis based GT Java container. But neither thing is a necessity:

• There is nothing specific to web services based remote protocols in the workspace service implementation, the messaging system just needs to be able to speak to Java based libraries.
• Workspace service dependencies have nothing to do with what container it is running in, they are more “straight Java” style dependencies like Spring, ehcache, backport-util-concurrent, and JDBC (currently using the embedded Derby database).

See the announcement: new strong internal interfaces and a new remote protocol implementation (compatible with EC2 clients) that can run alongside the WSRF based ones.

Check out the changelog and the new FAQ.

Go Ask ALICE, the iSGTW image of the week. (Funny headline, see Go_Ask_Alice).

Check out some screenshots here of Nimbus resources invovled in supporting this experiment. It’s a small part of things as you can see from the scope of the grid but exciting nonetheless. The AliEn based virtual cluster is now “one-click” and can be launched anywhere running a workspace cloud setup.

VWS RSS feed:

• http://workspace.globus.org/rss.xml

A lot of developments with the workspace service and science clouds recently!

The cluster technology lets you bootstrap generic images into new network and security contexts on the fly. We built a sample cluster on top of the technology that lets you create the cluster and be immediately ready to submit jobs to a Torque cluster fronted by GRAM and GridFTP that use a newly created self-signed certificate:

1. cloud-client.sh –run –hours 12 –cluster base-cluster.xml

2. Wait a few minutes, once launched note the head-node hostname

3. scp -r root@HOSTNAME:certs/*  lib/certs/

   (SSH was bootstrapped end to end already)

4. Make sure your grid tools trust this certificate and then submit work

This can be done with nearly anything that can run on a non-virtual cluster. Check out these links for more information:

• http://workspace.globus.org/clouds/clusters.html
• http://workspace.globus.org/vm/TP1.3.3/index.html

From http://www.gridtoday.com/grid/2370981.html

rPath Enables Cloud Computing for DoE, CERN

RALEIGH, N.C., June 4 — rPath, whose unique technology simplifies application distribution and management through virtual appliances, today announced that the U.S. Department of Energy (DOE) and the European Organization for Nuclear Research (CERN) have been using rBuilder to deliver virtual appliances to both scientists’ desktops and computational clouds. The use of rBuilder in these environments reduces the effort required to support users and allows researchers to take advantage of underutilized computational resources.

rBuilder is the first and only product that simplifies and automates the creation of virtual appliances. A virtual appliance is an application with a streamlined operating system, offered in a format that runs in virtualized environments.

CERN turned to virtual appliances to facilitate the analysis of data created by the Large Hadron Collider (LHC) experiments. The complete software environment needed by the LHC applications is assembled by rBuilder and distributed to run as a virtual machine on physicists’ desktops. Virtual appliances provide a consistent application environment for the LHC applications while, at the same time, allowing scientists to use their desktops for analysis, regardless of operating system.

“The coupling between the LHC applications and the operating system is very strong,” stated Predrag Buncic, virtualization R&D project leader. “By distributing these applications as virtual appliances, we are able to isolate the application from the underlying desktop or laptop operating system, allowing the researchers to run the applications on systems that normally would not be supported.”

The DOE is exploring the concept of using virtual appliances to provide customized environments for scientific applications. Scientific applications are turned into virtual appliances using rPath’s rBuilder. The “Science Clouds” project (http://workspace.globus.org/clouds) provides resources capable of hosting multiple scientific appliances using the Globus Virtual Workspaces software. Scientists submit their virtual appliances to any available resource, knowing that the application environment is controlled and isolated from the underlying system. By relying on portable appliances, the scientists can leverage the resources of science clouds, and seamlessly move to commercial providers, such as Amazon’s EC2, when additional resources are needed.

“For a proof-of-concept, anybody can just configure a virtual machine image by hand,” said Kate Keahey, a scientist at Argonne National Laboratory. “But providing appliance management and maintenance that will scale to many thousands of appliances and that will be truly interoperable between different resource providers requires a new approach.”

About rPath

For application providers that want to accelerate license growth, expand into new markets, and reduce support and development costs, rPath’s platform transforms applications into virtual appliances. A virtual appliance is an application combined with just enough operating system (JeOS) for it to run optimally in any virtualized environment. Virtual appliances eliminate the hassles of installing, configuring and maintaining complex application environments. Only rPath’s technology simplifies application distribution, lowers the customer service costs of maintenance and management, and produces multiple virtual machine formats. The company is headquartered in Raleigh, N.C. For more information, visit www.rpath.com.

—–

Source: rPath