Virtualization and Grid Computing

Besides the good stuff added to Nimbus, this release also introduces something called the AutoContainer which allows you to set up a Globus Java web services environment, from scratch and with security working, within about a minute (requires Linux/OSX and Java 1.5+).

The main new features provided in this release are tools facilitating the deployment, configuration and management of clouds. We also updated our implementation to match the current Amazon EC2 deployment. In addition, the release contains new documentation and bug fixes.

You can download the new release from:
http://workspace.globus.org/downloads/index.html

The full changelog can be found here:
http://workspace.globus.org/vm/changelog.html#TP2.1

A lot of developments with the workspace service and science clouds recently!

The cluster technology lets you bootstrap generic images into new network and security contexts on the fly. We built a sample cluster on top of the technology that lets you create the cluster and be immediately ready to submit jobs to a Torque cluster fronted by GRAM and GridFTP that use a newly created self-signed certificate:

1. cloud-client.sh –run –hours 12 –cluster base-cluster.xml

2. Wait a few minutes, once launched note the head-node hostname

3. scp -r root@HOSTNAME:certs/*  lib/certs/

   (SSH was bootstrapped end to end already)

4. Make sure your grid tools trust this certificate and then submit work

This can be done with nearly anything that can run on a non-virtual cluster. Check out these links for more information:

• http://workspace.globus.org/clouds/clusters.html
• http://workspace.globus.org/vm/TP1.3.3/index.html

This Better Know a VM entry, Virtual Cluster Appliances, gives an overview of VM contextualization technology which is scheduled to be part of the next workspace service release. This is not just relevant to classic grid computing, but any situation where you’d like to automatically launch many virtual machines that work together and want them to securely organize themselves and adapt to the deployment environment. It can even be used for one VM, we’ll look at such cases later.

Instead of a single allocation, EC2 announced you can run several different kinds of instances.

See the EC2 home page for details:

$0.10 - Small Instance (Default)

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

In many cases it may be more cost effective to still get the small instance but just get a lot of them, this will be interesting for our workspace EC2 adapter and contextualization users (and us!). Once we make the small alterations to accomodate requesting these types, it will be just as easy to get 100 x small instance as 25 x large instance, or whatever combination, because deployment configurations can be coordinated on the fly. What would be best for what situation would have to be examined closely. An extra large instance for the virtual cluster head node(s) or storage/transfer node(s) could be extremely useful for the typical grid-cluster bottlenecks.

Quoting from workspace news:

The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources.

The 100 node cluster was composed of a headnode and workernodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology.

[UPDATE, related: http://www.gridvm.org/virtual-cluster-appliances.html]

[UPDATE, see: One-click clusters, VWS TP1.3.3]

It’s been busy lately, attended the first dev.Globus All Hands Meeting and TeraGrid ‘07 right here in Madison.

At TG07, Kate gave a talk which is online. The paper she presented discusses among other things contextualization, the structure and mechanisms by which an appliance/workspace is “told” what it needs in order to adapt to its deployed environment. This is not just adaptation to site specific services but also to other appliances that may be deployed with it such as in a virtual cluster deployment.

Amidst the bustle we implemented a new backend to the Workspace Service, to Amazon’s Elastic Compute Cloud (EC2). We’ve deployed it to the University of Chicago’s Teraport cluster and will currently pay for usage by selected collaborators.

Besides being somewhat fun to implement (including getting the Globus and Amazon Secure Message stacks on the same wavelength), I think it’s going to be interesting.

Because grid resources are cautiously approaching the pioneering switch to virtualizing resources [1], even in part, it is going to be interesting and educational to see what people will be able to accomplish with workspaces when a large pool of resources is actually available on tap — today.

Because the same deployment protocols can be used for both native and EC2 resources, there are of course capacity overflow use cases. In the right situations, VMs are a good mechanism for providers to dynamically reach more consumers as the need arises.

For a feature list and description, see What is the EC2 backend?

——-

[1] and some would say inevitable switch, even with the performance costs. Consider also that ‘virtualizing resources’ may mean physical node re-imaging, cf. Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid.

Our paper about virtual appliance configuration and management was accepted to the TeraGrid 2007 conference and is now online: A Scalable Approach To Deploying And Managing Appliances.

This paper examines configuration and security issues that large and heterogeneous deployments of virtual appliances/workspaces will face.

From the introduction:

The goal of this paper is to develop a holistic approach that would provide scalable and sustainable ways of managing and deploying virtual workspaces implemented as VM images. We will discuss ways of leveraging existing configuration management tools, exemplified by the Bcfg2 system, for VM image lifecycle management that will allow systems staff to deploy robust virtualized resources for their users. We will also describe the process of contextualization — integration of an appliance in its deployment context — and discuss its reference implementation using Bcfg2 and the Workspace Service.

Add “On Appliance” to Your “On Premise” and “On Demand” Strategies

I saw this session in person, it was good. I don’t need convincing of the benefits that the software/virtual appliance model brings: I think the benefits exist for many scenarios and they most all stem from the presence of a strict separation of complexity concerns (not a reduction in overall complexity of software). But hearing what the panelists had to say about deployment scenarios and challenges in the commercial software business world was good for me.

I liked the idea of using a VM deployed at a customer’s site to be a sort of customer facing front-end in a SaaS model — a hybrid of over-the-WAN “on-demand” and “on-premise”. This session also moved me to evaluate Zimbra which looks pretty helpful for replacing some mail/collaboration daemons I run (though moving to their clients full time and abandoning Sylpheed/Claws seems unlikely).