Virtualization and Grid Computing

Cloud Computing and Its Applications

Also see Ian Foster’s blog entry

Interesting #13 here (well, they’re all interesting): 25 radical network research projects you should know about.

This points us to Cloud Control with Distributed Rate Limiting which is a paper about distributed bandwidth management.

From the conclusion:

As cloud-based services transition from marketing vaporware to real, deployed systems, the demands on traditional Web-hosting and Internet service providers are likely to shift dramatically. In particular, current models of resource provisioning and accounting lack the flexibility to effectively support the dynamic composition and rapidly shifting load enabled by the software as a service paradigm. We have identified one key aspect of this problem, namely the need to rate limit network traffc in a distributed fashion, and provided two novel algorithms to address this pressing need.

Check out the summary at networkworld but also here is an excerpt from a UCSD post about it:

If half your company’s bandwidth is allocated to your mirror in New York, and it’s the middle of the night there, and your sites in London and Tokyo are slammed, that New York bandwidth is going to waste. UC San Diego computer scientists have designed, implemented, and evaluated a new bandwidth management system for cloud-based applications capable of solving this problem.

The UCSD algorithm enables distributed rate limiters to work together to enforce global bandwidth rate limits, and dynamically shift bandwidth allocations across multiple sites or networks, according to current network demand.”

Old news, but here’s an interesting website: Online Home for the TeraGrid Planning Process. In particular, the Position Papers section.

This month’s OGF newsletter has an article about the Cloud Systems BoF.

If you scroll down to the bottom of the latter link, there are slides and PDFs to view.

The mailing list URL in the newsletter is currently broken, this is the right one: http://www.ogf.org/mailman/listinfo/clouds-bof

Some cool new features:

On behalf of the workspace team, I am happy to announce the TP 1.3.1 release of the Workspace Service. You can download the new release from: http://workspace.globus.org/downloads/index.html

The main new feature in this release is the implementation of the workspace pilot which provides non-invasive adaptations to batch schedulers (such as PBS) enabling sites to run virtual machines alongside jobs. The details of this approach are described in: http://workspace.globus.org/papers/workspace-pilot-paper-submitted.pdf

In addition, the release also contains the ensemble service that allows clients to create ensembles of heterogeneous virtual machines to be deployed and managed together, improvements to the client, and several bug fixes. The complete changelog can be found at: http://workspace.globus.org/vm/TP1.3.1/index.html#changelog

We welcome comments, feedback, and bug reports. Information about the project, software downloads, documentation and instructions on how to join the workspace-user mailing list for support questions can be found at: http://workspace.globus.org

Happy Valentine’s Day!

As you can read there, the main new feature is the pilot infrastructure. The paper Kate refers to in the announcement is a relatively short read and lays out the ideas (and a practical evaluation) in an organized way. But briefy: the pilot is a program the service will submit to a local site resource manager in order to obtain time on the VMM nodes. When not allocated to the workspace service, these nodes will be used for jobs as normal. Those jobs run in normal system accounts in Xen domain 0 with no guest VMs running.

Importantly, the approach leaves the site resource manager in full control of the nodes and requires no modifications to the site resource manager. Save perhaps possible configuration changes you might like to make. For example, you can mark particular nodes as able to accomodate guest VMs: the workspace service supports sending pilot requests to particular LRM queues, or providing a particular node property etc. This allows you to really organize not just when but where VMs can run.

Several extra safeguards have been added to make sure the node is returned from VM hosting mode at the proper time, including support for:

• the workspace service being down or malfunctioning
• LRM preemption (including deliberate LRM job cancellation)
• node reboot/shutdown

Also included is a one-command “kill 9″ facility for administrators as a “worst case scenario” contingency.

So as a buzzword experiment, I want to put in a particular keyword here and see how the search engine hits work out :-). I think you know what it may be…

Cloud computing

Go make a cloud!

And with the workspace pilot, you won’t have to switch over all at once. Take it for a test run and tell us about it on workspace-user.

We’ve got some exciting stuff in the pipeline for the next few months, too (see the last release announcement and the self-configuring 100 node VM cluster news). I am really happy with where the project is going and has been recently.

- Tim

This Better Know a VM entry, Virtual Cluster Appliances, gives an overview of VM contextualization technology which is scheduled to be part of the next workspace service release. This is not just relevant to classic grid computing, but any situation where you’d like to automatically launch many virtual machines that work together and want them to securely organize themselves and adapt to the deployment environment. It can even be used for one VM, we’ll look at such cases later.

http://www.utexas.edu/oncampus/2007/11/15/tacc-feature/

The Texas Advanced Computing Center (TACC) recently announced its partnership with the World Community Grid. It will assist the project by running World Community Grid software on its employee PCs, installing the client on the new Stampede cluster –helping scientists scale their research for the World Community Grid – and allowing other large TACC clusters to run Grid computations when there are idle processors.
[…]
“We look forward to working with IBM to explore how researchers can most effectively utilize both TACC advanced systems and the World Community Grid to address problems with deep impact to society as well as science.”

Kate will give a talk today: 10:30-11:00am at the AIST booth. Go check it out!

(In the future, you should be able to find the slides here).

I just ran across these GCE07 papers online, looks like some good things to read about (gateway authorization and GridTorrent papers look especially interesting). GCE == “Grid Computing Environments” (this workshop is colocated with SC in Reno next week).

Borja Sotomayor, my esteemed colleague, has an interesting article No CPU Left Behind about VMs and education. Check it out.

The date of the VTDC 2007 workshop has been confirmed. It will be on Monday, November 12th. See you there!

Last year’s International Workshop on Virtualization Technology in Distributed Computing (VTDC) was an interesting and productive day and an exciting complement to Supercomputing.

This year’s call for papers has been announced!

===============================================================
CALL FOR PAPERS (VTDC 2007)

Workshop on Virtualization Technologies in Distributed Computing

held in conjunction with SC 07, the International Conference for High Performance Computing, Networking and Storage.
===============================================================

date: November, 2007, Reno, NV, USA

SC 07: http://sc07.supercomputing.org/
VTDC 07: http://workspace.globus.org/vtdc07/
Last year’s workshop: http://workspace.globus.org/vtdc06/

General Information

The convergence of virtualization technologies and distributed computing is an exciting development and the subject of much research in both academia and industry. The VTDC workshop is intended to be a forum for the exchange of ideas and experiences on the use of virtualization technologies in distributed computing, the challenges and opportunities offered by the development of virtual systems themselves, as well as case studies of application of virtualization. The scope of “virtualization technologies” includes techniques and concepts to enable virtual machines, virtual networks, virtual data, virtual storage, virtual applications and virtual instruments. The scope of “distributed computing” includes Grid-computing, cluster computing, peer-to-peer computing and mobile computing.

VTDC 2007 topics of interest include, but are not limited to:

• Using virtualization technologies for resource management and QoS assurance
• Security aspects of using virtualization in a distributed environment
• Virtual networks
• Virtual data and storage systems
• Fault tolerance in virtualization
• Virtualization in P2P
• Monitoring techniques in virtualization
• Virtualization-based adaptive/autonomic systems
• Virtual datacenters
• Virtual environment factories and services
• Environment configuration
• Virtual machine management
• Modeling (applications and systems)
• Case studies of applications using virtual technologies
• Deployment studies of virtualization technologies
• Tools relevant to virtualization
• Virtualization as vehicle for outsourcing

Paper Submission

Authors are invited to submit original and unpublished work that exposes a new problem, advocates a specific solution, or reports on actual experience.

Papers should be submitted as full-length 8 page papers of double column text using single spaces 10pt size type on an 8.5″x11″ paper, as per IEEE manuscript guidelines. Paper submission instructions will be placed on the workshop web page at http://workspace.globus.org/vtdc07/.

Presentations will be invited based on the originality, technical merit, and topical relevance of their submissions. Please contact vtdc07@mcs.anl.gov with questions.

Important dates:

September 24, 2007 - Paper submission
October 16, 2007 - Notification of acceptance
October 26, 2007 - Final version due