Virtualization and Grid Computing

See Keir’s mail: ANNOUNCE: Xen 3.1.1 released!, this is a bugfix release.

Binaries aren’t up yet.

You’ve surely heard already that Citrix acquired XenSource.

The Xen guys I’ve met I’ve liked, I’m happy for them. Xen as an open source project and XenSource as its steward are supposed to live on through this. This could really happen given Ian Pratt’s (and Xen/XenSource as a whole) staunch take on avoiding “business infection” so far w/ the XenSource launch.

Congratulations Xen developers!

Folks,

We’re pleased to announce the official release of Xen 3.1!

This represents a major milestone in the Xen project containing performance and stability enhancements, additional features for all architectures, and a brand new management API. Highlights of this release include:

* XenAPI 1.0 support
- XML configuration files for virtual machines;
- VM life-cycle management operations; and
- Secure on- or off-box XML-RPC with bindings for many languages
* Preliminary save/restore/migrate support for HVM (e.g. Windows) VMs;
* Dynamic memory control for HVM guests;
* 32-on-64 PV guest support (run PAE PV VMs on a 64-bit Xen!); and
* Blktap copy-on-write disk support.

You can get the source using mercurial from:
http://xenbits.xensource.com/xen-3.1-testing.hg

Source and binary tarballs, and RPMs, are available from:
http://www.xensource.com/download/index_oss.html

Cheers,
Keir (on behalf of the whole Xen dev team)

Nicholas Carr has commentary on Amazon’s S3 service here: Amazon re-prices S3.

One of the interesting things is the statement by Jeff Bezos that EC2 is “completely capacity-constrained” right now.

This is good news for workspaces and utility computing in general: people are buying into the VM-as-compute-capsule model in practice (workspaces are a lot like EC2, the biggest difference being the need for your own resources).

Keir Fraser writes on xen-devel:

The imminent next Xen release introduces a host of important new features including PV 32-on-64, HVM save/restore, and XenAPI 1.0. Now is a good time to bump our version number and reclaim the redundant second digit!

We plan to rename the xen-3.0.5-testing.hg tree to xen-3.1.0-testing later today. The release candidate will be renamed to 3.1.0-rc4. The final release will be called 3.1.0 (as opposed to 3.0.5-0 in the old numbering scheme). Further bug-fix releases in the 3.1 series will be called 3.1.x (as opposed to 3.0.5-x in the old numbering scheme).

This change rationalises and simplifies our version numbering scheme. It avoids the need for the fourth digit “-x”, and incrementing the second rather than third digit makes it clearer that large changes, not just bug fixes, are incorporated in our quarterly major releases.

The interesting presentations from the 4th Xen Summit are now available:

http://www.xensource.com/xen/xensummit.html

I did not get to go this time unfortunately!

This morning on xen-devel, Keir Fraser writes:

The third release candidate for Xen 3.0.5 is now available at http://xenbits.xensource.com/staging/xen-3.0.5-testing.hg. Quite a few issues are fixed: 32-bit PV-on-HVM on 64-bit hypervisor, virtual framebuffer setup, HVM ioemu and rombios issues, to name a few. I would like the next release candidate to be a *realistic* candidate for release — hence it is absolutely bug fixes only from here on to reduce churn, and we need as much testing as possible to make -rc4 a good one. We’re getting close!

Kate Keahey writes on workspace-announce:

We are happy to announce the release TP 1.2.3 of the Workspace Service. The new release adds features enabling partition management for VMs scheduled to be deployed: partitions can be mounted on deployment and blank partitions of a required size can be automatically generated. In addition, we added HTTP transfer adapter for staging, improved scheduling criteria, and added a User Quickstart guide.

For a detailed changelog, see the TP 1.2.3 documentation:
http://workspace.globus.org/vm/TP1.2.3/index.html

You can download the new release from:
http://workspace.globus.org/downloads/index.html

Accompanying this release is a lot of documentation work including a new User Quickstart, Features and Roadmaps pages, the Workspace Marketplace, and sitewide updates.

One of the new features, the HTTP transfer adapter, allows for the service to trigger an HTTP GET on a configured image repository node on the cluster. As with any of the staging mechanisms, from there the file can be propagated to one or many hypervisor pool nodes.

The adapter includes configurations that allow for (a) checksum verification and (b) the ability to auto-sense gzip files (which are common for VMs online) and decompress them after the transfer.

In short, this adapter allows a workspace’s staging request to seamlessly include VMs from online VM appliance marketplaces such as rBuilder (which is much more than a repository of VM images if you have not checked that out yet).

rPath Brings Amazon Web Services to Customers

It will work like this: software developers use rBuilder to build an Amazon Machine Image (AMI) that is stored using the Amazon Simple Storage Service (Amazon S3). Then, with a single click, rBuilder and rBuilder Online users can boot their software appliances on Amazon EC2.

I stumbled upon a Xen wiki page, HVM_Compatible_Notebooks, which is a good complement to HVM_Compatible_Processors.

These pages are going to be important to me next year when I look at getting a new laptop :-)

Are those all that are out there so far?

Quoting from this article on the Novell/Microsoft deal:

Users can host Windows Server as a paravirtualized guest on SUSE Linux Enterprise Server 10, using the Xen virtualization technology embedded in the Linux operating system.

Very interesting. The only paravirtualized Windows we’ve heard of so far is the proof of concept the Xen team did at Cambridge several years ago which was protected by NDA. I wonder what the exact licensing terms are — and what mechanism will Microsoft use to (try to) prevent people from running their domU kernel on other Xen installations?

At home, I wanted a reliable disk solution for backups and also wanted a big, blank and resizable storage system for virtual machines. I knew I wanted to be able to get at the shared disk remotely from other nodes and wanted to be able to replace broken hardware quickly if something failed. I also didn’t want to spend a lot of time reconfiguring OSs and software in the case of a total system failure.

I have two cheap computers and so I put some big disks in them and mirrored the disks over the network. Instead of using one file server node and RAID1, this is something like a “whole system RAID”. If anything at all breaks in either computer, hosted services can keep running and data is unharmed except for whatever was unsynced in RAM.

To accomplish the disk mirror I used DRBD. DRBD is a special block device that is designed for highly available clusters, it mirrors activity directly at the block device level across the network to another disk. So like a RAID1 configuration over the network. It lets you build something like the shared storage devices on a SAN, but without any special hardware. This provides the basic reliability layer.

Linux Logical Volume Management (LVM) is a popular tool that lets you flexibly manage disk space. Instead of just partitioning the disk, using LVM lets us do on-the-fly logical partition resizing, snapshots (including hosting snapshot+diffs), and adding more physical disks into the volume group as needs grow (you can even resize a logical partition across multiple underlying disks). Each logical partition is formatted with a filesystem of its own. Using LVM avoided some future headaches I think.

That is how the disk is setup, now how to access it remotely? You could run a shared filesystem of course, exporting via an NFS server on host A (or B). Instead, having heard good things about Global Network Block Device (GNBD) on the Xen mailing lists, I chose to export the logical block devices (from LVM) directly over the network with GNBD. Another node makes a GNBD import and the block device appears to be a local block device there, ready to mount into the file hierarchy. This is like iSCSI but it is a snap to set up and use.

And if that other node is a Xen domain 0, that block device is very handily ready to be used as a VM image, just as if it was a raw partition on that node.

Here’s an example Xen configuration using the imported block device:

disk = [ ‘phy:/dev/gnbd/vmimage001,sda1,w’]

The guest VM needs no awareness of all these tools, it just sees its sda1 and mounts it like anything else Xen presents to it as one of its “hardware” partitions.

Instead of just using the file store for backups and VMs that are used intermittently, I’m also running persistent services like websites, the incremental-backup server and a media server in VMs stored there.

First, this allows for basic backups of the LAN services without any backup software, that’s nice to have, although I really prefer a combination of incremental backups and RAID1. (Here we also avoid a Russell’s paradox situation with the backup server).

Second, keeping time-consuming-to-configure services in a VM allows me to replace hardware quickly, including whole computers in the event of a total failure: the only software I’ll ever need to reinstall is {Linux, DRBD, LVM, GNBD} for a file server node and {Linux, Xen} for a VMM node.

As long as net latency is really low (here it is sub-ms) it doesn’t really matter that the disk is remote for any of my uses. The VMs always respond very well.

(I should mention: you could of course take GNBD out of the picture and run the VMs on host A and B if Xen were installed there)

Another bonus: using GNBD, you can live-migrate the VM to any node that can do a GNBD import. This is nice to have. I only live migrate manually, though. Both DRBD and GNBD have some features that allow for seamless failover but I don’t really need any of this at home.

To learn more about that, check out this paper on the new DRBD (it is interesting): http://www.drbd.org/fileadmin/drbd/publications/drbd8_wpnr.pdf

Thinking about high availability in this kind of setup for a minute, a possible and simple to execute arrangement for services that need to be up at all times would be to take two DRBD mirrored nodes, run VMs on one or both of them, and have the physical nodes heartbeat each other. This is a simpler approach than a centralized file server with block device export, here we just have two peer VMMs that are “watching out” for each other.

You’d have two master/slave arrangements, so in the normal operating case: one VMM with partition A as DRBD master and partition B as secondary, then on the other VMM you have partition A as secondary and partition B as master. VMs run from a partition that is the DRBD master.

Let’s say you split four services into four VMs and put two VMs on each physical node. One of the physical node’s disks fail entirely and a monitor process notices. The heartbeat script makes sure the OK node is now the DRBD master for both partition A and B. Then it boots the two VMs previously hosted on the failed node on the OK node, re-allocating RAM for the time being to accomodate all four VMs.

The applications in those VMs recover just as if they went through a normal hard system reset (their network addresses can stay the same since both physical nodes are on the same LAN). Once the administrator gets the alert email and puts a new disk in, another script is ready to resync DRBD and then migrate two of the VMs back to their normal place.

This seems like something to consider for a highly hammered and important head node (like a Globus GRAM node for example). All it takes is another node, commodity hardware and open source software!