Virtualization and Grid Computing

New KVM release today: kvm-65.

Anthony Ligouri notes some interesting things @ KVM for the Mainframe.

Qumranet is hosting the first KVM Forum August 29-31.

See this previous entry for more about KVM. KVM is steadily making progress, see the KVM status page for the most up to date information.

(This is part of a series of entries)

Because Xen and KVM both support unmodified guests, I’d speculate that in the long run their raw CPU performance will converge on whatever concrete limitation that hardware-assisted virtualization presents. And paravirtualization may continue to reign here, or it may not. The harder issues to think about are disk and network I/O.

I was part of an investigation into how to make resource guarantees for workspaces under even the worst conditions on non-dedicated VMMs (Division of Labor: Tools for Growth and Scalability of Grids). The amount of CPU needed to support the guests’ I/O work (what I like to casually call the “on behalf of” work in the service domain) was pretty high and we looked at how to measure what guarantees were needed for the service domain itself to make sure the guest guarantees were met. So we had to write code that would extrapolate the CPU reservations needed across all domains (including the service domain).

One major source of the extra CPU work is context switching overhead, the service domain needs to switch in to process pending I/O events (on large SMPs, I’ve heard recommendations to just dedicate a CPU to the service domain). Also, in networking’s case, the packets are zero copy but they must still traverse the bridging stack in the service domain.

One important thing to consider for the long run on this issue is that there is a lot of work being done to make slices of HW such as Infiniband available directly to guest VMs, this will obviate the need for a driver domain to context switch in. See High Performance VMM-Bypass I/O in Virtual Machines

Container based, kernelspace solutions offer a way out of a lot of this overhead by being implemented directly in the kernel that is doing the “on behalf of” work. They also take advantage of the resource management code already in the Linux kernel.

They can more effectively schedule resources being used inside their regular userspace right alongside the VMs (I’m assuming) — and more easily know what kernel work should be “charged” to what process (I’m assuming). These two things could prove useful, avoiding some of the monitoring and juggling that is needed to correctly do that in a Xen environment (see e.g., the Division of Labor paper mentioned above and the Xen related work from HP).

There is an interesting paper Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors out of Princeton.

The authors contrast Xen and VServer and present cases where hard-partitioning (that you find in Xen) breeds too much overhead for grid and high performance use cases. Where full fault isolation and OS heterogeneity are not needed, they advocate that the CPU overhead issues of Xen I/O and VM context switches can be avoided.

(The idea presented there of live updating the kernel (as you migrate the VM) is interesting. For jobs that take months (that will miss out on kernel updates to their template images) or services that should not be interrupted, this presents an interesting alternative for important security updates (though for Linux, I’m under the impression that security problems are far more of a problem in userspace).)

The news of KVM’s inclusion in the Linux kernel has me looking at KVM.

How the virtualization layer is implemented has important ramifications, and KVM is much different from Xen.

Xen itself sits directly on the hardware and treats even your old OS as a guest VM — for Xen, the Linux patches are necessary to make Linux a guest VM on the hypervisor. It implements a model known as paravirtualization that provides a lot of performance improvements over other recent x86 virtualization techniques such as VMware’s. This model includes a service VM that does privileged I/O work on behalf of the guests. (In a following entry I discuss some performance and efficiency issues, the larger story is not so simple as “paravirtualization makes everything very fast”)

Xen can also support unmodified guest VMs if you are using either a VT capable Intel processor or an SVM capable AMD processor.

KVM is a patch to the Linux kernel that is more like VServer, Solaris containers, or microkernels [see footnote], where the OS still sits directly on the hardware. Some aspects of the current KVM release (mostly pulled from their FAQ):

• KVM guest OSs appear as a process and can be managed with top, kill, etc.
• KVM requires either a VT capable Intel processor or an SVM capable AMD processor (which is a currently a big limitation for grid providers but this will become less and less of an issue as older clusters are phased out).
• Uses QEMU for guest instantiation, but is not an emulator itself
• Runs Windows 32bit already (with no apci)
• I have yet to figure out if it relies on these kernel enhancements for OS level virtualization
• The current implementation should probably be seen as a preview, Fraser Campbell reports that it was “decidedly slow” when he tried it.

[footnote]: Some argue that microkernels and Xen have a lot in common.

[UPDATE: This article on some KVM details was posted. Its comments also have a lot of information.]

Techworld reports:

A relatively obscure virtualisation system has leapfrogged better-known rivals to make its way into an upcoming Linux kernel.

KVM (Kernel-based Virtual Machine) has been accepted by kernel gurus such as Linus Torvalds and Andrew Morton for inclusion in version 2.6.20 of the Linux kernel, developers said earlier this week. The system consists of a loadable kernel module and a user component, and is licensed under the GNU General Public License.

KVM links: KVM website, KVM whitepaper, Qumranet.

This news has me thinking about several things:

• Kernel inclusion ramifications for grid resource providers
• Differences between Xen, KVM, and similar systems
• Some performance and efficiency considerations
• What container abstraction layer is most appropriate for grid applications to lock into?

The company behind KVM is Qumranet which is founded by Moshe Bar who started OpenMosix and Qlusters, and he was the original CTO of XenSource. The Qumranet website does not mention Moshe’s involvment, I read this on http://linuxvirtualization.com (where Fraser Campbell also reports that it was “decidedly slow” when he tried it).

[UPDATE: an anonymous commenter points out the Qumranet web page is updated and Moshe is listed at http://www.qumranet.com/team.php]

I think that Qumranet’s securing of venture capital and (agreeing with Fraser’s sentiment) Moshe’s involvement means that we can look forward to a lot of interesting developments.