Massive virtualisation on IBM Power5

After Aaron?s talk I hit the IBM Power5 virtualisation presentation; between this and their Xen talk they were all about the virtualisation stuff.

The talk was on massive virtualisation; they had an eight way Power5 box running eighty Miro2?equivalent virtual servers (actually, one of the virtual servers was considerably better, but more on that later). There is a limit of ten virtual servers per physical CPU for the Power virtualisation technology, but apparently this is just a #define in some source file somewhere that exists to prevent marketing from telling customers ??and you can run 100 virtual systems on just one CPU!?.

I mentioned earlier that one of the virtual machines was considerable more powerful than the others. This is an artifact of the very light weight hypervisor design: the powerful virtual server ?owns? the IO for all the other virtual servers. It?s this way so the hypervisor can be very simple (and hence reliable); in contrast, the Xen hypervisor is responsible for handling the IO of all the client OSes. If I understand things correctly, the Power architecture has some extra bits and bobs that let this work, as it wouldn?t play nicely on x86. Anyway, to stop all the extra IO impacting the performance of the ?more equal? virtual server, they gave it about a gig of memory.

I thought the performance graphs they had for the virtual servers were really quite nifty. For their benchmarking utility, they used Linux kernel compilations. There?s very little overhead added from simply having the virtual machines there: there was only a 3% difference in the compilation times between having only one virtual machine running and having all eighty going.

The hypervisor gives you some control over the amount of CPU time each virtual machine can use ? you can say how many processors it has access to and the most amount of CPU time it can use on those processors. These limits are sufficient to restrict each virtual machine to getting only its ?fair share? of time, which in principle should mean that on a system set up this way it should take as long for one virtual server to compile a kernel as for all eighty to do so. This isn?t the case in practice: there?s a linear increase in compilation time with the number of active virtual servers. This appears to be an IO issue, though, so loads that lean mainly on the CPU shouldn?t see this problem. It was mentioned, though, that multiple virtual servers can each assume some responsibility for the IO, spreading the load and improving the performance, but due to time constraints (they finished benchmarking the times about 15 minutes before the presentation started!) they were unable to show us how the system would improve in this case.