Red Hat Enterprise Linux 5.1 utilizes nested paging on AMD Barcelona Processor to improve performance of virtualized guests
by Sanjay Rao
by Sanjay Rao
Red Hat® Enterprise Linux® (RHEL) 5.1 provides paravirtualized (PV) drivers for guest operating systems to improve I/O performance for I/O intensive disk or network applications.
In addition, RHEL 5.1 supports a feature called “nested paging,” available on AMD Barcelona processors, that translates guest virtual memory addresses to machine physical addresses using two-level translation in the hardware. Without nested paging, the hypervisor marks all page table pages as read only. So whenever the guest tries to update a page table, the call is trapped by the hypervisor that does the update. The shadow page tables on the guest are refreshed on a context switch, resulting in TLB misses on the guest. With nested paging, page tables are not refreshed with context switches, lowering the TLB miss count. The hypervisor also does not need to trap the updates to the page tables, since the hardware does the translation.
We ran a series of tests with an online transaction processing (OLTP) workload on Oracle 10G databases. We used this workload because it is a very common type of database implementation that uses both memory and I/O sub-systems. The test results demonstrate the performance gains achieved by using nested paging and PV drivers.
As the table below shows, adding nested paging almost doubles the performance for this workload, which uses the memory sub-system aggressively, as there are many page modifications. However it is still quite a bit off the performance numbers from the base system because I/O is still going through the emulation layer. When both nested paging and PV drivers are combined, we were able to get to 75% of bare metal and 3.5 times the performance on a fully virtualized guest on this I/O intensive workload.
Oracle 10GR2 was configured to run an OLTP workload. It was chosen because Oracle is a widely used database, and it uses large shared memory for its Shared Global Area (SGA). As kernel implementation with nested paging will provide the best performance benefit for applications that rely heavily on memory usage and modify memory segments extensively, we believe that Oracle applications running OLTP will benefit the most. The table below shows the comparison of the transactions per minute between the different Red Hat Enterprise Linux implementations.
|OS||Oracle transactions per minute||Relative gain||% Efficiency|
|RHEL 5.1 PV guest||13297.33||0.91|
|RHEL 4.4. FV guest||3163.13||1||0.22|
|RHEL 4.4 FV + NP guest||6131.47||1.94||0.42|
|RHEL 4.4 FV + PV + NP guest||11041.53||3.49||0.75|
|Processor||Dual Socket – AMD Barcelona – 2.0 GHz|
|Storage||HP EVA 4000 (28 136 G disks)|
|Host bus adapter||QLogic ISP2432 4Gb FC|
There are two types of virtualized operating systems that can be implemented: paravirtualized guest (PV) and fully-virtualized guest (FV). With the PV type, the guest runs a modified OS that has the capability to understand the hardware and can access it directly, which makes it closest in performance to bare metal. In FV guests, the hardware is accessed via a software layer that emulates the newer hardware as generic hardware. In effect, every call to the hardware, memory, or I/O needs to go through this layer, which results in a drop in performance. For memory calls, the software implements shadow page tables, which means the guest OS has its own virtual-to-physical memory translation page table. The hypervisor traps the updates to the table and carries out the changes in the physical machine using the translation table that it maintains for the guest physical address to machine physical address. So TLB misses are very expensive, as the pages need to be tracked and addresses loaded into both the tables.
By combining AMD’s nested paging and Red Hat Enterprise Linux 5.1′s PV drivers, performance numbers improve significantly, bringing it much closer to bare metal performance. This allows environments that run older versions of Linux or other operating systems (like Windows and Unix) to run in a fully virtualized mode on the Barcelona platform, thereby allowing customers to consolidate or, in some cases, just take advantage of the increased CPU speed.