虛擬化技術

Virtualization Technique

System Virtualization
Memory Virtualization
 Agenda
• Basic concept of memory management
• Brief introduction to ARM v7 AMSA
• Memory Virtualization
 Shadow page table
 Hardware assistance
 Comparison
BASIC CONCEPT OF MEMORY MAN
AGEMENT
 Memory Management Unit
• A hardware component responsible for handling acces
ses to memory requested by the CPU
 Address translation: virtual address to physical address (VA t
o PA)
 Memory protection
 Cache control
 Bus arbitration
• Page tables are maintained by operating system, and M
MU only references them.
• TLB updates are performed automatically by page-tabl
e walking hardware
 Page Tables
• A page table is the data structure used by a virtual
memory system to store the mapping between virt
ual addresses and physical addresses
• Translation table base register(TTBR)
 Also called page table base register
 A register that stores the address of the base page table
for MMU
 TLB
• Translation look-aside buffer
 A CPU cache that memory management hardware uses to improve
virtual address translation speed
 The TLB is typically implemented as content-addressable memory
(CAM)
 The CAM search key is the virtual address and the search result is a
physical address
BRIEF INTRODUCTION TO ARMV7 V
MSA
 ARMv7 VMSA
• VMSA: Virtual memory system architecture
• Support four mapping types
 16MB Super section
 1MB Section
 64KB Large page
 4KB Small page

• Basically a 32-bit two-layer translation table

• Other features (virtualization extension and 64-bit large
physical address extension) would be mentioned in the f
uture class
 L1 Page Table
• First-level table
 Translation properties for a Section and Super-section
 Translation properties and pointers to a second-level table for a L
arge page or a Small page.

• L1 descriptor[1:0]
 0b00, invalid or fault entry
• Translation fault
 0b01, page table
• Address of 2nd level translation table
 0b10, section of super-section
 0b11, special usage with PXN attribute
1st Page Table Descriptor
 L2 Page Table
• Second-level tables
 The base address and translation properties for a Small
page or a Large page.
 That is, page tables
 1KB
• L2 descriptor[1:0]
 0b00, invalid or fault entry
• Translation fault
 0b01, large page
 0b1x, small page
2nd Page Table Descriptor
Concepts
Shadow page table
Hardware assistance
Comparison

MEMORY VIRTUALIZATION
 Memory Management on a VM
• Traditionally, OS fully controls all physical memor
y space and provides a continuous addressing spa
ce to each process
• Guest OS is just one of user space processes of hos
t OS
• If guest OS is allowed to access the physical memo
ry arbitrarily, then what happens?
• In system virtualization, VMM should make all virt
ual machines share the physical memory space
 Memory Virtualization
• Memory virtualization architecture

15
 Memory Virtualization
• The performance drop of memory access is usually unb
earable. VMM needs further optimization.

• VMM maintains shadow page tables :

 Direct virtual-to-physical address mapping
 Use hardware TLB for address translation

16
 Goals of Memory Virtualization
• Address Translation
 Control table-walking hardware that accesses translation tables in main memory.
• Memory Protection
 Define access permission which uses the Access Control Hardware.
• Access Attribute
 Define attribute and type of memory region to direct how memory operation to b
e handled.

• How to implement?
 Software solution: shadow page table
 Hardware solution
• NPT on SVM from AMD
• EPT on VMX from Intel
• ARM v7 VMSA (Virtual Memory System Architecture) with virtualization extension
Concepts
Shadow page table
Hardware assistance
Comparison

MEMORY VIRTUALIZATION
 Shadow Page Table
• Map guest virtual address to host physical address
 Shadow page table
• Guest OS will maintain its own virtual memory page table in the gu
est physical memory frames.
• For each guest physical memory frame, VMM should map it to hos
t physical memory frame.
• Shadow page table maintains the mapping from guest virtual addr
ess to host physical address.
 Page table protection
• VMM will apply write protection to all the physical frames of guest
page tables, which lead the guest page table write exception and tr
ap to VMM.
Shadow Page Table: Overview
 Shadow Page Table
• How does this technique work ?
 VMM should make MMU virtualized
• VMM manages the real PTBR and a virtual PTBR for each VM
• When a guest OS is activated, the real PTBR points to the corresponding shadow
page table of the guest OS
• When the guest OS attempts to modify the PTBR, it will be intercepted by VMM
for further emulation
 Shadow Page Table
• Construct shadow page table
 Guest OS will maintain its own page table for each process.
 VMM maps each guest physical page to host physical page.
 Create shadow page tables for each guest page table.
 VMM should protect host frame which contains guest page table. (that
means to write protect the guest page tables in host memory)

Write protection
 Shadow Page Table
• Implement with PTBR :
 For example, process 2 in guest OS wants to access its me
mory whose page number is 1.
 SPT Maintenance
• If guest OS modify one of its page tables, then the correspo
nding entry of SPT must be updated.
 We call it “shadow” because SPT is just like the shadow of page ta
bles of guest OS.

• How to identify this kind of modification?

 Guest OS could read/write a physical frame with the help of SPT.
 Mark those physical frames used as guest page tables read-only, s
o that when a guest OS tries to modify its guest page table, an exc
eption would be triggered.
 Then VMM checks the modification and updates the correspondi
ng entry on SPT
 Shadow Page Table
• Shadow page table operations :
Context switch New process

Access

Process 1 Process 2 Process 3

Page Table Page Table Page Table
Virtual PTPR Guest OS
Real PTPR VMM
Page fault !

Shadow 1 Shadow 1 Shadow 3

Page Table Page Table Page Table Load !
CorrespondingCreate Switch the pointer to new location
mapping
newtable 25
shadow page table mapping to new process
 Big Overhead
• A page fault caused by guest OS would launch the walki
ng process that costs a lot of overhead.
 Several steps to get a new entry on shadow page table
• Walk page tables on guest OS
• Check the permission on guest
• Offset shift: GPA to HVA
• Walk page tables on VMM
• Check the permission on VMM
• New entry established!
• Invalidate the TLB entry
 Each new process on guest OS would consume two pages. On
e is the page table on guest OS, and the other is the correspon
ding shadow page table.
 Page Walking Process on ARM

PABT / DABT
Trap Real MMU Behavior Shadow Table Behavior

1 2 3 Initial Synchronization
Guest Guest MMIO Shadow Page Shadow Page
Page Table Permission Access Table Table
Walker Checker Checker Mapping Update

True permission fault Hidden translation fault

True translation fault MMIO emulation

 Step 1
• While a page fault is occurred, Guest Page Table
Walker will walk through guest page table to ch
eck if the fault is from guest

PABT / DABT
Trap

1
Guest Guest MMIO Shadow Page Shadow Page
Page Table Permission Access Table Table
Walker Checker Checker Mapping Update

True permission fault Hidden translation

fault
True translation fault MMIO emulation
 Step 2
• Step 2 will check if guest access permission is
not allowed

PABT / DABT
Trap

2
Guest Guest MMIO Shadow Page Shadow Page
Page Table Permission Access Table Table
Walker Checker Checker Mapping Update

True permission fault Hidden translation

fault
True translation fault MMIO emulation
 Step 3
• Step 3 will check if the guest physical memory a
ddress used is located in the range of MMIO add
ress

PABT / DABT
Trap

3
Guest Guest MMIO Shadow Page Shadow Page
Page Table Permission Access Table Table
Walker Checker Checker Mapping Update

True permission fault Hidden translation

fault
True translation fault MMIO emulation
 Steps 4 & 5
• Step 4 and step 5 are used to build up shadow pa
ge tables and maintain their consistency betwee
n guest and shadow ones

PABT / DABT
Trap
4 5

Guest Guest MMIO Shadow Page Shadow Page

Page Table Permission Access Table Table
Walker Checker Checker Mapping Update

True permission fault Hidden translation

fault
True translation fault MMIO emulation
 Optimization For SPT
• Para-virtualization
 Reduce VM exits
 Guest OS would send a hyper call to VMM when guest OS sets the
page table entries.
 This method will eliminate from using write-protection for synch
ronization

• No trap for non-present PTEs

 Reduce VM exits
 If the page-fault is caused by PTE not present, it is not intercepte
d by the host
 Only works on VMX
 Optimization For SPT
• Un-synchronize shadow page table pages
 Reduce VM exits
 Allow the guest page table to be writable if and only if the page is the last level
page-structure (level 1)
 Based on TLB rules
• We need to flush TLB
to ensure the translation
use the modified page
structures, then we can
Intercept the TLB flush
operations and sync shadow
pages
• Sometimes, TLB need not
be flushed, then it can be
synced through page fault
 Optimization for SPT
• KSM: kernel shared memory

Kernel space Kernel space

User space User space

Guest Process 1 Guest Process 2

Shadow
Kernel space table
Shadow The shadow tables of kernel space
table are shared by all guest processes
User space
Shadow
table
 Other Issues
• Page fault and page protection issue
 When a physical page fault occurs, VMM needs to decide
whether this exception should be injected to guest OS or
not
• If the page entry in a guest page table is still valid, VMM prepares f
or the corresponding page and does not inject any exception to gu
est OS.
• If the page entry in a guest page table is invalid, then VMM directl
y injects the virtual page fault to guest OS.
Concepts
Shadow page table
Hardware assistance
Comparison

MEMORY VIRTUALIZATION
Hardware Assistance: Overview
 Hardware Solution
• Difficulties of shadow page table technique :
 Shadow page table implementation is extremely complex.
 Page fault mechanism and synchronization issues are critical.
 Host memory space overhead is considerable.

• But why we need this technique to virtualize MMU ?

 MMU do not first implemented for virtualization.
 MMU is knowing nothing about two level page address transl
ation.

• Now, let us consider hardware solution.

 Extended Page Table
• Concept of Extended Page Table (EPT) :
 Instead of walking along with only one page table hierarchy
, EPT technique implement one more page table hierarchy.
• One page table is maintained by guest OS, which is used to generate g
uest physical address.
• The other page table is maintained by VMM, which is used to map gue
st physical address to host physical address.

 For each memory access operation, EPT MMU directly gets

the guest physical address from guest page table, and then
gets the host physical address by the VMM mapping table a
utomatically.
 Extended Page Table
• Memory operation :
6 8
9 7
8 4 Data
Concepts
Shadow page table
Hardware assistance
Comparison

MEMORY VIRTUALIZATION
 Question
• Computer architecture with virtualization extensio
n is a trend.

• Hardware-assisted techniques replace many softw

are methods of virtualization.

• However, is hardware-assisted implementation a d

efinite winner?
 Comparisons
Hardware-assisted Software solution
• Walk any requested address • Only walk when SPT entry miss
 Appropriate to programs that have  Appropriate to programs that woul
a large amount of page table miss d access only some addresses freq
when executing uently
 Less chance to exit VM (less contex  Every access might be intercepted
t switch) by VMM (many traps)
• Two-layer EPT • One reference
 Means each access needs to walk t  Fast and convenient when page hit
wo tables • Hard to develop
• Easier to develop  Two-layer structure
 Many particular registers  Complicated reverse map
 Hardware helps guest OS to notify  Permission emulation
the VMM
 Combination?
• Is hardware-assisted implementation a definite winner?
 No
 How about combining these two methods?
• Selective memory virtualization
 The VMM can dynamically choose the memory management mechanis
m depending on the executing status.
 Challenge
• Hard to figure out the standard that we judge a program performs too many page tabl
e misses
• An accurate algorithm to sample behaviors of a program is necessary
 Group from Peking University completed a selective solution on VEE 20
11
• Gain just a little bit performance improvement
• For now, it’s not worth to do so.
 Selective Memory Virtualization
• Experiment data
 The bar shows the normalized exec
ution time
 The lower, the better.
 HAP: hardware-assisted page table
 SP: shadow page table
 DSP: dynamic selective page table
 Take hardware extension as 100%
• We can see that…
 Generally HAP is faster than SPT, bu
t not always.
 The performance of DSP is best , but
only tiny disparities. (about 2%)
 Memory Virtualization Summary
• Software implementation
 Memory architecture
• MMU (memory management unit)
• TLB (translation look-aside buffer)
 Shadow page table
• MMU virtualization by virtual PTBR
• Shadow page table construction
• Page fault and page table protection

• Hardware assistance
 Extended page table
• Hardware walk guest and host page table simultaneously
 Reference
• Selective hardware/software memory virtualizatio
n
 http://www.cs.mtu.edu/~zlwang/papers/vee11.pdf
• ARM® Architecture Reference Manual: ARMv7-A a
nd ARMv7-R edition