Huge Page Calculation
Huge Page Calculation
Huge Page Calculation
NOTE: I have recently discovered that Oracle, hugepages, and NUMA are incompatible,
at least on Linux. NUMA must be disabled to use hugepages with Oracle.
RAM is managed in 4k pages in 64 bit Linux. When memory sizes were limited, and
systems with more than 16G RAM were rare, this was not a problem.
However, as systems get more memory, the number of memory pages increased and
become less manageable.
Hugepages make managing the large amounts of memory available in modern servers
much less CPU intensive. In particular, with the number of memory pages reduced by
typically three orders of magnitude, the chance that a particular page pointer will
be available in the processor cache goes up dramatically.
First some caveats on using hugepages: Hugepages are not swappable, thus Oracle
SGA memory must either be all hugepages are no hugepages.
If you allocate hugepages for Oracle, and don’t allocate enough for the entire
SGA, Oracle will not use any hugepage memory. If there is not enough non-
hugepage memory, your database will not start. Finally, enabling hugepages will
require a server restart, so if you do not have the ability to restart your server,
do not attempt to enable hugepages.
There are both Oracle database settings and Linux OS settings that must be adjusted
in order to enable hugepages. The Linux and oracle settings of concern are below:
Linux OS settings:
/etc/sysctl.conf:
vm.nr_hugepages
kernel.shmmax
kernel.shmall
/etc/security/limits.conf:
SGA_TARGET
SGA_MAX_SIZE
MEMORY_TARGET
MEMORY_MAX_TARGET
USE_LARGE_PAGES
First, calculate the Linux OS settings. Kernel.shmmax should be set to the size of
the largest SGA_TARGET on the server plus 1G, to account for other processes.
For a single instance with 180G RAM, that would be 181G.
Kernel.shmall should be set to the sum of the SGA_TARGET values divided by the
pagesize. Use ‘getconf pagesize’ command to get the page size.
Units are bytes. The standard pagesize on Linux x86_64 is 4096, or 4k.
Oracle soft memlock and oracle hard memlock should be set to slightly less than the
total memory on the server,
I chose 230G. Units are kbytes, so the number is 230000000. This is the total
amount of memory Oracle is allowed to lock.
Now for the hugepage setting itself: vm.nr_hugepages is the total number of
hugepages to be allocated on the system.
The number of hugepages required can be determined by finding the maximum amount
of SGA memory expected to be used by the system (the SGA_MAX_SIZE value normally,
or the sum of them on a server with multiple instances) and dividing it by the
size of the hugepages, 2048k, or 2M on Linux.
To account for Oracle process overhead, add five more hugepages .
So, if we want to allow 180G of hugepages, we would use this equation:
(180*1024*1024/2048)+5. This gives us 92165 hugepages for 180G.
Note: I took a shortcut in this calculation, by using memory in MEG rather than
the full page size.
To calculate the number in the way I initial described, the equation would be:
(180*1024*1024*1024)/(2048*1024).
/etc/security/limits.conf
/etc/sysctl.conf
vm.nr_hugepages = 92165
kernel.shmall = 47448064
Next, set SGA_TARGET and SGA_MAX_SIZE to the desired size. I generally recommend
setting both to the same size.
Oracle recommends explicitly setting the MEMORY_TARGET and MEMORY_MAX_TARGET to 0
when enabling hugepages.
So these are the values in the spfile that we change:
USE_LARGE_PAGES=only
SGA_TARGET=180G
SGA_MAX_SIZE=180G
MEMORY_MAX_TARGET=0
MEMORY_TARGET=0
In order to verify that hugepages are being used, run this command:
Note that this example uses Linux hugepage size of 2M (2048k). On Itanium systems
the hugepage size is 256M.
These instructions should allow you successfully implement huge pages in Linux.
Note that everything would be the same for Oracle 10gR2, with the exception that
the USE_LARGE_PAGES parameter is unavailable.