G Very Large Memory and HugePages

G.1.1 Implementing VLM on 32-bit Linux

With 32-bit architectures, VLM is accessed through a VLM window of a specific size. The VLM window is a data structure in the process address space that provides access to the whole virtual address space from a window of a specific size. On 32-bit Linux, you must set the parameter USE_INDIRECT_DATA_BUFFERS=TRUE, and mount a shmfs or tmpfs or ramfs type of in-memory filesystem over /dev/shm to increase the usable address space.

G.1.2 Prerequisites for Implementing VLM

The following are some of the prerequisites for implementing VLM on a 32-bit operating system:

• The computer on which Oracle Database is installed must have more than 4GB of memory.

• The computer must be configured to use a kernel with PAE support upon startup.

• The USE_INDIRECT_DATA_BUFFERS=TRUE must be present in the initialization parameter file for the database instance that uses VLM support.

• Initialization parameters DB_BLOCK_BUFFERS and DB_BLOCK_SIZE must be set to values you have chosen for the Oracle Database.

G.1.3 Methods To Increase SGA Limits

In a typical 32-bit Linux kernel, one can create an SGA of up to 2.4GB size. Using a Linux Hugemem kernel enables the creation of an SGA of upto 3.2GB size. To go beyond 3.2GB on a 32-bit kernel, you must use the VLM feature.

The following are the methods to increase SGA limits on a 32-bit computer:

• Hugemem Kernel

• Hugemem Kernel with Very Large Memory

$ uname -r
2.6.9-5.0.3.ELhugemem

G.1.4 Configuring Very Large Memory for Oracle Database

Complete the following procedure to configure Very Large Memory on Red Hat Enterprise Linux 4/ Oracle Linux 4 using ramfs:

1. Log in as a root user:

   sudo -sh
   Password:
   

2. Edit the /etc/rc.local file and add the following entries to it to configure the computer to mount ramfs over the /dev/shm directory, whenever you start the computer:

   umount /dev/shm
   mount -t ramfs ramfs /dev/shm
   chown oracle:oinstall /dev/shm
   

   In the preceding commands, oracle is the owner of Oracle software files and oinstall is the group for Oracle owner account. If the new configuration disables /etc/rc.local file or you start an instance of Oracle database using a Linux service script present under the /etc/init.d file, then you can add those entries in the service script too.

   Note, this configuration will make ramfs ready even before your system autostarts crucial Oracle Database instances. The commands can also be included in your startup scripts. It is important that you test the commands extensively by repeated restart action, after you complete configuring the computer using the following steps:

3. Restart the server.

4. Log in as a root user.

5. Run the following command to check if the /dev/shm directory is mounted with the ramfs type:

   /dev/shm directory is mounted with the ramfs type:
   
   # mount | grep shm
   ramfs on /dev/shm type ramfs (rw)
   

6. Run the following command to check the permissions on the /dev/shm directory:

   # ls -ld /dev/shm
   drwxr-xr-x  3 oracle oinstall 0 Jan 13 12:12 /dev/shm
   

7. Edit the /etc/security/limits.conf file and add the following entries to it to increase the max locked memory limit:

   soft    memlock        3145728
   hard    memlock        3145728
   

8. Switch to the oracle user:

   # sudo - oracle
   Password:
   

9. Run the following command to check the max locked memory limit:

   $ ulimit -l
   3145728
   

10. Complete the following procedure to configure instance parameters for Very Large Memory:

    1. Replace the DB_CACHE_SIZE, DB_xK_CACHE_SIZE, sga_target, and memory_target parameters with DB_BLOCK_BUFFERS parameter.

    2. Add the USE_INDIRECT_DATA_BUFFERS=TRUE parameter.

    3. Configure SGA size according to the SGA requirements.

    4. Remove SGA_TARGET, MEMORY_TARGET, or MEMORY_MAX_TARGET parameters, if set.

11. Start the database instance.

12. Run the following commands to check the memory allocation:

    $ ls -l /dev/shm
    $ ipcs -m
    

G.1.5 Restrictions Involved in Implementing Very Large Memory

Following are the limitations of running a computer in the Very Large Memory mode:

• You cannot use Automatic Memory Management (AMM) while implementing VLM using ramfs, because AMM works on dynamic SGA tuning. With AMM swapping is possible. For example, you can unmap the unused SGA space and map it to PGA. Dynamic SGA and multiple block size are not supported with Very Large Memory because ramfs is not swappable. To enable Very Large Memory, you must ensure that you set the value of MEMORY_TARGET to zero.

• VLM can be implemented only if Database Buffer Cache size is greater than 512MB.

G.2.1 Tuning SGA With HugePages

Without HugePages, the operating system keeps each 4KB of memory as a page, and when it is allocated to the SGA, then the lifecycle of that page (dirty, free, mapped to a process, and so on) is kept up to date by the operating system kernel.

With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, since each page table entry is pointing to pages from 2MB to 256MB. Also, the kernel has fewer pages whose lifecyle must be monitored.

The following are the advantages of using HugePages:

• Increased performance through increased TLB hits.

• Pages are locked in memory and are never swapped out which guarantees that shared memory like SGA remains in RAM.

• Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (for example, SGA)

• Less bookkeeping work for the kernel for that part of virtual memory due to larger page sizes

G.2.2 Configuring HugePages on Linux

Complete the following steps to configure HugePages on the computer:

1. Edit the memlock setting in the /etc/security/limits.conf file. The memlock setting is specified in KB and set slightly lesser than the installed RAM. For example, if you have 64GB RAM installed, add the following entries to increase the max locked memory limit:

   *   soft   memlock    60397977
   *   hard   memlock    60397977
   

   You can also set the memlock value higher than your SGA requirements.

2. Login as the oracle user again and run the ulimit -l command to verify the new memlock setting:

   $ ulimit -l
   60397977
   

3. Run the following command to display the value of Hugepagesize variable:

   $ grep Hugepagesize /proc/meminfo
   

4. Complete the following procedure to create a script that computes recommended values for hugepages configuration for the current shared memory segments:

   Note:

   Following is an example that may require modifications.

   1. Create a text file named hugepages_settings.sh.

   2. Add the following content in the file:

      #!/bin/bash
      #
      # hugepages_settings.sh
      #
      # Linux bash script to compute values for the
      # recommended HugePages/HugeTLB configuration
      #
      # Note: This script does calculation for all shared memory
      # segments available when the script is run, no matter it
      # is an Oracle RDBMS shared memory segment or not.
      # Check for the kernel version
      KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
      # Find out the HugePage size
      HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
      # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
      NUM_PG=1
      # Cumulative number of pages required to handle the running shared memory segments
      for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
      do
         MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
         if [ $MIN_PG -gt 0 ]; then
            NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
         fi
      done
      # Finish with results
      case $KERN in
         '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
                echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
         '2.6'|'3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
          *) echo "Unrecognized kernel version $KERN. Exiting." ;;
      esac
      # End
      

   3. Run the following command to change the permission of the file:

      $ chmod +x hugepages_settings.sh
      

5. Run the hugepages_settings.sh script to compute the values for hugepages configuration:

   $ ./hugepages_settings.sh
   

6. Set the following kernel parameter:

   # sysctl -w vm.nr_hugepages=value_displayed_in_step_5
   

7. To make the value of the parameter available for every time you restart the computer, edit the /etc/sysctl.conf file and add the following entry:

   vm.nr_hugepages=value_displayed_in_step_5
   

8. Restart the server.

   Note:

   To check the available hugepages, run the following command:

   $ grep Huge /proc/meminfo
   

G.2.3 Restrictions for HugePages Configurations

Following are the limitations of using HugePages:

• Automatic Memory Management (AMM) and HugePages are not compatible. When you use AMM, the entire SGA memory is allocated by creating files under /dev/shm. When Oracle Database allocates SGA with AMM, HugePages are not reserved. To use HugePages on Oracle Database 12c, You must disable AMM.

• If you are using VLM in a 32-bit environment, then you cannot use HugePages for the Database Buffer cache. You can use HugePages for other parts of the SGA, such as shared_pool, large_pool, and so on. Memory allocation for VLM (buffer cache) is done using shared memory file systems (ramfs/tmpfs/shmfs). Memory file systems do not reserve or use HugePages.

• HugePages are not subject to allocation or release after system startup, unless a system administrator changes the HugePages configuration, either by modifying the number of pages available, or by modifying the pool size. If the space required is not reserved in memory during system startup, then HugePages allocation fails.

G.2.4 Disabling Transparent HugePages

Transparent HugePages memory is enabled by default with Red Hat Enterprise Linux 6, SUSE 11, and Oracle Linux 6 with earlier releases of Oracle Linux Unbreakable Enterprise Kernel 2 (UEK2) kernels. Transparent HugePages memory is disabled by default in later releases of UEK2 kernels.

Transparent HugePages can cause memory allocation delays at runtime. To avoid performance issues, Oracle recommends that you disable Transparent HugePages on all Oracle Database servers. Oracle recommends that you instead use standard HugePages for enhanced performance.

Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.

To check if Transparent HugePages is enabled run one of the following commands as the root user:

Red Hat Enterprise Linux kernels:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

Other kernels:

# cat /sys/kernel/mm/transparent_hugepage/enabled

The following is a sample output that shows Transparent HugePages is being used as the [always] flag is enabled.

[always] never

To disable Transparent HugePages perform the following steps:

1. Add the following entry to the kernel boot line in the /etc/grub.conf file:

   transparent_hugepage=never
   

   For example:

   title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
           root (hd0,0)
           kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
           initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img