当前位置 博文首页 > LuciferLiu_DBA:Oracle中如何配置HugePages(标准大页)和Trans

    LuciferLiu_DBA:Oracle中如何配置HugePages(标准大页)和Trans

    作者:[db:作者] 时间:2021-06-16 12:13

    一、HugePages(标准大页)

    1、什么是标准大页?

    HugePages is a feature integrated into the Linux kernel with release 2.6 and later. This feature basically provides the alternative to the 4K page size (16K for IA64) providing bigger pages.

    HugePages in 2.4 Kernels

    The HugePages feature is backported to some 2.4 kernels. Kernel versions 2.4.21-* has this feature (See?Note 311504.1?for the distributions with 2.4.21 kernels) but it is implemented in a different way. The feature is completely available. The difference from 2.6 implementation is the organization within the source code and the kernel parameters that are used for configuring HugePages. See Parameters/Setup section below.

    2、如何查看Hugepagesize

    $ grep Hugepagesize /proc/meminfo
    HW PlatformSource Code TreeKernel 2.4

    Kernel 2.6

    and later

    Linux x86 (IA32)i3864 MB2 MB
    Linux x86-64 (AMD64, EM64T)x86_642 MB2 MB
    Linux Itanium (IA64)ia64256 MB256 MB
    IBM Power Based Linux (PPC64)ppc64/powerpcN/A **16 MB
    IBM zSeries Based Linuxs390N/A1 MB
    IBM S/390 Based Linuxs390N/AN/A

    3、HugePages and Oracle 11g Automatic Memory Management (AMM)

    The AMM and HugePages are not compatible. One needs to disable AMM on 11g to be able to use HugePages. See?Document 749851.1?for further information.

    4、如何设置HugePages

    Kernel Version 2.4

    The kernel parameter used for HugePages is vm.hugetlb_pool which is based on MB of memory. RHEL3, Asianux 1.0, SLES8 (Service Pack 3 and over) are examples of distributions with the 2.4 kernels with HugePages support. For the configuration, follow steps below:
    
    1. Start database instance(s)
    2. Calculate hugetlb_pool using script from Note 401749.1
    3. Shutdown database instances 
    4. Set kernel parameter:
             # sysctl -w vm.hugetlb_pool=<value from above>
        and make sure that the parameter is persistent to reboots. e.g. On Asianux 1.0 by editing /etc/sysctl.conf adding/modifying as below:
            vm.hugetlb_pool=<value from above>
    
    5. Check available hugepages:
            $ grep Huge /proc/meminfo
    
    6. Restart database instances
    7. Check available hugepages:
            $ grep Huge /proc/meminfo
    
    Notes:
    
    If the setting of hugetlb_pool is not effective, you will need to reboot the server to make HugePages allocation during system startup.
    The HugePages are allocated in a lazy fashion, so the "Hugepages_Free" count drops as the pages get touched and are backed by physical memory. The idea is that it's more efficient in the sense that you don't use memory you don't touch.
    If you had set the instance initialization parameter PRE_PAGE_SGA=TRUE (for suitable settings see Document 30793.1), all of the pages would be allocated from HugePages up front.

    Kernel Version 2.6 and later

    The kernel parameter used for HugePages is vm.nr_hugepages which is based on the number of the pages. SLES9, RHEL4 and Asianux 2.0 are  examples of distributions with the 2.6 kernel. For the configuration, follow steps below:
    
    1. Start database instance(s)
    2. Calculate nr_hugepages using script from Document 401749.1
    3. Shutdown database instances 
    4. Set kernel parameter:
            # sysctl -w vm.nr_hugepages=<value from above>
    and make sure that the parameter is persistent to reboots. e.g. On SLES9:
            # chkconfig boot.sysctl on
    
    5. Check available hugepages:
            $ grep Huge /proc/meminfo
    
    6. Restart database instances
    7. Check available hugepages:
            $ grep Huge /proc/meminfo
    
    Notes:
    
    If the setting of nr_hugepages is not effective, you will need to reboot the server to make HugePages allocation during system startup.
    The HugePages are allocated in a lazy fashion, so the "Hugepages_Free" count drops as the pages get touched and are backed by physical memory. The idea is that it's more efficient in the sense that you don't use memory you don't touch.
    If you had set the instance initialization parameter PRE_PAGE_SGA=TRUE (for suitable settings see Document  30793.1), all of the pages would be allocated from HugePages up front.
    Notes on HugePages in General
    The userspace application that employs HugePages should be aware of permission implications. Permissions HugePages segments in memory can strictly impose certain requirements. e.g. Per Bug 6620371 on Linux x86-64 port of Oracle RDBMS until 11g was setting the shared memory flags to hugetlb, read and write by default. But that shall depend on the configuration environment and with Patch 6620371 on 10.2 and with 11g, the read and write permissions are set based on the internal context.

    Calculate hugetlb_pool/nr_hugepages using script from Document 401749.1

    [oracle@p11g01 ~]$ cat /home/oracle/hugepages_settings.sh 
    #!/bin/bash
    #
    # hugepages_settings.sh
    #
    # Linux bash script to compute values for the
    # recommended HugePages/HugeTLB configuration
    # on Oracle Linux
    #
    # Note: This script does calculation for all shared memory
    # segments available when the script is run, no matter it
    # is an Oracle RDBMS shared memory segment or not.
    #
    # This script is provided by Doc ID 401749.1 from My Oracle Support
    # http://support.oracle.com
    
    # Welcome text
    echo "
    This script is provided by Doc ID 401749.1 from My Oracle Support
    (http://support.oracle.com) where it is intended to compute values for
    the recommended HugePages/HugeTLB configuration for the current shared
    memory segments on Oracle Linux. Before proceeding with the execution please note following:
     * For ASM instance, it needs to configure ASMM instead of AMM.
     * The 'pga_aggregate_target' is outside the SGA and
       you should accommodate this while calculating the overall size.
     * In case you changes the DB SGA size,
       as the new SGA will not fit in the previous HugePages configuration,
       it had better disable the whole HugePages,
       start the DB with new SGA size and run the script again.
    And make sure that:
     * Oracle Database instance(s) are up and running
     * Oracle Database 11g Automatic Memory Management (AMM) is not setup
       (See Doc ID 749851.1)
     * The shared memory segments can be listed by command:
         # ipcs -m
    
    
    Press Enter to proceed..."
    
    read
    
    # Check for the kernel version
    KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
    
    # Find out the HugePage size
    HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
    if [ -z "$HPG_SZ" ];then
        echo "The hugepages may not be supported in the system where the script is being executed."
        exit 1
    fi
    
    # Initialize the counter
    NUM_PG=0
    
    # Cumulative number of pages required to handle the running shared memory segments
    for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
    do
        MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
        if [ $MIN_PG -gt 0 ]; then
            NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
        fi
    done
    
    RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`
    
    # An SGA less than 100MB does not make sense
    # Bail out if that is the case
    if [ $RES_BYTES -lt 100000000 ]; then
        echo "***********"
        echo "** ERROR **"
        echo "***********"
        echo "Sorry! There are not enough total of shared memory segments allocated for
    HugePages configuration. HugePages can only be used for shared memory segments
    that you can list by command:
    
        # ipcs -m
    
    of a size that can match an Oracle Database SGA. Please make sure that:
     * Oracle Database instance is up and running
     * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
        exit 1
    fi
    
    # Finish with results
    case $KERN in
        '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
               echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
        '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '4.14') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '4.18') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        '5.4') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
        *) echo "Kernel version $KERN is not supported by this script (yet). Exiting." ;;
    esac
    
    # End
    

    二、Transparent HugePages(透明大页)

    1、什么是Transparent HugePages(默认开启)

    Starting with RedHat6, OL6, SLES11, and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management.? Transparent HugePages are similar to the HugePages that have been available in previous Linux releases.? The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time.

    2、开启Transparent HugePages会有什么影响

    Starting with SLES11, RHEL6, OL6, and UEK2 Kernels, Transparent HugePages are introduced and enabled by default, and this can cause node reboots and performance problems.
    Node reboots.

    The ocssd.log may show some of the threads are blocked (but this does not show all the time):
    
    2013-05-01 14:30:45.255: [    CSSD][224204544]clssscMonitorThreads clssnmvKillBlockThread not scheduled for 7500 msecs
    2013-05-01 14:30:46.945: [    CSSD][224204544]clssscMonitorThreads clssnmvWorkerThread not scheduled for 8030 msecs

    3、Oracle建议关闭Transparent HugePages

    Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.

    4、如何关闭Transparent HugePages(透明大页)

    For RedHat 6, Oracle Linux 6, SLES 11 and UEK2 kernels please have at "ALERT: Disable Transparent HugePages on SLES11, RHEL6, Oracle Linux 6 and UEK2 Kernels (Doc ID?1557478.1)"

    The following instruction are for Redhat 6 or lower, Oracle Linux 6 or lower, and SLES 11 or lower:

    To check if the?Transparent HugePages are enabled in your server execute the following:

    Default/Enabled setting is? [always]:

    # cat /sys/kernel/mm/transparent_hugepage/enabled

    [always] never

    ?

    Note 1:?For RHEL kernel, the file path is different from above:

    # cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

    [always] never
    Please modify this file accordingly.

    Note 2:?For UEK2 kernel, as of 2.6.39-400.116.0 Transparent Hugepages has been removed from the kernel.? If it is not compiled into the kernel then /sys/kernel/mm/transparent_hugepage will not exist.
    ?Disabled setting is [never]:

    # cat /sys/kernel/mm/transparent_hugepage/enabled

    always [never]

    If "enabled" is?NOT?set to "[never]", the Transparent HugePages are being used.

    You can also issue:

    # grep AnonHugePages /proc/meminfo

    If the output contains a line like "AnonHugepages: xxxx kB", with a value > 0kB, the kernel is using Transparent HugePages.

    Because the kernel currently uses?Transparent HugePages only for the anonymous memory blocks like stack and heap, the value of AnonHugepages in /proc/meminfo is the current amount of?Transparent HugePages that the kernel is using.
    To disable?Transparent HugePages boot time either one of the following 2 methods may be used:

    Add the following to the kernel boot line in /etc/grub.conf (a symlink to /boot/grub/grub.conf) and reboot the server (this is the preferred method):

    transparent_hugepage=never

    Once modified the line will read similar to the following example:

    title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
    ??????? root (hd0,0)
    ????????kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
    ??????? initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img

    OR

    Add the following lines in /etc/rc.local and reboot the server (this still can be done on Redhat 7 although rc.local is being deprecated):

    cat <<EOF >> /etc/rc.local
    if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    ?? echo never > /sys/kernel/mm/transparent_hugepage/enabled
    fi
    if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
    ?? echo never > /sys/kernel/mm/transparent_hugepage/defrag
    fi
    EOF

    ?Please change the file path for RHEL kernel to /sys/kernel/mm/redhat_transparent_hugepage/ accordingly.

    ===========================================================================

    The following comment was received about SLES11 SP3.??Please note that this is not tested, so this is provided only as a talking point to your sys admin who knows SLES11 SP3 and who should also contact SLES for correct information.

    in SLES11 SP3 with YAST/GRUB boot loader /etc/grub.conf overrided usind YAST. parameter transparent_hugepage=never must be set in YAST-bootloader- Edit settings - in line: optional kernel parameter [before showopts].

    Oracle Linux 7: How to Disable Transparent HugePages for RHCK Kernel? (Doc ID 2066217.1)?? ?

    SOLUTION

    "tuned.service" on OL7 set the?transparent_hugepage?to?always?by default, even if it is disabled in?grub?kernel command line.

    #?grep transparent_hugepage /boot/grub2/grub.cfg

    linux16 /vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/vgsystem-root ro ?rd.lvm.lv=vgsystem/swap rd.lvm.lv=vgsystem/root rhgb quiet numa=off?transparent_hugepage=never? ? ?? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ??

    Verify the THP (Transparent hugepage)?mode

    # uname -r

    3.10.0-229.el7.x86_64

    # cat /sys/kernel/mm/transparent_hugepage/enabled

    [always]?madvise never? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?←?THP is enabled?

    Disable it globally on throughput-performance

    1. Take a backup of "/usr/lib/tuned/throughput-performance/tuned.conf", then change?"transparent_hugepages=always"?to?"transparent_hugepages=never".?
      #cp?/usr/lib/tuned/throughput-performance/tuned.conf?/usr/lib/tuned/throughput-performance/tuned.conf.bkp_original
      #vi?/usr/lib/tuned/throughput-performance/tuned.conf

      ?[vm]
      ?transparent_hugepages=always
      ? ?to?
      ?[vm]
      ?transparent_hugepages=never?

    2. ?Reboot the server and verify the outcome.? ?
      #cat /sys/kernel/mm/transparent_hugepage/enabled

      ?always madvise?[never]? ? ? ? ? ? ? ? ? ? ? ? ? ←?THP is?disabled

    Disable it on active tuned profile

    1. ?First Identify which profile is active.? ? ?
      # tuned-adm active

      Current active profile: virtual-guest? ? ? ? ? ? ←?Virtual-guest is the active profile

    2. Now edit?"/usr/lib/tuned/virtual-guest/tuned.conf"?file and append?"transparent_hugepages=never"?in?vm?section to disable THP.??
      #cp?/usr/lib/tuned/virtual-guest/tuned.conf?/usr/lib/tuned/virtual-guest/tuned.conf.bkp_original
      #vi?/usr/lib/tuned/virtual-guest/tuned.conf

      [main]
      include=throughput-performance
      [vm]
      transparent_hugepages=never? ? ? ? ? ? ? ? ? ? ? ?

    3. Reboot the server and verify the outcome.

      #cat /sys/kernel/mm/transparent_hugepage/enabled
      always madvise [never]? ? ? ? ? ? ? ? ? ? ? ? ? ??← THP Disabled??

    Transparent Huge Pages cannot be enabled or disabled on a running machine and requires a reboot.

    ?THP status in other UEK kernel versions

    • UEK3 disabled by default .

      [root@ol6 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-3.8.13-118.6.2.el6uek.x86_64
      # CONFIG_TRANSPARENT_HUGEPAGE is not set??

    • UEK4 enabled by default.

      [root@ol7 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-4.1.12-37.2.2.el7uek.x86_64
      CONFIG_TRANSPARENT_HUGEPAGE=y
      CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y