2 Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks

2.4.1 Determining If the Oracle Inventory and Oracle Inventory Group Exists

When you install Oracle software on the system for the first time, OUI creates the oraInst.loc file. This file identifies the name of the Oracle Inventory group (by default, oinstall), and the path of the Oracle Central Inventory directory. An oraInst.loc file has contents similar to the following:

inventory_loc=central_inventory_location
inst_group=group

In the preceding example, central_inventory_location is the location of the Oracle central inventory, and group is the name of the group that has permissions to write to the central inventory (the OINSTALL group privilege).

If you have an existing Oracle central inventory, then ensure that you use the same Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users you intend to use for installation have permissions to write to this directory.

To determine if you have an Oracle central inventory directory (oraInventory) on your system:

Enter the following command:

# more /etc/oraInst.loc

If the oraInst.loc file exists, then the output from this command is similar to the following:

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

In the previous output example:

• The inventory_loc group shows the location of the Oracle Inventory

• The inst_group parameter shows the name of the Oracle Inventory group (in this example, oinstall).

Use the command grep groupname /etc/group to confirm that the group specified as the Oracle Inventory group still exists on the system. For example:

$ grep oinstall /etc/group
oinstall:x:1000:grid,oracle

2.4.2 Creating the Oracle Inventory Group If an Oracle Inventory Does Not Exist

If the oraInst.loc file does not exist, then create the Oracle Inventory group by entering a command similar to the following:

# mkgroup id=1000 adms='root' oinstall

The preceding command creates the oraInventory group oinstall, with the group ID number 1000. Members of the oraInventory group are granted privileges to write to the Oracle central inventory (oraInventory).

By default, if an oraInventory group does not exist, then the installer lists the primary group of the installation owner for the Oracle Grid Infrastructure for a Cluster software as the oraInventory group. Ensure that this group is available as a primary group for all planned Oracle software installation owners.

2.4.3 Creating the Oracle Grid Infrastructure User

You must create a software owner for Oracle Grid Infrastructure in the following circumstances:

• If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system

• If an Oracle software owner user exists, but you want to use a different operating system user, with different group membership, to separate grid infrastructure administrative privileges from Oracle Database administrative privileges.

  In Oracle documentation, a user created to own only Oracle Grid Infrastructure software installations is called the grid user. A user created to own either all Oracle installations, or only Oracle database installations, is called the oracle user.

# id oracle

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

2.4.4 Creating the Oracle Base Directory Path

The Oracle base directory for the grid installation owner is the location where diagnostic and administrative logs, and other logs associated with Oracle ASM and Oracle Clusterware are stored.

If you have created a path for the Oracle Clusterware home that is compliant with Oracle Optimal Flexible Architecture (OFA) guidelines for Oracle software paths then you do not need to create an Oracle base directory. When OUI finds an OFA-compliant path, it creates the Oracle base directory in that path.

For OUI to recognize the path as an Oracle software path, it must be in the form u[00-99]/app, and it must be writable by any member of the oraInventory (oinstall) group. The OFA path for the Oracle base is /u01/app/user, where user is the name of the software installation owner.

Oracle recommends that you create an Oracle Grid Infrastructure Grid home and Oracle base homes manually, particularly if you have separate Oracle Grid Infrastructure for a cluster and Oracle Database software owners, so that you can separate log files.

For example:

# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# mkdir -p /u01/app/oracle
# chown grid:oinstall /u01/app/11.2.0/grid
# chown grid:oinstall /u01/app/grid
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/
# chown -R grid:oinstall /u01

2.4.5 Creating Job Role Separation Operating System Privileges Groups and Users

A job role separation privileges configuration of Oracle ASM is a configuration with groups and users that divide administrative access privileges to the Oracle ASM installation from other administrative privileges users and groups associated with other Oracle installations. Administrative privileges access is granted by membership in separate operating system groups, and installation privileges are granted by using different installation owners for each Oracle installation.

If you prefer, you can allocate operating system user privileges so that you can use one administrative user and one group for operating system authentication for all system privileges on the storage and database tiers.

For example, you can designate the oracle user to be the installation owner for all Oracle software, and designate oinstall to be the group whose members are granted all system privileges for Oracle Clusterware, Oracle Automatic Storage Management, and all Oracle Databases on the servers, and all privileges as installation owners. This group must also be the Oracle Inventory group.

Oracle recommends that you use at least two groups: A system privileges group whose members are granted administrative system privileges, and an installation owner group (the oraInventory group) to provide separate installation privileges the OINSTALL privilege. To simplify using the defaults for Oracle tools such as Cluster Verification Utility, if you do choose to use a single operating system group to grant all system privileges and the right to write to the oraInventory, then that group name should be oinstall.

• Overview of Creating Job Role Separation Groups and Users

• Creating Database Groups and Users with Job Role Separation

# mkgroup -'A' id='1031' adms='root' dba

# mkgroup -'A' id='1032' adms='root' oper1

# mkgroup -'A' id='1020' adms='root' asmadmin

# mkgroup -'A' id='1022' adms='root' asmoper

# mkgroup -'A' id='1021' adms='root' asmdba

# id oracle

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

2.4.6 Example of Creating Standard Groups, Users, and Paths

The following is an example of how to use command lines to create the Oracle Inventory group (oinstall), and a single group (dba) as the OSDBA, OSASM and OSDBA for ASM groups. In addition, it shows how to create the Oracle Grid Infrastructure software owner (grid), and one Oracle Database owner (oracle) with correct group memberships. This example also shows how to configure an Oracle base path compliant with OFA structure with correct permissions:

# mkgroup -'A' id='1000' adms='root' oinstall 
# mkgroup -'A' id='1031' adms='root' dba
# mkuser id='1100' pgrp='oinstall' groups='dba' home='/home/grid' grid
# mkuser id='1101' pgrp='oinstall' groups='dba' home='/home/oracle' oracle
# mkdir -p  /u01/app/grid
# chown -R grid:oinstall /u01
# mkdir /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/

After running these commands, you have the following groups and users:

• An Oracle central inventory group, or oraInventory group (oinstall). Members who have the central inventory group as their primary group, are granted the OINSTALL permission to write to the oraInventory directory.

• A single system privileges group that is used as the OSASM, OSDBA, OSDBA for ASM, and OSOPER for ASM group (dba), whose members are granted the SYSASM and SYSDBA privilege to administer Oracle Clusterware, Oracle ASM, and Oracle Database, and are granted SYSASM and OSOPER for ASM access to the Oracle ASM storage.

• An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM group as the secondary group, with its Oracle base directory /u01/app/grid.

• An Oracle Database owner (oracle) with the oraInventory group as its primary group, and the OSDBA group as its secondary group, with its Oracle base directory /u01/app/oracle.

• /u01/app owned by grid:oinstall with 775 permissions before installation, and by root after the root.sh script is run during installation. This ownership and permissions enables OUI to create the Oracle Inventory directory, in the path /u01/app/oraInventory.

• /u01 owned by grid:oinstall before installation, and by root after the root.sh script is run during installation.

• /u01/app/11.2.0/grid owned by grid:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.

• /u01/app/grid owned by grid:oinstall with 775 permissions before installation, and 755 permissions after installation.

• /u01/app/oracle owned by oracle:oinstall with 775 permissions.

2.4.7 Example of Creating Role-allocated Groups, Users, and Paths

The following is an example of how to create role-allocated groups and users that is compliant with an Optimal Flexible Architecture (OFA) deployment:

# mkgroup -'A' id='1000' adms='root' oinstall
# mkgroup -'A' id='1020' adms='root' asmadmin
# mkgroup -'A' id='1021' adms='root' asmdba
# mkgroup -'A' id='1031' adms='root' dba1
# mkgroup -'A' id='1041' adms='root' dba2
# mkgroup -'A' id='1022' adms='root' asmoper
# mkuser id='1100' pgrp='oinstall' groups='asmadmin' home='/home/grid' grid
# mkuser id='1101' pgrp='oinstall' groups='dba1,asmdba' home='/home/oracle' oracle1
# mkuser id='1102' pgrp='oinstall' groups='dba2,asmdba' home='/home/oracle' oracle2
# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# chown -R grid:oinstall /u01/app/grid
# mkdir -p /u01/app/oracle1
# chown oracle1:oinstall /u01/app/oracle1
# mkdir -p /u01/app/oracle2
# chown oracle2:oinstall /u01/app/oracle2
# chmod -R 775 /u01

After running these commands, you have the following groups and users:

• An Oracle central inventory group, or oraInventory group (oinstall), whose members that have this group as their primary group are granted permissions to write to the oraInventory directory.

• A separate OSASM group (asmadmin), whose members are granted the SYSASM privilege to administer Oracle Clusterware and Oracle ASM.

• A separate OSDBA for ASM group (asmdba), whose members include grid, oracle1 and oracle2, and who are granted access to Oracle ASM.

• A separate OSOPER for ASM group (asmoper), whose members are granted limited Oracle ASM administrator privileges, including the permissions to start and stop the Oracle ASM instance.

• An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM (asmadmin), OSDBA for ASM (asmdba) group as a secondary group.

• Two separate OSDBA groups for two different databases (dba1 and dba2) to establish separate SYSDBA privileges for each database.

• Two Oracle Database software owners (oracle1 and oracle2), to divide ownership of the Oracle database binaries, with the OraInventory group as their primary group, and the OSDBA group for their database (dba1 or dba2) and the OSDBA for ASM group (asmdba) as their secondary groups.

• An OFA-compliant mount point /u01 owned by grid:oinstall before installation.

• An Oracle base for the grid installation owner /u01/app/grid owned by grid:oinstall with 775 permissions, and changed during the installation process to 755 permissions.

• An Oracle base /u01/app/oracle1 owned by oracle1:oinstall with 775 permissions.

• An Oracle base /u01/app/oracle 2 owned by oracle2:oinstall with 775 permissions.

• A Grid home /u01/app/11.2.0/grid owned by grid:oinstall with 775 (drwxdrwxr-x) permissions. These permissions are required for installation, and are changed during the installation process to root:oinstall with 755 permissions (drwxr-xr-x).

• /u01/app/oraInventory. This path remains owned by grid:oinstall, to enable other Oracle software owners to write to the central inventory.

2.6.1 Network Hardware Requirements

The following is a list of requirements for network configuration:

• Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).

  With Redundant Interconnect Usage, you should identify multiple interfaces to use for the cluster private network, without the need of using bonding or other technologies. This functionality is available starting with Oracle Database 11g Release 2 (11.2.0.2).

  When you define multiple interfaces, Oracle Clusterware creates from one to four highly available IP (HAIP) addresses. Oracle RAC and Oracle ASM instances use these interface addresses to ensure highly available, load-balanced interface communication between nodes. The installer enables Redundant Interconnect Usage to provide a high availability private network.

  By default, Oracle Grid Infrastructure software uses all of the HAIP addresses for private network communication, providing load-balancing across the set of interfaces you identify for the private network. If a private interconnect interface fails or become non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces.

  Note:

  If you define more than four interfaces as private network interfaces, be aware that Oracle Clusterware activates only four of the interfaces at a time. However, if one of the four active interfaces fails, then Oracle Clusterware transitions the HAIP addresses configured to the failed interface to one of the reserve interfaces in the defined set of private interfaces.

  When you upgrade a node to Oracle Grid Infrastructure 11g release 2 (11.2.0.2) and later, the upgraded system uses your existing network classifications.

  To configure multiple public interfaces, use a third-party technology for your platform to aggregate the multiple public interfaces before you start installation, and then select the single interface name for the combined interfaces as the public interface. Oracle recommends that you do not identify multiple public interface names during Oracle Grid Infrastructure installation. Note that if you configure two network interfaces as public network interfaces in the cluster without using an aggregation technology, the failure of one public interface on a node does not result in automatic VIP failover to the other public interface.

  Oracle recommends that you use the Redundant Interconnect Usage feature to make use of multiple interfaces for the private network. However, you can also use third-party technologies to provide redundancy for the private network.

  Note:

  NIC bonding is not required to use multiple NICs. During installation, you are given the opportunity to specify the planned use for all interfaces detected by the Installer. Oracle Clusterware uses all NICs that you identify as Private for the private interconnect.

  Redundant Interconnect Usage requires a complete release 11.2.0.2 or higher stack (Oracle Grid Infrastructure and Oracle Databases). Prior release Oracle Databases cannot use this feature, and must use third-party NIC bonding technologies. If you consolidate different database releases in one cluster, and use databases prior to Oracle Database release 11.2.0.2, then you may require both technologies.

• If you install Oracle Clusterware using OUI, then the public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes. This restriction does not apply if you use cloning, either to create a new cluster, or to add nodes to an existing cluster.

  For example: With a two-node cluster, you cannot configure network adapters on node1 with eth0 as the public interface, but on node2 have eth1 as the public interface. Public interface names must be the same, so you must configure eth0 as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2.

  See Also:

  Oracle Clusterware Administration and Deployment Guide for information about how to add nodes using cloning

• For the public network, each network adapter must support TCP/IP.

• For the private network, the interface must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (minimum requirement 1 Gigabit Ethernet).

  Note:

  UDP is the default interface protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. You must use a switch for the interconnect. Oracle recommends that you use a dedicated switch.

  Oracle does not support token-rings or crossover cables for the interconnect.

• Each node's private interface for interconnects must be on the same subnet, and those subnets must connect to every node of the cluster. For example, if the private interfaces have a subnet mask of 255.255.255.0, then your private network is in the range 192.168.0.0--192.168.0.255, and your private addresses must be in the range of 192.168.0.[0-255]. If the private interfaces have a subnet mask of 255.255.0.0, then your private addresses can be in the range of 192.168.[0-255].[0-255].

  For clusters using Redundant Interconnect Usage, each private interface should be on a different subnet. However, each cluster member node must have an interface on each private interconnect subnet, and these subnets must connect to every node of the cluster. For example, you can have private networks on subnets 192.168.0 and 10.0.0, but each cluster member node must have an interface connected to the 192.168.0 and 10.0.0 subnets.

• For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test if an interconnect interface is reachable using ping.

2.6.2 IP Address Requirements

Before starting the installation, you must have at least two interfaces configured on each node: One for the private IP address and one for the public IP address.

You can manage IP addresses and name resolution in the cluster in one of the following ways:

• Dynamic IP address assignment using Oracle Grid Naming Service (GNS). If you select this option, then network administrators assign static IP address for the physical host name and dynamically allocated IPs for the Oracle Clusterware managed VIP addresses. In this case, IP addresses for the VIPs are assigned by a DHCP and resolved using a multicast domain name server configured as part of Oracle Clusterware within the cluster. If you plan to use GNS, then you must have the following:

  • A DHCP service running on the public network for the cluster

  • Enough addresses on the DHCP to provide 1 IP address for each node's virtual IP, and 3 IP addresses for the cluster used by the Single Client Access Name (SCAN) for the cluster

• Static IP address assignment. If you select this option, then network administrators assign a fixed IP address for each physical host name in the cluster and for IPs for the Oracle Clusterware managed VIPs. In addition, domain name server (DNS) based static name resolution is used for each node. Selecting this option requires that you request network administration updates when you modify the cluster.

root@node1]$ nslookup mycluster-scan
Server:         dns.example.com
Address:        192.0.2.001
 
Name:   mycluster-scan.example.com
Address: 192.0.2.201
Name:   mycluster-scan.example.com
Address: 192.0.2.202
Name:   mycluster-scan.example.com
Address: 192.0.2.203

2.6.3 Broadcast Requirements for Networks Used by Oracle Grid Infrastructure

Broadcast communications (ARP and UDP) must work properly across all the public and private interfaces configured for use by Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases.

The broadcast must work across any configured VLANs as used by the public or private interfaces.

2.6.4 Multicast Requirements for Networks Used by Oracle Grid Infrastructure

With Oracle Grid Infrastructure release 2 (11.2), on each cluster member node, the Oracle mDNS daemon uses multicasting on all interfaces to communicate with other nodes in the cluster.

With Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases, multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster:

• Across the broadcast domain as defined for the private interconnect

• On the IP address subnet ranges 224.0.0.0/24 and 230.0.1.0/24

You do not need to enable multicast communications across routers.

2.6.5 DNS Configuration for Domain Delegation to Grid Naming Service

If you plan to use GNS, then before Oracle Grid Infrastructure installation, you must configure your domain name server (DNS) to send to GNS name resolution requests for the subdomain GNS serves, which are the cluster member nodes. The following is an overview of what needs to be done for domain delegation. Your actual procedure may be different from this example.

Configure the DNS to send GNS name resolution requests using delegation:

1. In the DNS, create an entry for the GNS virtual IP address, where the address uses the form gns-server.CLUSTERNAME.DOMAINNAME. For example, where the cluster name is mycluster, and the domain name is example.com, and the IP address is 192.0.2.1, create an entry similar to the following:

   mycluster-gns.example.com  A  192.0.2.1
   

   The address you provide must be routable.

2. Set up forwarding of the GNS subdomain to the GNS virtual IP address, so that GNS resolves addresses to the GNS subdomain. To do this, create a BIND configuration entry similar to the following for the delegated domain, where cluster01.example.com is the subdomain you want to delegate:

    cluster01.example.com  NS  mycluster-gns.example.com
   

3. When using GNS, you must configure resolve.conf on the nodes in the cluster (or the file on your system that provides resolution information) to contain name server entries that are resolvable to corporate DNS servers. The total timeout period configured—a combination of options attempts (retries) and options timeout (exponential backoff)—should be less than 30 seconds. For example, where xxx.xxx.xxx.42 and xxx.xxx.xxx.15 are valid name server addresses in your network, provide an entry similar to the following in /etc/resolv.conf:

   options attempts: 2
   options timeout: 1
   
   search cluster01.example.com example.com
   nameserver xxx.xxx.xxx.42
   nameserver xxx.xxx.xxx.15
   

   /etc/nsswitch.conf controls name service lookup order. In some system configurations, the Network Information System (NIS) can cause problems with Oracle SCAN address resolution. Oracle recommends that you place the nis entry at the end of the search list. For example:

   /etc/nsswitch.conf
        hosts:    files   dns   nis
   

   Note:

   Be aware that use of NIS is a frequent source of problems when doing cable pull tests, as host name and username resolution can fail.

2.6.6 Grid Naming Service Configuration Example

If you use GNS, then you need to specify a static IP address for the GNS VIP address, and delegate a subdomain to be delegated to that static GNS IP address.

As nodes are added to the cluster, your organization's DHCP server can provide addresses for these nodes dynamically. These addresses are then registered automatically in GNS, and GNS provides resolution within the subdomain to cluster node addresses registered with GNS.

Because allocation and configuration of addresses is performed automatically with GNS, no further configuration is required. Oracle Clusterware provides dynamic network configuration as nodes are added to or removed from the cluster. The following example is provided only for information.

With a two node cluster where you have defined the GNS VIP, after installation you might have a configuration similar to the following for a two-node cluster, where the cluster name is mycluster, the GNS parent domain is example.com, the subdomain is grid.example.com, 192.0.2 in the IP addresses represent the cluster public IP address network, and 192.168.0 represents the private IP address subnet:

^Footnote 1 Node host names may resolve to multiple addresses, including VIP addresses currently running on that host.

2.6.7 Manual IP Address Configuration Example

If you choose not to use GNS, then before installation you must configure public, virtual, and private IP addresses. Also, check that the default gateway can be accessed by a ping command. To find the default gateway, use the route command, as described in your operating system's help utility.

For example, with a two node cluster where each node has one public and one private interface, and you have defined a SCAN domain address to resolve on your DNS to one of three IP addresses, you might have the configuration shown in the following table for your network interfaces:

^Footnote 1 Node hostnames may resolve to multiple addresses.

You do not need to provide a private name for the interconnect. If you want name resolution for the interconnect, then you can configure private IP names in the hosts file or the DNS. However, Oracle Clusterware assigns interconnect addresses on the interface defined during installation as the private interface (en1, for example), and to the subnet used for the private subnet.

The addresses to which the SCAN resolves are assigned by Oracle Clusterware, so they are not fixed to a particular node. To enable VIP failover, the configuration shown in the preceding table defines the SCAN addresses and the public and VIP addresses of both nodes on the same subnet, 192.0.2.

2.6.8 Network Interface Configuration Options

The precise configuration you choose for your network depends on the size and use of the cluster you want to configure, and the level of availability you require.

If certified Network-attached Storage (NAS) is used for Oracle RAC and this storage is connected through Ethernet-based networks, then you must have a third network interface for NAS I/O. Failing to provide three separate interfaces in this case can cause performance and stability problems under load.

2.11.1 Checking Asynchronous Input Output Processes

On AIX 5, run the rootpre.sh script to enable the Asynchronous Input Output (AIO) device drivers. On AIX 6, the AIO device drivers are enabled by default. For both AIX 5 and AIX 6, increase the number of aioserver processes from the default value. The recommended value for aio_maxreqs is 64k (65536). Confirm this value for both AIX 5 and AIX 6.

Confirm the aio_maxreqs value using the procedure for your release:

AIX 6.1 and 7.1:

# ioo -o aio_maxreqs
aio_maxreqs = 65536

AIX 5.3:

# lsattr -El aio0 -a maxreqs
maxreqs 65536 Maximum number of REQUESTS True

When performing an asynchronous I/O to a file system, note that each asynchronous I/O operation is tied to an asynchronous I/O server. Thus, the number of asynchronous I/O servers limits the number of concurrent asynchronous I/O operations in the system.

The initial number of servers that are started during a system restart is determined by the minservers parameter. As concurrent asynchronous I/O operations occur, additional asynchronous I/O servers are started, up to a maximum of the value set in the maxservers parameter.

On AIX 5.3, if you are using Oracle Database with data files on a file system, then increase the default values for minservers and maxservers, as the default values for these parameters are too small. Increase the minservers and maxservers values based on I/O kprocs for each processor.

In general, to set the number of asynchronous I/O servers, complete the following procedure:

1. Adjust the initial value of maxservers to 10 times the number of disks divided by the number of CPUs that are to be used concurrently but no more than 80.

2. Monitor the performance effects on the system during periods of high I/O activity. If all AIO server processes are started, then increase the maxservers value. Also, continue to monitor the system performance during peak I/O activity to determine if there was a benefit from the additional AIO servers. Too many asynchronous I/O servers increase memory and processor overload of additional processes, but this disadvantage is small. See your operating system vendor documentation for information about tuning AIO parameters.

To monitor the number of AIO server processes that have started, enter the following command:

# ps -ek|grep -v grep|grep –v posix_aioserver|grep -c aioserver

2.11.2 Tuning Virtual Memory Manager (VMM)

Oracle recommends that you use the vmo command to tune virtual memory using the following values:

For example:

vmo -p -o minperm%=3
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
vmo -p -o lru_file_repage=0
vmo -p -o strict_maxclient=1
vmo -p -o strict_maxperm=0

You must restart the system for these changes to take effect.

2.11.3 Increasing System Block Size Allocation

Oracle recommends that you increase the space allocated for ARG/ENV list to 128. The size is specified by number of 4K blocks.

For example:

/usr/sbin/chdev -l sys0 -a ncargs='128'

2.11.4 Configuring SSH LoginGraceTime Parameter for AIX

On AIX systems, the OpenSSH parameter LoginGraceTime by default is commented out, and the default behavior of OpenSSH on AIX can sometimes result in timeout errors. To avoid these errors, complete the following procedure:

1. Log in as root.

2. Using a text editor, open the OpenSSH configuration file /etc/ssh/sshd_config.

3. Locate the comment line #LoginGraceTime 2m.

4. Uncomment the line, and change the value to 0 (unlimited). For example:

   LoginGraceTime 0
   

5. Save /etc/ssh/sshd_config.

6. Restart SSH.

2.11.5 Configuring Shell Limits

Set shell limits for the Oracle Grid Infrastructure installation owner and for root. Verify that unlimited is set for both accounts either by using the smit utility or by editing the /etc/security/limits file. The root user requires these settings because the crs daemon (crsd) runs as root.

To increase the shell limits:

1. Add the following lines to the /etc/security/limits file:

   default:
           fsize = -1
           core = 2097151
           cpu = -1
           data = -1
           rss = -1
           stack = -1
           nofiles = -1
   

2. Enter the following command to list the current setting for the maximum number of process allowed by the Oracle software user:

   /usr/sbin/lsattr -E -l sys0 -a maxuproc
   

   If necessary, change the maxuproc setting using the following command:

   /usr/sbin/chdev -l sys0 -a maxuproc=16384
   

3. Repeat this procedure on all other nodes in the cluster.

   Caution:

   Use shell programs supported by your operating system vendor. If you use a shell program that is not supported by your operating system, then you can encounter errors during installation.

2.11.6 Configuring User Process Parameters

Verify that the maximum number of processes allowed for each user is set to 2048 or greater:

1. Enter the following command:

   # smit chgsys
   

2. Verify that the value shown for Maximum number of PROCESSES allowed for each user is greater than or equal to 2048.

   If necessary, edit the existing value.

3. When you have finished making changes, press Enter, then Esc+0 (Exit) to exit.

2.11.7 Configuring Network Tuning Parameters

Verify that the network tuning parameters shown in the following table are set to the values shown or higher values. The procedure following the table describes how to verify and set the values.

To view the current value specified for these parameters, and to change them if necessary:

1. To check the current values of the network tuning parameters, enter commands similar to the following:

   # no -a | more
   

2. If you must change the value of any parameter, then enter the following command to determine whether the system is running in compatibility mode:

   # lsattr -E -l sys0 -a pre520tune
   

   If the system is running in compatibility mode, then the output is similar to the following, showing that the value of the pre520tune attribute is enabled:

   pre520tune enable Pre-520 tuning compatibility mode True
   

3. If the system is running in compatibility mode, then follow these steps to change the parameter values:

   1. Enter commands similar to the following to change the value of each parameter:

      # no -o parameter_name=value
      

      For example:

      # no -o udp_recvspace=655360
      

   2. Add entries similar to the following to the /etc/rc.net file for each parameter that you changed in the previous step:

      if [ -f /usr/sbin/no ] ; then
         /usr/sbin/no -o udp_sendspace=65536
         /usr/sbin/no -o udp_recvspace=655360
         /usr/sbin/no -o tcp_sendspace=65536
         /usr/sbin/no -o tcp_recvspace=65536
         /usr/sbin/no -o rfc1323=1
         /usr/sbin/no -o sb_max=4194304
         /usr/sbin/no -o ipqmaxlen=512
      fi
      

      By adding these lines to the /etc/rc.net file, the values persist when the system restarts.

      You can also set RFC1323 value without restarting the system:

      chdev -l en5 -a rfc1323=1
      

4. If the system is not running in compatibility mode, then enter commands similar to the following to change the parameter values:

   • ipqmaxlen parameter:

     /usr/sbin/no -r -o ipqmaxlen=512
     

   • Other parameter:

     /usr/sbin/no -p -o parameter=value
     

   Note:

   If you modify the ipqmaxlen parameter, then you must restart the system.

   These commands modify the /etc/tunables/nextboot file, causing the attribute values to persist when the system restarts.

5. If you need to change parameters, and you do not restart your system, then use the ifconfig command to check each network parameter after you change the no global setting.

   For example:

   # ifconfig en0
   en0:
   flags=1e080863,2c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,6
   4BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN,MONITOR>
           inet 192.0.2.1 netmask 0xfffff800 broadcast 192.0.2.0
           inet 192.0.2.2 netmask 0xfffff800 broadcast 192.0.2.0
           inet 192.0.2.3 netmask 0xfffff800 broadcast 192.0.2.0
           inet 192.0.2.4 netmask 0xfffff800 broadcast 192.0.2.0
            tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0 
   

   For the ISNO parameter tcp_sendspace, use the following command to set it:

   # ifconfig en0 tcp_sendspace 65536
   

   See Also:

   The IBM documentation for more information about tuning AIX parameters for your system.

2.14.1 Environment Requirements for Oracle Grid Infrastructure Software Owner

You must make the following changes to configure the Oracle Grid Infrastructure software owner environment:

• Set the installation software owner user (grid, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

• Set ulimit settings for file descriptors and processes for the installation software owner (grid, oracle)

• Set the software owner's environment variable DISPLAY environment variables in preparation for the Oracle Grid Infrastructure installation

2.14.2 Environment Requirements for Oracle Database and Oracle ASM Owners

If you intend to install Oracle Database or Oracle ASM, then complete the following additional tasks. If you plan to install other software using the role-based privileges method, then complete the following tasks for the Oracle Database software owner (oracle) and Oracle ASM software owner (asm).

• Create an Oracle Base path. The Optimal Flexible Architecture path for the Oracle Base is /u01/app/user, where user is the name of the user account that you want to own the Oracle Database software. For example: /u01/app/oracle.

  Note:

  Do not create the Oracle Clusterware home under Oracle base. Creating an Oracle Clusterware installation in an Oracle base directory path will cause succeeding Oracle installations to fail.

• Set the installation software owner user (asm, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

• Set the software owners' environment variable DISPLAY environment variables in preparation for the Oracle ASM or Oracle Database installation

2.14.3 Procedure for Configuring Oracle Software Owner Environments

To set the Oracle software owners' environments, follow these steps, for each software owner (grid, oracle):

1. Start a new terminal session; for example, start an X terminal (xterm).

2. Enter the following command to ensure that X Window applications can display on this system:

   $ xhost + hostname
   

   The hostname is the name of the local host.

3. If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.

4. If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the grid user:

   $ su - grid
   

5. To determine the default shell for the user, enter the following command:

   $ echo $SHELL
   

6. Open the user's shell startup file in any text editor:

   • Bourne shell (sh) or Korn shell (ksh):

     % vi .profile
     

   • C shell (csh or tcsh):

     % vi .login
     

7. Enter or edit the following line, specifying a value of 022 for the default file mode creation mask:

   umask 022
   

8. If the ORACLE_SID, ORACLE_HOME, or ORACLE_BASE environment variable is set in the file, then remove the appropriate lines from the file.

9. Save the file, and exit from the text editor.

10. To run the shell startup script, enter one of the following commands:

    • Bourne, Bash, or Korn shell:

      $ . ./.profile
      

    • C shell:

      % source ./.login
      

11. If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:

    • Bourne, Bash, or Korn shell:

      $ DISPLAY=local_host:0.0 ; export DISPLAY
      

    • C shell:

      % setenv DISPLAY local_host:0.0
      

    In this example, local_host is the host name or IP address of the system that you want to use to display OUI (your workstation or PC).

12. If you determined that the /tmp directory has less than 1 GB MB of free disk space, then identify a file system with at least 1 GB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system:

    Note:

    You cannot use a shared file system as the location of the temporary file directory (typically /tmp) for Oracle RAC installation. If you place /tmp on a shared file system, then the installation fails.

    1. Use the df -k command to identify a suitable file system with sufficient free space.

    2. If necessary, enter commands similar to the following to create a temporary directory on the file system that you identified, and set the appropriate permissions on the directory:

       $ su - root
       # mkdir /mount_point/tmp
       # chmod 775 /mount_point/tmp
       # exit
       

    3. Enter commands similar to the following to set the TEMP and TMPDIR environment variables:

       • Bourne, Bash, or Korn shell:

         $ TEMP=/mount_point/tmp
         $ TMPDIR=/mount_point/tmp
         $ export TEMP TMPDIR
         

       • C shell:

         % setenv TEMP /mount_point/tmp
         % setenv TMPDIR /mount_point/tmp
         

2.14.4 Setting Display and X11 Forwarding Configuration

If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:

Bourne, Korn, and Bash shells

$ export DISPLAY=hostname:0

C shell:

$ setenv DISPLAY hostname:0

For example, if you are using the Bash shell, and if your host name is node1, then enter the following command:

$ export DISPLAY=node1:0

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:

1. Using any text editor, edit or create the software installation owner's ~/.ssh/config file.

2. Make sure that the ForwardX11 attribute is set to no. For example:

   Host *
         ForwardX11 no
   

2.14.5 Preventing Installation Errors Caused by Terminal Output Commands

During an Oracle Grid Infrastructure installation, OUI uses SSH to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc or .cshrc) will cause makefile and other installation errors if they contain stty commands.

To avoid this problem, you must modify these files in each Oracle installation owner user home directory to suppress all output on STDOUT or STDERR (for example, stty, xtitle, and other such commands) as in the following examples:

• Bourne, Bash, or Korn shell:

  if [ -t 0 ]; then
     stty intr ^C
  fi
  

• C shell:

  test -t 0
  if ($status == 0) then
     stty intr ^C
  endif
  

  Note:

  When SSH is not available, the Installer uses the rsh and rcp commands instead of ssh and scp.

  If there are hidden files that contain stty commands that are loaded by the remote shell, then OUI indicates an error and stops the installation.