D Oracle RAC Configuration Audit Tool

When to Use RACcheck

You can use RACcheck at any time but Oracle recommends that you use RACcheck:

• After initially deploying Oracle RAC

• Before and after any planned system maintenance

• At least once every 90 days

Options to Use with RACcheck

Table D-1 lists and describes options you can use with RACcheck.

-a

-b

-p

-v

-m

-u

$ raccheck -u -o pre

-o

-f

-clusternodes

-localonly

-nopass

-diff report_1 report_2
 [-outfile output_HTML]

-daemon

-nodaemon

-d command

-set "parameter_1=value_1
;parameter_2=value_2..."

-get parameter | all

-profile profile_name

-h

Guidelines for Using RACcheck

If the oracle user exists on the system and all the Oracle components are installed or running (including Oracle Clusterware, Oracle Database, and Oracle ASM), then Oracle recommends that you run RACcheck as the oracle (or Oracle Database software installation owner) user. RACcheck does perform some audit checks that require root privileges, in which case, if sudo is not configured or the root password is not available, then you can configure RACcheck to skip these audit checks by selecting option 3 on the root password menu.

RACcheck can run as root only if you specify the sysadmin profile using the raccheck -profile sysadmin command. In this case, RACcheck skips all database-related best practices but you must set up SSH user equivalence for the root user among cluster nodes, so RACcheck will not prompt you for the root password.

How RACcheck Handles Passwords

RACcheck does not store or save any passwords. The handling of root passwords depends on whether the expect utility is installed.

If the expect utility is not installed (which is the default for all platforms, except Oracle Enterprise Linux 5), then the root password prompts are deferred and you must closely monitor RACcheck as it runs and enter the passwords, as prompted, once for each node of the cluster. Otherwise, RACcheck uses the expect utility for interactive password automation. You can install the expect utility on other Linux distributions in order to automate interactive password handling.

When RACcheck finds the expect utility, the tool gathers the root passwords at the beginning of the process, and the expect utility will supply them, when needed, at the root password prompts for each node, so that RACcheck can continue without further input from you.

RACcheck inquires if the root password is the same for all database servers of the cluster. If you respond affirmatively (which is the default), then you will be prompted for the root password once and it will be validated and subsequently used for all nodes in the cluster. If you respond negatively (that the root password is not the same for all nodes in the cluster), then RACcheck will prompt for and validate the root password for each individual node in the cluster.

Additionally, when RACcheck finds the expect utility, when validating the root passwords, you have three opportunities to type the correct password. If you fail to enter the correct password after three attempts, then RACcheck proceeds to the next node and displays a message stating that the password is still incorrect and that the checks dependent upon data collected from that node will be skipped. At this point, you can either cancel running RACcheck and obtain the correct root password, or continue with the understanding that important data may be missing from the report.

When RACcheck uses the expect utility, it is possible that, between the time that the root passwords are entered and validated and nodes for those passwords are reached, that the passwords could have been changed. In that case, RACcheck displays a message stating that the password must have been changed and that the collections for that node will be skipped, which means the checks for that node will also be skipped. You can either allow RACcheck to continue to completion knowing that data and checks will be skipped or cancel running RACcheck and resolve the problem.

If RACcheck skips any checks for any reason, then the tool reports in the log any checks that were skipped and on which nodes, when it finishes running.

Running RACcheck in Silent Mode without the RACcheck Daemon

To run RACcheck in silent mode:

1. You must first configure SSH user equivalence. For instructions about configuring SSH user equivalence, go to the following URL:

   https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=372795.1

   Note:

   SSH user equivalence is only necessary if your system is running Oracle Grid Infrastructure to support a cluster. This is not required for nonclustered databases or Oracle Restart configurations.

2. RACcheck requires root access to run the root-specific checks. To facilitate these checks in silent mode, you must configure passwordless sudo and use the -s option to run RACcheck.

   Note:

   If sudo is not allowed within your environment, you can skip this step. In this case, you can still run RACcheck silently without the root-specific checks using the -S option. Eliminating the root-specific checks limits the capabilities of RACcheck and Oracle does not recommend this method of running RACcheck.

3. To run RACcheck silently, you must specify one of the following arguments, depending on your configuration:

   • -s: Use this option to run RACcheck unattended when the oracle user can sudo to root without a password

   • -S: Use this option to run RACcheck unattended without the root password and make no root-privileged collections or audits

   Note:

   Oracle recommends that you implement passwordless sudo to root for /tmp/root_raccheck.sh for full functionality of RACcheck, as described in "RACcheck Usage".

   RACcheck runs in silent mode using the information it gathers from Oracle Clusterware. When RACcheck runs in silent mode, it performs data collection and audit checks on all databases running on the local node that are defined as Oracle Clusterware resources.

Running RACcheck Using the RACcheck Daemon

To use the RACcheck daemon, you must first configure the daemon parameters by running the raccheck -set parameter=value command. You can set the following configuration parameters:

• AUTORUN_INTERVAL: This parameter defines the time interval at which RACcheck runs, specified in days or hours (d | h). For example:

  $ raccheck -set AUTORUN_INTERVAL=1d
  

  The preceding command configures the RACcheck daemon to run every day. If you set this parameter to 0, then the RACcheck daemon will not run automatically. This is the default setting for this parameter.

• AUTORUN_FLAGS: This parameter defines how RACcheck runs, using the flags listed in Table D-1. For example:

  $ raccheck -set "AUTORUN_INTERVAL=12h;AUTORUN_FLAGS=-profile sysadmin"
  

  The preceding command configures RACcheck to run the sysadmin profile every 12 hours.

• PASSWORD_CHECK_INTERVAL: This parameter defines the frequency (specified in hours) with which the running daemon validates the passwords entered when the daemon starts. If the daemon encounters an invalid password (due to a password change), then the daemon stops running and enters a notification in the daemon log (raccheck_daemon.log), and also sends an email, if configured to do so. For example:

  $ raccheck -set PASSWORD_CHECK_INTERVAL=1
  

  The preceding command validates passwords every hour.

• NOTIFICATION_EMAIL: This parameter configures the RACcheck daemon to email notifications to a specific person. For example:

  $ raccheck -set NOTIFICATION_EMAIL=firstname.lastname@company.com
  

$ raccheck -set "AUTORUN_INTERVAL=1d;AUTORUN_FLAGS= -o verbose;
NOTIFICATION_EMAIL=firstname.lastname@company.com;PASSWORD_CHECK_INTERVAL=1"

To obtain the current configuration of all of the parameters of the RACcheck daemon, run the following command:

$ raccheck -get all

The preceding command returns output similar to the following:

AUTORUN_INTERVAL = 1d
AUTORUN_FLAGS = -o verbose
NOTIFICATION_EMAIL = firstname.lastname@company.com
PASSWORD_CHECK_INTERVAL = 1

Comparing Reports with RACcheck

You can use RACcheck to compare results from two RACcheck reports. You can use the results of comparisons to monitor trends and best practice changes over time or after planned maintenance presented in a user-friendly HTML report. Ensure that the RACcheck reports in the RACcheck output directories, .zip output files, or HTML reports are accessible.

To compare two RACcheck reports, run the following command:

$ raccheck -diff report_1 report_2

Specify the names of the two report files you want to compare. When RACcheck finishes comparing the two files, the utility prints a summary and provides a location of the comparison report for viewing.

Running RACcheck to Determine Upgrade Readiness

You can use RACcheck to obtain an automated upgrade readiness assessment to help facilitate upgrade planning for Oracle RAC and Oracle Clusterware. Use RACcheck to automate many of the manual pre- and post-upgrade checks detailed in the following upgrade-related documents:

• Oracle Database Upgrade Guide

• Oracle Grid Infrastructure Installation Guide for your platform

• The following My Oracle Support notes:

  • https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=785351.1

  • https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=837570.1

  • https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1363369.1

  • https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=810394.1

Maintaining the Output File

When you run RACcheck, the tool creates a subdirectory using a naming convention that begins with "raccheck" and includes a date and time (such as, raccheck_SIEBXL_072613_141001), and a zip file that contains the contents of the subdirectory (such as, raccheck_SIEBXL_072611_141001.zip) at the same level on the file system as RACcheck, itself. The total size of the subdirectory and zip file should be less than 5 MB on the file system. The exact varies depending upon how many nodes and how many databases there are in the system. While Oracle recommends that you run RACcheck when there is the least load on the system, over time, the number of files will build up and you must maintain and clean out older files, subdirectories, or both.

Multiple Database Support

Oracle designed RACcheck to support multiple databases. The tool presents a list of running databases, which are registered in the Oracle Grid Infrastructure. You can choose one, all, or enter a comma-delimited list of numbers that designate the listed databases. You do not have to install the tool on multiple nodes to check database instances running on other nodes in the cluster.

RACcheck logs into all databases by using local bequeath connections and assumes the existence of operating system authentication in the database for the user running the tool. In some configurations, there could be multiple database homes all owned by the same operating system user (such as oracle), while in other configurations, there could be any number of database homes all owned by different operating system users.

In the former case, run RACcheck as oracle. In the latter case, you may want to deploy RACcheck on the home with the greatest number of databases to obtain the most information possible, in which case install RACcheck as the owner of that home. To scan the other databases, RACcheck must be installed and run under each database home user account.

Uploading RACcheck Results and Patches to a Database for Reporting

Oracle supports uploading the results of audit checks done by RACcheck or the list of installed patches into database tables for use as a source of data for reporting.

To take advantage of this optional feature, you must set a number of environment variables in the runtime environment, and you must create two tables to receive the data: one for the audit check results and another for the patches installed on the systems.

To create a results table called auditcheck_result, run the following DDL command:

create table
  auditcheck_result
    (
    COLLECTION_DATE TIMESTAMP,
     CHECK_NAME VARCHAR2(256),
     PARAM_NAME VARCHAR2(256),
     STATUS VARCHAR2(256),
     STATUS_MESSAGE VARCHAR2(256),
     ACTUAL_VALUE VARCHAR2(256),
     RECOMMENDED_VALUE VARCHAR2(256),
     COMPARISON_OPERATOR VARCHAR2(256),
     HOSTNAME VARCHAR2(256),
     INSTANCE_NAME VARCHAR2(256),
     CHECK_TYPE VARCHAR2(256),
     DB_PLATFORM VARCHAR2(256),
     OS_DISTRO VARCHAR2(256),
     OS_KERNEL VARCHAR2(256),
     OS_VERSION NUMBER,
     DB_VERSION VARCHAR2(256),
     CLUSTER_NAME VARCHAR2(256),
     DB_NAME VARCHAR2(256),
     ERROR_TEXT VARCHAR2(256)
     CHECK_ID VARCHAR2(40),
     NEEDS_RUNNING VARCHAR2(100),
     MODULES VARCHAR2(4000),
     DATABASE_ROLE VARCHAR2(100),
     CLUSTERWARE_VERSION VARCHAR2(100),
     GLOBAL_NAME VARCHAR2(256)
);

To create a table for patches called auditcheck_patch_result, run the following DDL command:

create table
  auditcheck_patch_result
       ( COLLECTION_DATE     TIMESTAMP(6),
          HOSTNAME            VARCHAR2(256),
          ORACLE_HOME_TYPE    VARCHAR2(256),
          ORACLE_HOME_PATH    VARCHAR2(256),
          ORACLE_HOME_VERSION VARCHAR2(256),
          PATCH_NUMBER NUMBER,
         CLUSTER_NAME VARCHAR2(256),
         DESCRIPTION         VARCHAR2(256),
         PATCH_TYPE VARCHAR2(128),
         APPLIED NUMBER,
         RECOMMENDED NUMBER
         );

Set the following environment variables (shown with example values):

$ export RAT_UPLOAD_CONNECT_STRING="(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)
(HOST = bonanza)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl)))"
$ export RAT_UPLOAD_TABLE=auditcheck_result (name must match that of the table created for the purpose)
$ export RAT_PATCH_UPLOAD_TABLE=auditcheck_patch_result (name must match the name of the table created for the purpose)
$ export RAT_UPLOAD_USER=auditcheck (schema owner of the table created for the purpose)
$ export RAT_UPLOAD_PASSWORD=auditcheck (password for the schema owner)
$ export RAT_UPLOAD_ORACLE_HOME=path_of_database_home (optional, alternate home
containing sqlplus that you want to use for connecting in case it is not the
current $ORACLE_HOME as derived by RACcheck from the environment)

Excluding Audit Checks

Optionally, you can exclude one or more audit checks after the first run of RACcheck, as follows:

1. Create a text file named excluded_check_ids.txt in the same directory as the RACcheck script and driver files.

2. Open the raccheck.log file, which is located in the output directory of the previous run of the tool.

3. Search for the audit checks that you want to exclude in subsequent runs.

4. Note the CHECK_ID of the audit checks you want to exclude. The CHECK ID is an alphanumeric string similar to CHECK_ID = 65E4DC8B76BC4DA6E040E50A1EC03704. If the audit check block does not contain a CHECK_ID line, then you cannot exclude that audit check.

5. Enter the CHECK_IDs for the audit checks that you want to exclude in the excluded_check_ids.txt file, one CHECK_ID per line.

6. Subsequently, when you run RACcheck, before the tool runs an audit check, it checks the excluded_check_ids.txt file for any excluded checks and skips them.

Runtime Command Timeouts

If any non-root-privileged individual commands timeout before they finish running, then RACcheck stops that process and does not collect the desired data. If this happens, then you can lengthen the timeout by setting the following environment variable in the script execution environment:

$ export RAT_TIMEOUT=120

The default value for this environment variable is 90 seconds.

RACcheck runs a set of root-privileged data collections, once for each node in the cluster. If collections timeout before the data can be collected, then RACcheck stops that process and does not collect the desired data. If this happens, then you can lengthen the timeout by setting the following environment variable in the script execution environment:

$ export RAT_ROOT_TIMEOUT=600

The default value for this environment variable is 300 seconds.

The raccheck_error.log File Contains Errors

You can ignore the following errors that do appear in the raccheck_error.log file:

• /bin/sh: /u01/app/11.2.0/grid/OPatch/opatch: Permission denied

• chmod: changing permissions of '/u01/app/oracle_ebs/product/11.2.0.2/VIS_RAC/.patch_storage': Operation not permitted

• OPatch could not open log file, logging will not be possible

• Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

The preceding errors occur in role-separated environments when RACcheck, which is run as the database software owner, attempts to list the patch inventories of homes owned by other users (grid, for example, or other database home owners) using OPatch. When you run OPatch to list the patch inventories for those other users it causes errors because the current user does not have permissions on the other homes. In these cases the OPatch errors are ignored and the patch inventories for those homes are gathered by other means.

Additionally, you can ignore errors similar to the following:

./raccheck: line [N]: [: : integer expression expected

The line number may change over time but this error occurs when RACcheck expects an integer return value but no value was found. RACcheck returns that error when trying to make the comparison. You could see this error repeated many times for the same command, once for each node.

Remote root Login Problems

If remote root login is not permitted over SSH, then the root-privileged commands will fail. To verify the Oracle software owner (oracle), run the following command manually from whichever node is not working and ensure you get similar output, as follows:

$ ssh root@remotehostname "id"
root@remotehostname's password:
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

If remote root login is not working, then RACcheck cannot check the remote nodes. Contact a system administrator to correct this, if only temporarily, for running RACcheck.

If you can configure the remote root login, then edit the /etc/ssh/sshd_config file, as follows:

PermitRootLogin to yes

Run the following command as root on all nodes of the cluster:

# /etc/init.d/sshd restart

Local Environmental Issues

RACcheck attempts to derive all the data it requires from the environment (operating system and Oracle Cluster Registry) but the tool may not work as expected because of local system differences, and it is difficult to anticipate and test every possible scenario. Therefore, support for a number of environment variables has been included so that you can override the default behavior of the tool or provide the required information. Following is a list of environment variables and descriptions for each:

• RAT_INV_LOC: If the oraInst.loc file is not located where expected (/u01/app/oraInventory), then you can specify the correct location of the oraInventory directory in this environment variable.

• RAT_CRS_HOME: You can set this environment variable if RACcheck cannot determine the correct Grid_home path, and displays information stating that the software is not installed, even though you know that Oracle Clusterware is installed.

• RAT_ORACLE_HOME: You can set this environment variable if RACcheck cannot determine the correct ORACLE_HOME paths for the databases registered with Oracle Clusterware, and displays information stating that the software is not installed, even though you know that the database software is installed. RACcheck performs best practice and recommended patch checks for all the databases running from the home specified in this environment variable.

• RAT_ASM_HOME: You can set this environment variable if RACcheck cannot determine the correct Oracle ASM home path (which is the same as the Grid_home path) from Oracle Clusterware, and displays information stating that the software is not installed, even though you know that the Oracle ASM software is installed.

• RAT_OS: You can set this environment variable if RACcheck fails to determine the correct platform, and the tool informs you that the data necessary for the determined platform could not be found.

• RAT_DB: You can set this environment variable if RACcheck determines an incorrect database version.

• RAT_DBNAMES: You can set this environment variable if RACcheck fails to determine valid database names from Oracle Clusterware. You can specify a space-delimited list of database names, and RACcheck will use that list instead of what it derives from Oracle Clusterware. For example:

  $ export RAT_DBNAMES="ORCL ORADB PROD"
  

  Use double quotation marks ("") if specifying more than one database name.

  Note:

  If you configure RAT_DBNAMES as a subset of databases registered with Oracle Clusterware, and you want the patch inventories of all databases found registered with Oracle Clusterware to have their patch inventories checked for recommended patches, then Oracle recommends that you also configure RAT_DBHOMES.

• RAT_DBHOMES: If you set the RAT_DBNAMES environment variable, then, by default, the recommended patch analysis will be limited to the homes for the database names you listed. If you want to perform the recommended patch analysis for additional database homes than those specified in RAT_DBNAMES, then specify a space-delimited list of databases whose homes you want checked for recommended patches. For example, assume that you run the export RAT_DBNAMES="ORCL ORADB" command but that you also want to check the PROD database home, even if the PROD database is down. Run the export RAT_DBHOMES="ORCL ORADB PROD" command so that best practices will be checked for the ORCL and ORADB databases but the recommended patches will be checked for the ORCL, ORADB, and PROD database homes. Use double quotation marks ("") if specifying more than one database name.

• RAT_SSHELL: Set this environment variable to specify a secure shell location that overrides the default location (typically, /usr/bin/ssh), in case ssh is not where it is expected, and ssh commands return the following error:

  -bash: /usr/bin/ssh -q: No such file or directory
  

• RAT_SCOPY: Set this environment variable to specify a secure copy location that overrides the default secure copy location (typically, /usr/bin/scp), in case scp is not where it is expected, and scp commands return the following error:

  /usr/bin/scp -q: No such file or directory
  

Database Login Problems

If you intend to run RACcheck as a user other than the database software installation owner (such as root or grid), and if you experience problems connecting to the database, then do the following:

1. Log in to the system as grid.

2. Run the following commands:

   $ export ORACLE_HOME=path_to_Oracle_Database_home
   $ export ORACLE_SID=database_SID
   $ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/lib:$PATH
   

3. Add an alias in the $ORACLE_HOME/network/admin/tnsnames.ora file for the database_SID.

4. Connect to the database as follows, including the password:

   $ORACLE_HOME/bin/sqlplus sys@SID as sysdba
   

5. Ensure that you have successful connection.

If this method of connecting to the database fails, then RACcheck will be unable to connect, as well. Consider running the tool as the Oracle database software installation owner.

User Profiles

The presence of prompts or traps in the user profile can lead to RACcheck hanging while it is running, because it sources the profile at runtime. For this reason, RACcheck checks the profile in the environment for these statements and presents a message advising you to temporarily comment out those statements on all nodes.