5 Switchover and Failover Operations

5.2.1 Choosing a Target Standby Database for Switchover

When performing a switchover in a configuration whose standby databases are all of the same type (all physical or all logical standby databases), choose the standby database that has the least amount of unapplied redo. By choosing the standby database with the least amount of unapplied redo, you can minimize the overall time it takes to complete the switchover operation. For example:

• Using DGMGRL, you can do this by examining the Apply Lag row of the SHOW DATABASE output for each standby database in the configuration.

• Using Enterprise Manager, you can view the value of the ApplyLag column for each standby database in the Standby Databases section of the Data Guard Overview Page.

If the configuration contains both physical and logical standby databases, consider choosing a physical standby database (that has the least amount of unapplied redo) to be the target standby database. A switchover to a physical standby database is preferable because all databases in the configuration will be available as standby databases to the new primary database after the switchover operation completes. Whereas a switchover to a logical standby database will invalidate and disable all of the physical and snapshot standby databases in the configuration. You will then need to re-create the physical standby databases from a copy of the new primary database before you can reenable them.

You cannot perform a switchover to a snapshot standby database unless you first convert it back to a physical standby database.

5.2.2 Choosing a Target Standby Database for Failover

When performing a failover in a configuration whose standbys are all of the same type, choose the standby database that has the smallest transport lag. Doing so can minimize the amount of data loss and in some cases, incur no data loss at all.

If the configuration contains physical, snapshot, and logical standby databases, consider choosing a physical standby database as the target standby database. A failover to a physical standby database is preferable because it is likely that all standby databases in the configuration will still be available as standby databases to the new primary database after the failover operation completes.

You may failover to a snapshot standby database. However failing over to a snapshot standby database will require more time because the broker must first convert it back to a physical standby database. After the conversion, the broker will start Redo Apply to apply accumulated redo data, before failing the database over to the primary role. Because the broker performs the failover after converting the snapshot standby database to a physical standby database, it is likely that all standby databases in the configuration will still be available as standby databases to the new primary database after the failover operation completes.

A failover to a logical standby database requires that all physical and snapshot standby databases be re-created from a copy of the new primary database after the failover completes. In addition, a logical standby database may contain only a subset of the data present in the primary database. (For example, if the DBMS_LOGSTDBY.SKIP procedure was used to specify which database operations done on the primary database will not be applied to the logical standby database.)

However, there may be exceptions to the recommendation to choose a physical standby database as the target standby database. For example, if all your physical standbys are also unavailable, then failing over to a logical standby is your only choice.

5.3.1 Before You Perform a Switchover Operation

Consider the following points before you begin a switchover:

• When you start a switchover, the broker verifies that at least one standby database, including the primary database that is about to be transitioned to the standby role, is configured to support the overall protection mode (maximum protection, maximum availability, or maximum performance) after the switchover is completed.

• Prepare the primary database in advance for its possible future role as a standby database in the context of the overall protection mode (see Section 4.6). Such preparation includes:

  • Ensuring that standby redo log files are configured on the primary database.

  • Presetting database properties related to redo transport services, such as LogXptMode, NetTimeout, StandbyArchiveLocation, and AlternateLocation. For more details about managing redo transport services using database properties, see Section 4.4.

  • Presetting database properties related to Redo Apply services, such as DelayMins. For more details about managing Redo Apply services using properties, see Section 4.5.

  • For each temporary table, verifying that temporary files associated with that table on the primary database also exist on the standby database.

  Note that the broker does not use the properties to set up redo transport services and Redo Apply services until you actually switch over the primary database to the standby role. Thus, the validity of the values of these properties is not verified until after the switchover. Once you set these properties, their values persist through role changes during switchover and failover.

• Before performing a switchover to a physical standby database that is in real-time query mode, consider bringing all instances of that standby database to the mounted but not open state to achieve the fastest possible role transition and to cleanly terminate any user sessions connected to the physical standby database prior to the role transition.

• If fast-start failover is enabled in either maximum availability or maximum performance mode, a switchover can be performed only to the pre-specified target standby database and only if the standby database is synchronized with the primary database. For information about enabling fast-start failover, see Section 5.5.2.

After a switchover completes, the broker preserves the overall Data Guard protection mode as part of the switchover process by keeping the protection mode at the same protection level (maximum protection, maximum availability, or maximum performance) it was at before the switchover. Also, the network transmission mode (SYNC or ASYNC) for transporting redo to other bystander standby databases does not change after a switchover. Apply services on all other bystander standby databases automatically begin applying redo data received from the new primary database.

If there are physical or snapshot standby databases in the configuration and the switchover occurs to a logical standby database, you need to re-create those databases from a copy of the new primary database and then reenable those databases, as described in Section 5.4.3.

5.3.2 Starting a Switchover

The act of switching roles should be a well-planned activity. The primary and standby databases involved in the switchover should have as small a redo lag as possible. Oracle Data Guard Concepts and Administration provides information about setting up the databases in preparation of a switchover.

To start a switchover using Enterprise Manager, select the standby database that you want to change to the primary role and click Switchover. When using DGMGRL, you need to issue only one SWITCHOVER command to specify the name of the standby database that you want to change into the primary role.

The broker controls the rest of the switchover, as described in Section 5.3.3.

5.3.3 How the Broker Performs a Switchover

Once you start the switchover, the broker:

1. Verifies that the primary and the target standby databases are in the following states:

   1. The primary database is enabled and is in the TRANSPORT-ON state.

   2. The target standby database is enabled and is in the APPLY-ON state.

   The broker allows the switchover to proceed as long as there are no errors for the primary database and the standby database that you selected to participate in the switchover operation. Errors occurring for any other bystander standby databases will not impede the switchover.

2. Shuts down all instances except one, if required.

   If you are switching over to a physical standby database, the broker shuts down all but one instance on the current primary database. No instances will be shut down on the target physical standby database.

   No instances will be shut down if switching over to a logical standby database. You cannot switch over to a snapshot standby database.

3. Switches roles between the primary and standby databases.

   The broker first converts the original primary database to run in the standby role. Then, the broker transitions the target standby database to the primary role. If any errors occur during either conversion, the broker stops the switchover. See Section 9.3, "Troubleshooting Problems During a Switchover Operation" for more information.

4. Updates the broker configuration file to record the change in roles.

   Because the configuration file profiles all database objects in the configuration, this ensures that each database will run in the correct role and state should it be restarted later for any reason.

5. Restarts the new standby (former primary) database if the switchover occurs to a physical standby database, and Redo Apply begins applying redo data from the new primary database. If this is an Oracle RAC physical standby database, the broker directs Oracle Clusterware to restart the instances that were shut down prior to the switchover. In a configuration operating in maximum protection mode, the new primary database will also be restarted.

6. The new primary database is opened in read/write mode and redo transport services are started.

   If the former physical standby database was running with real-time query enabled, the new physical standby database will run with real-time query enabled.

The broker verifies the state and status of the databases to ensure that the switchover transitioned the databases to their new role correctly. Bystander standby databases that are not disabled by the broker after the switchover will continue operating in the state they were in before the switchover. Redo Apply and SQL Apply on all other bystander standby databases automatically begin applying redo data received from the new primary database.

5.4.1 Complete and Immediate Manual Failovers

Using Enterprise Manager or DGMGRL, you can perform either a complete (recommended) or an immediate failover:

• A complete failover is the recommended and default failover option. It automatically recovers the maximum amount of redo data for the protection mode the configuration is operating in. A complete failover also attempts to avoid disabling any standby databases that were not the target of the failover, so that they may continue serving as standby databases to the new primary database.

  Whether or not standby databases that were not the target of failover (bystander standby databases) are disabled depends upon how much redo data they have applied relative to the failover target and the standby type of the failover target:

  • If the failover target is a physical or snapshot standby database, the original primary database must be reinstated or re-created in order to be a standby database for the new primary database. In addition, some standby databases may be disabled by the broker during the failover if the broker detects that they have applied redo beyond where the new primary database had applied. Any standby database that was disabled by the broker must be reinstated or re-created, as described in Section 5.4.3, before it can be a standby database for the new primary database.

    Note that if failover was performed on a snapshot standby database, the old primary must be either reinstated or re-created as a physical standby database.

  • If the failover target is a logical standby database, the original primary database and all physical and snapshot standby databases in the configuration will be disabled. The primary database can be reinstated if it had flashback database enabled. The physical and snapshot standby databases will have to be re-created from a copy of the new primary database. See Section 5.4.3 for more information.

  If the primary database can be mounted, it may be possible to flush any unsent redo data from the primary database to the target standby database using the ALTER SYSTEM FLUSH REDO SQL statement. If this operation is successful, a zero data loss failover may be possible even if the primary database is not in a zero data loss protection mode. See Oracle Data Guard Concepts and Administration for more information on using the ALTER SYSTEM FLUSH REDO statement.

  During a complete failover, the broker performs the failover steps described in Section 5.4.2.1.

• An immediate failover is the fastest type of failover. However, no additional data is applied on the standby database once you invoke the failover. Another consequence of immediate failover is that all other databases in the configuration are disabled and must be reinstated or re-created before they can serve as standby databases for the new primary database. Section 5.4.3 describes how to do this. During an immediate failover, the broker performs the failover steps described in Section 5.4.2.2.

  Caution:

  Always try to perform a complete failover first unless redo apply has stopped at the failover target due to an ORA-752 or ORA-600 [3020] error. If one of these errors has occurred, follow the guidelines in "Resolving ORA-752 or ORA-600 [3020] During Standby Recovery" in My Oracle Support Note 1265884.1 before proceeding. This support note is available at http://support.oracle.com.

  An immediate failover should only be performed when a complete failover is unsuccessful or in the case just noted above. A complete failover can occur without any data loss, depending on the destination attributes of redo transport services, but an immediate failover usually results in some data loss.

5.4.2 Performing a Manual Failover Operation

After determining that there is no possibility of recovering the primary database in a timely manner, ensure that the primary database is shut down and then begin the failover operation.

The steps in this section describe how to perform a manual failover. Depending on the failover and the types of standby databases involved, some of the databases may need to be reinstated or re-created. The instructions guide you through the appropriate steps for each type of situation.

Step 1   Determine which of the available standby databases is the best target for the failover.

Follow the guidelines described in Section 5.2, "Choosing a Target Standby Database".

Step 2   Start the failover.

Using Enterprise Manager or DGMGRL, perform either a complete (recommended) or an immediate failover.

Manual Failover Using Enterprise Manager:

On the Data Guard Overview Page in Enterprise Manager, select the standby database that you want to change to the primary role and click Failover. Then, on the Failover Confirmation Page, click Yes to invoke the default Complete failover option.

Manual Failover Using DGMGRL:

On the target standby database, issue the FAILOVER command to perform a failover, specifying the name of the standby database that you want to become the primary database:

DGMGRL> FAILOVER TO database-name;

Specify the optional IMMEDIATE clause to perform an immediate failover if any of the the following conditions are true:

• An ORA-752 error has occurred at the standby database

• An ORA-600 [3020] error has occurred at the standby database and Oracle support has determined that it was caused by a lost write at the primary database

• A complete failover is not possible

DGMGRL> FAILOVER TO database-name IMMEDIATE;

See Also:

• FAILOVER

If the target is a snapshot standby database, the broker first converts the database to a physical standby database. If performing a complete failover, all accumulated redo data will be applied before changing the role of the database to primary. If performing an immediate failover, the role of the database will be changed to primary without applying any accumulated redo data.

No instances will be shut down if failing over to a physical or logical standby database.

Step 3   Reset the protection mode.

After a manual failover (complete or immediate), the overall Data Guard protection mode is handled as follows:

• If the protection mode was at maximum protection, it is reset to maximum performance. You can upgrade the protection mode later, if necessary, as described in Section 4.6.1.

• If the protection mode was at maximum availability or maximum performance, it remains unchanged.

Note:

If you perform a manual failover when fast-start failover is enabled:

• The failover can only be performed to the pre-selected target standby database.

• The broker preserves the protection mode that was in effect prior to the failover.

Step 4   Re-establish a disaster-recovery configuration.

To maintain a viable disaster-recovery solution in the event of another disaster, you may need to perform the additional steps described in Section 5.4.3 to:

• Reinstate the original primary database to act as a standby database in the new configuration.

  Caution:

  Do not attempt to reinstate the old primary database if an ORA-752 or ORA-600 [3020] error has occurred at the failover target, because doing so may lead to data loss. Instead, the old primary database must be re-created as a standby from a backup of the new primary using the procedure described in Section 5.4.3.2.

• Reinstate or re-create standby databases in the configuration that were disabled by the broker.

  After a complete failover finishes, any bystander standby database that is not viable as a standby for the new primary database will be disabled by the broker. This can happen for either of the following reasons:

  • A bystander standby database has applied more redo data than the new primary database itself had applied when it was a standby database. The standby database must be re-created or reinstated before it can serve as a standby for the new primary database.

  • The failover was to a logical standby database. The broker disables all of the physical and snapshot standby databases in the configuration. They must be re-created before they can serve as standby to the new primary database.

5.4.3 Reenabling Disabled Databases After a Role Change

To restore your original disaster-recovery solution after switchover to a logical standby database or after failover to any standby database, you may need to perform additional steps.

Databases that have been disabled after a role transition are not removed from the broker configuration, but they are no longer managed by the broker.

To reenable broker management of these databases, you must reinstate or re-create the databases using one of the following procedures:

• If a database can be reinstated, the database will show the following status:

  ORA-16661: the standby database needs to be reinstated
  

  Reinstate the database using the DGMGRL REINSTATE DATABASE command or the reinstate option in Enterprise Manager, as described in Section 5.4.3.1, "How to Reinstate a Database". The broker automatically reenables the database as part of reinstating it.

• If a database must be re-created from a copy of the new primary database, it will have the following status:

  ORA-16795: the standby database needs to be re-created
  

  Re-create the standby database from a copy of the primary database and then reenable it. The procedures for creating a standby database are documented in Oracle Data Guard Concepts and Administration. See Section 5.4.3.2, "How to Re-create and Reenable a Disabled Database" for more information.

Whether you reinstate or re-create a database depends on if you performed a switchover or failover and on the type of standby database that was the target of the operation. Note that role changes to logical and snapshot standby databases will always result in physical standby database bystanders being disabled. They cannot be reinstated. They must be re-created from a copy of the new primary database.

The following sections describe how to reinstate or reenable a database.

ORA-16661: the standby database needs to be reinstated

DGMGRL> REINSTATE DATABASE db_unique_name;

5.5.1 Prerequisites for Enabling Fast-Start Failover

The following prerequisites must be met before the broker allows you to enable fast-start failover:

• Ensure the broker configuration is operating in either maximum availability mode or maximum performance mode.

  See Section 4.6.1 for information about configuring the protection mode, standby redo logs, and the LogXptMode property.

• Ensure that the standby database you choose to be the target of fast-start failover has its LogXptMode property set to SYNC if you wish to enable fast-start failover in maximum availability mode, or to ASYNC if you wish to enable fast-start failover in maximum performance mode. The current primary database must have its LogXptMode property set accordingly and must have standby redo logs configured.

• Enable Flashback Database and set up a fast recovery area on both the primary database and the target standby database.

  See Oracle Database Backup and Recovery User's Guide.

• Install the DGMGRL command-line interface on the observer computer as described in Section 2.1.

• Configure the TNSNAMES.ORA file on the observer system so that the observer is able to connect to the primary database and the pre-selected target standby database.

• Create static service names so that the observer can automatically restart a database as part of reinstatement. See Section 2.2, "Prerequisites" for more information.

5.5.2 Enabling Fast-Start Failover

You can enable fast-start failover from any site while connected to any database in the broker configuration. Enabling fast-start failover does not trigger a failover. Instead, it allows the observer that is monitoring the configuration to initiate a fast-start failover should conditions warrant a failover.

Perform the following steps to enable fast-start failover and start the observer. The steps assume that you are connected as SYS and that a primary and standby database are already set up in a broker configuration.

Step 1   Determine which of the available standby databases is the best target for the failover.

Follow the guidelines described in Section 5.2, "Choosing a Target Standby Database".

Step 2   Specify the target standby database with the FastStartFailoverTarget configuration property.

You can specify only one target standby database when setting the FastStartFailoverTarget configuration property on the current primary database:

• If there is only one standby database in the configuration, you can skip this step and continue with Step 3. When enabling fast-start failover, the broker automatically sets the FastStartFailoverTarget property on the primary and standby databases to point to each other as their respective target during a failover.

• If there is more than one standby database in the configuration, you must explicitly set the FastStartFailoverTarget property on the primary database to select a target standby database. When enabling fast-start failover, the broker verifies that the property indicates an existing standby, and then reciprocally sets the standby database's FastStartFailoverTarget property to the primary database.

  Note:

  To change the FastStartFailoverTarget property to point to a different standby database, disable fast-start failover, set the FastStartFailoverTarget property, and reenable fast-start failover.

See Section 8.3.10, "FastStartFailoverTarget" for more information about this property.

Step 3   Determine the protection mode you want

Fast-start failover can be enabled for either maximum availability mode or maximum performance mode. If you cannot tolerate any loss of data, then ensure that the configuration protection mode is set to maximum availability. To do this, the LogXptMode database property for both the primary and target standby database must be set to SYNC. Then set the configuration protection mode to maximum availability. For example:

DGMGRL> EDIT DATABASE 'North_Sales' SET PROPERTY LogXptMode=SYNC;
DGMGRL> EDIT DATABASE 'South_Sales' SET PROPERTY LogXptMode=SYNC;
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;

If you are more concerned about the performance of the primary database than a minimal loss of data, consider enabling fast-start failover when the configuration protection mode is set to maximum performance. In this mode you will need to consider how much data loss is acceptable in terms of seconds and set the FastStartFailoverLagLimit configuration property accordingly. This property specifies the amount of data, in seconds, that the target standby database can lag behind the primary database in terms of redo applied. If the standby database's redo applied point is within that many seconds of the primary database's redo generation point, a fast-start failover will be allowed. The FastStartFailoverLagLimit configuration property is only used by the broker when enabling fast-start failover for configurations operating in maximum performance mode. The default value is 30 seconds and the lowest possible value is 10 seconds.

In addition to setting the configuration protection mode to maximum performance, you will also need to ensure that the LogXptMode database property for both the primary and target standby database is set to ASYNC. For example:

DGMGRL> EDIT DATABASE 'North_Sales' SET PROPERTY LogXptMode=ASYNC;
DGMGRL> EDIT DATABASE 'South_Sales' SET PROPERTY LogXptMode=ASYNC;
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxPerformance;
DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverLagLimit=45;

Step 4   Set the FastStartFailoverThreshold configuration property.

Fast-start failover will occur if both the observer and the target standby database lose connection to the primary database for the period of time specified by the FastStartFailoverThreshold configuration property.

Set the FastStartFailoverThreshold property to specify the number of seconds you want the observer and target standby database to wait (after detecting the primary database is unavailable) before initiating a failover. For example:

DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = 45;

The default value for the FastStartFailoverThreshold property is 30 seconds and the lowest possible value is 6 seconds. If you have an Oracle RAC primary database, consider specifying a higher value to minimize the possibility of a false failover in the event of an instance failure.

The time interval starts when the observer first loses its connection to the primary database. If the observer is unable to regain a connection to the primary database within the specified time, then the observer begins a fast-start failover provided the standby database is ready to fail over. Although the default value of 30 seconds is typically adequate for detecting outages and failures on most configurations, you can adjust failover sensitivity with this property to decrease the probability of false failovers in a temporarily unstable environment.

If the FastStartFailoverPmyShutdown configuration property is set to TRUE, the primary database will shut down after FastStartFailoverThreshold seconds has elapsed if redo generation has been stalled and the primary database is unable to reestablish connectivity with either the observer or target standby database.

Note that the FastStartFailoverThreshold property can be changed even when fast-start failover is enabled.

See Also:

Section 8.1.8 for reference information about the FastStartFailoverThreshold property

Step 5   Set other database properties (optional).

You can optionally set the database properties described in the following table:

Property Name	Description	Default Value
FastStartFailoverPmyShutdown	This configuration property causes the primary database to shut down if fast-start failover is enabled and V$DATABASE.FS_FAILOVER_STATUS indicates the primary has been STALLED for longer than FastStartFailoverThreshold seconds. A value of TRUE helps to ensure that an isolated primary database cannot satisfy user queries. This property cannot be used to prevent the primary database from shutting down if a fast-start failover occurred because a user configuration condition was detected or was requested by an application by calling the DBMS_DG.INITIATE_FS_FAILOVER function.	TRUE
FastStartFailoverLagLimit	This configuration property establishes an acceptable limit, in seconds, that the standby is allowed to fall behind the primary in terms of redo applied, beyond which a fast-start failover will not be allowed. The lowest possible value is 10 seconds. This property is used when fast-start failover is enabled and the configuration is operating in maximum performance mode.	30 seconds
FastStartFailoverAutoReinstate	This configuration property causes the former primary database to be automatically reinstated if a fast-start failover was initiated because the primary database was either isolated or had crashed. To prevent automatic reinstatement of the former primary database in these cases, set this configuration property to FALSE. The broker never automatically reinstates the former primary database if a fast-start failover was initiated because a user configuration condition was detected or was requested by an application calling the DBMS_DG.INITIATE_FS_FAILOVER function.	TRUE
ObserverConnectIdentifier	This database property is used to specify how the observer should connect to and monitor the primary and standby database. Set this property for the primary and target standby database if you want the observer to use a different connect identifier than that used to ship redo data (that is, the connect identifier specified by the DGConnectIdentifier property).	Observer uses the value of the DGConnectIdentifier property to connect to and monitor the primary and target standby databases.

Step 6   Enable additional fast-start failover conditions (optional)

By default, a fast-start failover is done when both the observer and the standby cannot reach the primary after the configured time threshold (FastStartFailoverThreshold) has passed.

You can optionally indicate the database health conditions that should cause fast-start failover to occur. These conditions are described in the following table:

Health Condition	Description	Enabled by Default
Datafile Offline	A datafile is offline because of a write error.	Yes
Corrupted Dictionary	Dictionary corruption of a critical database object. Currently, this state can be detected only when the database is open	Yes
Corrupted Controlfile	Controlfile is permanently damaged because of a disk failure.	Yes
Inaccessible Logfile	LGWR is unable to write to any member of the log group because on an I/O error	No
Stuck Archiver	Archiver is unable to archive a redo log because the device is full or unavailable.	No

In Oracle RAC configurations, the Inaccessible Logfile and Stuck Archiver health conditions may only be applicable to a single instance. Careful consideration should be given before enabling fast-start failover for either of these conditions because doing so will supersede availability options provided by Oracle Clusterware.

You can also specify errors raised by the Oracle server (ORA errors) as conditions for which fast-start failover should occur.

You can specify particular conditions for which a fast-start failover should occur using either Oracle Enterprise Manager or the DGMGRL ENABLE FAST_START FAILOVER CONDITION and DISABLE FAST_START FAILOVER CONDITION commands.

Step 7   Enable fast-start failover.

Use the Enterprise Manager Fast-Start Failover wizard or the DGMGRL ENABLE FAST_START FAILOVER command to enable fast-start failover. To enable fast-start failover, both the primary and target standby databases must be running and have connectivity, and satisfy all of the prerequisite conditions listed in Section 5.5.1.

Enable Fast-Start Failover Using Enterprise Manager

To enable fast-start failover in Enterprise Manager, use the Fast-Start Failover wizard. On the Data Guard Overview Page next to the "Fast-Start Failover" status field, click Disabled to invoke the Fast-Start Failover Page. Then, on the Fast-Start Failover Change Mode Page, click Enabled. Enterprise Manager will start the observer. Then, on the Fast-Start Failover Configure Page, select the standby database that should be the target of a failover. See Section 5.2, "Choosing a Target Standby Database" for helpful advice. This page will not allow you to alter the protection mode. Rather, fast-start failover will be enabled in accordance with the current protection mode. If the currently configured mode is maximum protection, Enterprise Manager will downgrade the mode to maximum availability.

Enable Fast-Start Failover Using DGMGRL

To enable fast-start failover with DGMGRL, issue the ENABLE FAST_START FAILOVER command while connected to any database in the broker configuration, including on the observer computer. For example:

DGMGRL> ENABLE FAST_START FAILOVER;
Enabled.

See Also:

• ENABLE FAST_START FAILOVER

• Section 6.6, "Scenario 5: Enabling Fast-Start Failover and Starting the Observer"

Note:

Administration at the target standby site should be as comprehensive as that at the primary site because the standby database may assume the primary role without prior notice. Staff support, hardware and software, security (both software and site), network connections, and bandwidth should be equivalent at both sites.

Step 8   Start the Observer.

The primary database must be running in order to start the observer.

You can start the observer before or after you enable fast-start failover. If fast-start failover is already enabled, the observer immediately begins monitoring the status and connections to the primary and target standby databases. If fast-start failover is not already enabled, the observer waits until fast-start failover gets enabled and then begins monitoring.

Starting the Observer Using Enterprise Manager

If the Enterprise Manager agent is installed on the observer computer, it automatically starts the observer when you enable fast-start failover through Enterprise Manager. If the agent is not present, you must start the observer manually using the following instructions for the DGMGRL command-line interface.

Starting the Observer Using DGMGRL

To start the observer with DGMGRL, issue the following command on the observer computer:

DGMGRL> START OBSERVER;

The observer is a continuously executing process that is created when the START OBSERVER command is issued. Thus, the command-line prompt on the observer computer does not return until you issue the STOP OBSERVER command from another DGMGRL session. To issue commands and interact with the broker configuration, you must connect through another DGMGRL client session.

See the START OBSERVER command for more information.

Step 9   Verify the fast-start failover environment.

To verify the readiness of the fast-start failover configuration, issue the DGMGRL SHOW CONFIGURATION VERBOSE command or the SHOW FAST_START FAILOVER command on the primary database. For example:

DGMGRL> SHOW FAST_START FAILOVER;
 
Fast-Start Failover: ENABLED
 Threshold:           60 seconds
 Target:              South_Sales
 Observer:            observer.example.com
 Lag Limit:           30 seconds (not in use)
 Shutdown Primary:    TRUE
 Auto-reinstate:      TRUE
 
Configurable Failover Conditions
 Health Conditions:
   Corrupted Controlfile          YES
   Corrupted Dictionary           YES
   Inaccessible Logfile            NO
   Stuck Archiver                  NO
   Datafile Offline               YES
 
 Oracle Error Conditions:
   (none)

The following sections provide more information about the fast-start failover environment:

• When Fast-Start Failover Is Enabled and the Observer Is Running

• Restrictions When Fast-Start Failover is Enabled

• Shutting Down the Primary Database When Fast-Start Failover Is Enabled

• Performing Manual Role Changes When Fast-Start Failover Is Enabled

5.5.3 Directing a Fast-Start Failover From an Application

You can use the DBMS_DG PL/SQL package to allow an application to direct a fast-start failover when it encounters specific conditions. When a serious condition uniquely known to the application is detected, it may call the DBMS_DG.INITIATE_FS_FAILOVER function, thus alerting either the primary or fast-start failover target standby database that it wants a fast-start failover to occur immediately. The database on which the procedure was called notifies the observer of this. The observer immediately initiates a fast-start failover, as long as the standby database is in a valid fast-start failover state ("observed" and either "synchronized" or "within lag") to accept a failover. Once the observer has initiated a fast-start failover, the primary database shuts down automatically. The observer does not attempt to reinstate the former primary database.

If the configuration is not failable, the DBMS_DG.INITIATE_FS_FAILOVER function returns an ORA error number (it does not signal an exception) informing the caller that a fast-start failover could not be performed.

5.5.4 Viewing Fast-Start Failover Configuration Statistics and Status

To verify the observer is started and the configuration is ready for fast-start failover, you can issue the DGMGRL SHOW DATABASE <target-standby-database> command or query the V$DATABASE view on the target standby database.

You can also query the V$FS_FAILOVER_STATS view to display statistics about fast-start failover occurring on the system.

The rest of this section provides examples of using DGMGRL SHOW commands to display fast-start failover information and includes sections describing the following views:

• V$DATABASE View

• V$FS_FAILOVER_STATS View

Example 1   SHOW FAST-START FAILOVER

The DGMGRL SHOW FAST-START FAILOVER command displays all the fast-start failover related information. For example:

DGMGRL> SHOW FAST_START FAILOVER;
 
Fast-Start Failover: ENABLED
 Threshold:           60 seconds
 Target:              South_Sales
 Observer:            observer.example.com
 Lag Limit:           30 seconds (not in use)
 Shutdown Primary:    TRUE
 Auto-reinstate:      TRUE
 
Configurable Failover Conditions
 Health Conditions:
   Corrupted Controlfile          YES
   Corrupted Dictionary           YES
   Inaccessible Logfile            NO
   Stuck Archiver                  NO
   Datafile Offline               YES
 
 Oracle Error Conditions:
   (none)

Example 2   SHOW CONFIGURATION VERBOSE

The following example shows the fast-start failover information for the DRSolution configuration:

DGMGRL> SHOW CONFIGURATION VERBOSE;
 
Configuration - DRSolution
 
  Protection Mode: MaxAvailability
  Databases:
    North_Sales  - Primary database
    South_Sales  - (*) Physical standby database
 
  (*) Fast-Start Failover target
 
  Properties:
    FastStartFailoverThreshold      = '60'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
 
Fast-Start Failover: ENABLED
 
  Threshold:        180 seconds
  Target:           South_Sales
  Observer:         observer.example.com
  Lag Limit:        30 seconds (not in use)
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE
 
Configuration Status:
SUCCESS

SQL> SELECT LAST_FAILOVER_TIME, LAST_FAILOVER_REASON FROM V$FS_FAILOVER_STATS;
 
LAST_FAILOVER_TIME
--------------------
LAST_FAILOVER_REASON
------------------------------------------------------------------------------------------------------------------------------------
02/13/2007 16:53:10
Primary Disconnected

5.5.5 Disabling Fast-Start Failover

Disabling fast-start failover prevents the observer from initiating a failover to the target standby database. In this case, manual failover may still be possible. See Section 5.4 for information about manual failover.

To disable fast-start failover, use the Fast-Start Failover wizard in Enterprise Manager or the DGMGRL DISABLE FAST_START FAILOVER [FORCE] command. The FORCE option disables fast-start failover on the database to which you are connected even when errors occur. Whether or not you need the FORCE option depends mostly on if the primary and target standby database have network connectivity:

• If the primary and target standby database have network connectivity, and the database to which you are connected has network connectivity with the primary database, the FORCE option has no effect. Simply use DISABLE FAST_START FAILOVER. This method will disable fast-start failover on all databases in the broker configuration.

  If errors occur during the disable operation, the broker returns an error message and stops the disable operation.

• If the primary and target standby databases do not have network connectivity or if the database to which you are connected does not have network connectivity with the primary database, consider using DISABLE FAST_START FAILOVER with the FORCE option.

  The broker may not be able to disable fast-start failover on all databases in the broker configuration when you issue the DISABLE FAST_START FAILOVER FORCE command. As a result, there is no guarantee that the observer will not perform a fast-start failover to the target standby database if the observer determines that conditions warrant a failover. The following list indicates the extent to which fast-start failover is disabled in the broker configuration when the DISABLE FAST_START FAILOVER FORCE command is issued on the primary database, target standby database, and a standby database that is not the fast-start failover target.

  If you issue this command on:

  • The target standby database when it does not have connectivity with the primary database, fast-start failover is disabled only on the target standby database. In this case, the observer cannot perform a fast-start failover even if conditions warrant a failover. Disabling fast-start failover with the FORCE option when connected to the target standby database guarantees that fast-start failover will not occur.

    When the primary database and the target standby database regain network connectivity, the broker will disable fast-start failover for the entire broker configuration.

  • The primary database, it attempts to disable fast-start failover on as many databases in the configuration with which it has a network connection. If the primary database does not have connectivity with the target standby database, fast-start failover remains enabled on the target standby database and the observer may still attempt a fast-start failover if conditions warrant a failover.

    Caution:

    This action may result in two databases in the configuration simultaneously assuming the primary database role should fast-start failover occur. For this reason, you should first issue this command on the target standby database.

  • Another standby database that does not have connectivity with the primary database, fast-start failover is disabled for this database. Because fast-start failover was not disabled on the target standby database, the observer may still attempt a fast-start failover to the target standby database should conditions warrant a failover.

    When the primary database and the (non-target) standby database regain network connectivity, the broker will propagate its current fast-start failover setting (ENABLED or DISABLED) to the non-target standby.

    Caution:

    When you are experiencing network disconnections and you issue the DISABLE FAST_START FAILOVER FORCE command on the primary database or a standby database that does not have connectivity with the primary database, fast-start failover may not be disabled for all databases in the broker configuration. As a result the observer may still initiate fast-start failover to the target standby database, if conditions warrant a failover. This may result in two databases in the configuration simultaneously assuming the primary database role.

Conditions Requiring the FORCE Option

Disabling fast-start failover without the FORCE option can succeed only if the database on which the command is issued has a network connection with the primary database and if the primary database and target standby database have a network connection. This is the recommended method for disabling fast-start failover.

However, there may be situations in which you must disable fast-start failover when the primary database and the target standby database do not have a network connection, or the database on which you issued the disable fast-start failover command does not have a network connection to the primary database. In cases where there is a lost network connection, be aware that the observer may attempt a fast-start failover to the target standby database if conditions warrant a failover.The FORCE option may be the preferred method for disabling fast-start failover when:

• A network outage isolates the primary database from the observer and the target standby database before conditions exist that warrant a failover.

  In this case, the primary database stalls and prevents any further transactions from committing because a fast-start failover may have occurred while it was isolated. If you expect the network to be disconnected for a long time and you need to make the primary database available, first confirm that a fast-start failover has not occurred to the target standby database. Then, disable fast-start failover with the FORCE option on the primary database.If possible, confirm that fast-start failover has not occurred to the target standby database prior to disabling fast-start failover with the FORCE option on the primary database.

  Caution:

  This action may result in two databases in the configuration simultaneously assuming the primary database role. This can be avoided by first disabling fast-start failover with the FORCE option on the target standby.

• You want to conduct a manual failover to any standby database in the configuration (for example, because a failure occurred on the primary database at a time when the primary and target standby database were not ready to failover).

  In this case fast-start failover cannot occur because the databases are not ready to failover. You cannot perform a manual failover to the target standby database for the same reason. To proceed, you must first disable fast-start failover using the FORCE option, and then perform a manual failover.

  Caution:

  This action will result in loss of data and the possibility of two databases in the configuration simultaneously assuming the primary database role. This can be avoided by first disabling fast-start failover with the FORCE option on the target standby.

• A fast-start failover to the target standby database fails.

  If the failover fails for any reason, it could leave the target standby database inoperable, regardless of whether the target standby database is ready to failover. If there is another standby database that is available for failover, you can perform a manual failover to that standby database after you first disable fast-start failover using the FORCE option on that standby database.

• You want to prevent fast-start failover from occurring because the primary database will resume service soon.

  In this case, disable fast-start failover using the FORCE option on the target standby database. Once the primary database regains connectivity with the target standby database, fast-start failover will be disabled for all the databases in the configuration.

Disabling Fast-Start Failover Using Enterprise Manager

Click Disable in the Fast-Start Failover wizard. Then, click Continue to proceed to the next page. See the Enterprise Manager online Help system for more information.

Disabling Fast-Start Failover Using DGMGRL

Issue the DISABLE FAST_START FAILOVER command or the DISABLE FAST_START FAILOVER FORCE command. See the "DISABLE FAST_START FAILOVER" command in Chapter 7 for more information.

5.5.6 Performance Considerations for Fast-Start Failover

Consider the following recommendations to obtain better performance when using fast-start failover:

• The failover time is dependent upon whether the target standby database (physical or logical standby database) has applied all of the redo data it has received from the primary database.

• Enabling fast-start failover in a configuration operating in maximum performance mode provides better overall performance on the primary database because redo data is sent asynchronously to the target standby database. Note that this does not guarantee no data will be lost.

• Fast-start failover is faster when you take steps to optimize recovery so that the application of redo data to the standby database is kept up to date with the primary database's rate of redo application. To optimize the log apply rate:

  • Do not configure the DelayMins database property to delay applying archived redo log files to the standby database (see Section 4.5 for more information).

  • See Oracle Data Guard Concepts and Administration for information about tuning the log apply rate for a physical standby database.

  • See the Oracle Maximum Availability Architecture white papers at:

    http://www.oracle.com/goto/maa

• When setting the FastStartFailoverLagLimit configuration property, consider these tradeoffs between performance and potential data-loss:

  • A low lag limit will minimize data loss but may impact the performance of the primary database.

  • A high lag limit may lead to more data loss but may lessen the performance impact of the primary database.

5.5.7 Managing the Observer

The observer is integrated in the DGMGRL client-side component of the broker and typically runs on a different computer from the primary or standby databases and from the computer where you manage the broker configuration. The observer continuously monitors the fast-start failover environment to ensure the primary database is available (described in Section 5.5.2.1). The observer's main purpose is to enhance high availability and lights out computing by reducing the human intervention required by the manual failover process that can add minutes or hours to downtime.

You can manage the observer through either the Data Guard Overview pages in Oracle Enterprise Manager or using DGMGRL commands. Figure 5-2 shows the observer monitoring a fast-start failover configuration.

Figure 5-2 The Observer in the Fast-Start Failover Environment

Description of ''Figure 5-2 The Observer in the Fast-Start Failover Environment''

The following sections provide information about managing the observer:

• Installing and Starting the Observer

• Viewing Information About the Observer

• What Happens if the Observer Fails?

• Stopping the Observer

• Moving the Observer to Another Computer

• How the Observer Maintains Fast-Start Failover Configuration Information

ORA-16647: could not start more than one observer

ORA-16658: unobserved fast-start failover configuration
ORA-16820: fast-start failover observer is no longer observing this database

% dgmgrl -logfile $ORACLE_HOME/rdbms/log/config1.log
DGMGRL> CONNECT /@primary1;
DGMGRL> START OBSERVER FILE=$ORACLE_HOME/dbs/config1.dat;
 
% dgmgrl -logfile $ORACLE_HOME/rdbms/log/config2.log
DGMGRL> CONNECT /@primary2;
DGMGRL> START OBSERVER FILE=$ORACLE_HOME/dbs/config2.dat;

5.5.8 Reinstating the Former Primary Database in the Broker Configuration

If a fast-start failover was initiated because the primary database had crashed or lost connectivity with the observer and target standby database, the observer automatically attempts to reinstate the former primary database as a standby database, if the FastStartFailoverAutoReinstate configuration property is set to TRUE. Reinstatement restores high availability to the broker configuration so that, in the event of a failure of the new primary database, another fast-start failover can occur. The reinstated database acts as the fast-start failover target for the new primary database, making a subsequent fast-start failover possible. The new standby database is a viable target of a failover when it begins receiving redo data received from the new primary database.

To allow the observer to automatically reinstate the former primary database, the database must be started and mounted, but it cannot be opened. The observer will restart the former primary database to the mounted state if it is open, prior to reinstating the database. The broker reinstates the database as a standby database of the same type as the former standby database of the new primary database.

If the former primary database cannot be reinstated automatically, you can manually reinstate it using either the DGMGRL REINSTATE command or Enterprise Manager. Step-by-step instructions for manual reinstatement are described in Section 5.4.3.

5.5.9 Shutting Down Databases In a Fast-Start Failover Environment

Perform the following steps if you need to shut down the primary or standby databases:

1. Stop the observer and wait for the FS_FAILOVER_OBSERVER_PRESENT column in the V$DATABASE fixed view to contain the value "NO" for both the primary and target standby databases. This ensures that a fast-start failover will not occur while you are shutting down the primary database.

2. Shut down the primary database and the target standby database using either DGMGRL SHUTDOWN command or the SQL*Plus SHUTDOWN statement.

When restarting the databases, you may restart them in any order. When both databases have been restarted, you may restart the observer.

5.6.1 Data Guard Specific FAN and FCF Configuration Requirements

This section describes configuration requirements that must be met in order to publish and properly handle FAN events generated as the result of a broker-managed failover.

These requirements are supplemental to those described in the documents previously referenced and in the following client-specific guides:

• Oracle Database JDBC Developer's Guide

• Oracle Call Interface Programmer's Guide

• Oracle Data Provider for .NET Developer's Guide

srvctl add service –d NORTH –s PAYROLL –l PRIMARY -q TRUE -e SELECT -m BASIC -w 10 -z 150

srvctl add service –d SOUTH –s PAYROLL –l PRIMARY -q TRUE -e SELECT -m BASIC -w 10 -z 150

srvctl add service -d NORTH -s sales -l PHYSICAL_STANDBY -q TRUE -e SELECT -m BASIC -w 10 -z 150
srvctl start service –d NORTH –s sales
srvctl stop service –d NORTH –s sales

srvctl add service -d SOUTH -s sales -l PHYSICAL_STANDBY -q TRUE -e SELECT -m BASIC -w 10 -z 150