3 Automatic Database Performance Monitoring

Automatic Database Diagnostic Monitor (ADDM) automatically detects and reports performance problems with the database. The results are displayed as ADDM findings on the Database Home page in Oracle Enterprise Manager (Enterprise Manager). Reviewing the ADDM findings enables you to quickly identify the performance problems that require your attention.

Each ADDM finding provides a list of recommendations for reducing the impact of the performance problem. You should review ADDM findings and implement the recommendations every day as part of regular database maintenance. Even when the database is operating at an optimal performance level, you should continue to use ADDM to monitor database performance on an ongoing basis.

This chapter contains the following sections:

See Also:

Overview of Automatic Database Diagnostic Monitor

ADDM is self-diagnostic software built into Oracle Database. ADDM examines and analyzes data captured in Automatic Workload Repository (AWR) to determine possible database performance problems. ADDM then locates the root causes of the performance problems, provides recommendations for correcting them, and quantifies the expected benefits. ADDM also identifies areas where no action is necessary.

This section contains the following topics:

ADDM Analysis

An ADDM analysis is performed after each AWR snapshot (every hour by default), and the results are saved in the database. You can then view the results using Enterprise Manager. Before using another performance tuning method described in this guide, review the results of the ADDM analysis first.

The ADDM analysis is performed from the top down, first identifying symptoms and then refining the analysis to reach the root causes of performance problems. ADDM uses the DB time statistic to identify performance problems. DB time is the cumulative time spent by the database in processing user requests, including both the wait time and CPU time of all user sessions that are not idle.

The goal of database performance tuning is to reduce the DB time of the system for a given workload. By reducing DB time, the database can support more user requests by using the same or fewer resources. ADDM reports system resources that are using a significant portion of DB time as problem areas and sorts them in descending order by the amount of related DB time spent. For more information about the DB time statistic, see "Time Model Statistics".

ADDM Recommendations

In addition to diagnosing performance problems, ADDM recommends possible solutions. When appropriate, ADDM recommends multiple solutions from which you can choose. ADDM recommendations include the following:

  • Hardware changes

    Adding CPUs or changing the I/O subsystem configuration

  • Database configuration

    Changing initialization parameter settings

  • Schema changes

    Hash partitioning a table or index, or using automatic segment space management (ASSM)

  • Application changes

    Using the cache option for sequences or using bind variables

  • Using other advisors

    Running SQL Tuning Advisor on high-load SQL statements or running the Segment Advisor on hot objects

ADDM benefits apply beyond production systems. Even on development and test systems, ADDM can provide an early warning of potential performance problems.

Performance tuning is an iterative process. Fixing one problem can cause a bottleneck to shift to another part of the system. Even with the benefit of the ADDM analysis, it can take multiple tuning cycles to reach a desirable level of performance.

See Also:

ADDM for Oracle Real Application Clusters

In an Oracle Real Application Clusters (Oracle RAC) environment, you can use ADDM to analyze the throughput performance of a database cluster. ADDM for Oracle RAC considers DB time as the sum of database times for all database instances and reports findings that are significant at the cluster level. For example, the DB time of each cluster node may be insignificant when considered individually, but the aggregate DB time may be a significant problem for the cluster as a whole.

See Also:

Configuring Automatic Database Diagnostic Monitor

This section contains the following topics:

Setting Initialization Parameters to Enable ADDM

Automatic database diagnostic monitoring is enabled by default and is controlled by the CONTROL_MANAGEMENT_PACK_ACCESS and the STATISTICS_LEVEL initialization parameters.

Set CONTROL_MANAGEMENT_PACK_ACCESS to DIAGNOSTIC+TUNING (default) or DIAGNOSTIC to enable automatic database diagnostic monitoring. Setting CONTROL_MANAGEMENT_PACK_ACCESS to NONE disables many Oracle Database features, including ADDM, and is strongly discouraged.

Set STATISTICS_LEVEL to TYPICAL (default) or ALL to enable automatic database diagnostic monitoring. Setting STATISTICS_LEVEL to BASIC disables many Oracle Database features, including ADDM, and is strongly discouraged.

To determine whether ADDM is enabled: 

  1. From the Database Home page, click Server.

    The Server subpage appears.

  2. In the Database Configuration section, click Initialization Parameters.

    The Initialization Parameters page appears.

  3. In the Name field, enter statistics_level and then click Go.

    The table shows the setting of this initialization parameter.

    Description of statistics_level.gif follows
    Description of the illustration statistics_level.gif

  4. Do one of the following:

    • If the Value list shows ALL or TYPICAL, then do nothing.

    • If the Value list shows BASIC, then select ALL or TYPICAL, and then click Apply.

  5. In the Name field, enter control_management_pack_access, and then click Go.

    The table shows the setting of this initialization parameter.

  6. Do one of the following:

    • If the Value column shows DIAGNOSTIC or DIAGNOSTIC+TUNING, then do nothing.

    • If the Value column shows NONE, then select DIAGNOSTIC or DIAGNOSTIC+TUNING and click Apply.

See Also:

Setting the DBIO_EXPECTED Parameter

ADDM analysis of I/O performance partially depends on a single argument, DBIO_EXPECTED, that describes the expected performance of the I/O subsystem. The value of DBIO_EXPECTED is the average time it takes to read a single database block, in microseconds. Oracle Database uses the default value of 10 milliseconds, which is an appropriate value for most hard drives. You can choose a different value based on the characteristics of your hardware.

To determine the correct setting for the DBIO_EXPECTED initialization parameter: 

  1. Measure the average read time of a single database block for your hardware.

    This measurement must be taken for random I/O, which includes seek time if you use standard hard drives. Typical values for hard drives are between 5000 and 20000 microseconds. See Oracle Database Performance Tuning Guide to learn how to assess the I/O capability of the storage subsystem.

  2. Set the value one time for all subsequent ADDM executions.

    For example, if the measured value is 8000 microseconds, then execute the following PL/SQL code as the SYS user:

    EXECUTE DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER(
                     'ADDM', 'DBIO_EXPECTED', 8000);

Managing AWR Snapshots

By default, the Automatic Workload Repository (AWR) generates snapshots of performance data once every hour, and retains the statistics in the workload repository for 8 days. You can change the default values for both the snapshot interval and the retention period.

Oracle recommends that you adjust the AWR retention period to at least one month. You can also extend the period to one business cycle so you can compare data across time frames such as the close of the fiscal quarter. You can also create AWR baselines to retain snapshots indefinitely for important time periods.

The data in the snapshot interval is analyzed by ADDM. ADDM compares the differences between snapshots to determine which SQL statements to capture, based on the effect on the system load. The ADDM analysis shows the number of SQL statements that need to be captured over time.

This section contains the following topics:

Creating Snapshots

Manually creating snapshots is usually not necessary because AWR generates snapshots of the performance data once every hour by default. In some cases, however, it may be necessary to manually create snapshots to capture different durations of activity, such as when you want to compare performance data over a shorter period than the snapshot interval.

To create snapshots: 

  1. From the Database Home page, click Performance.

    The Performance page appears.

  2. Under Additional Monitoring Links, click Snapshots.

    The Snapshots page appears with a list of the most recent snapshots.

  3. Click Create.

    The Confirmation page appears.

  4. Click Yes.

    The Processing: Create Snapshot page is displayed while the snapshot is being taken.

    After the snapshot is taken, the Snapshots page reappears with a Confirmation message.

    In the following example, the ID of the snapshot that was created is 96.

    Description of snapshots.gif follows
    Description of the illustration snapshots.gif

Modifying Snapshot Settings

By default, AWR generates snapshots of performance data once every hour. You can modify the default values of both the interval between snapshots and their retention period.

To modify the snapshot settings: 

  1. From the Database Home page, click Server.

    The Server subpage appears.

  2. In the Statistics Management section, click Automatic Workload Repository.

    The Automatic Workload Repository page appears.

    Description of snapshot_settings.gif follows
    Description of the illustration snapshot_settings.gif

    In this example, snapshot retention is set to 8 days and snapshot interval is set to 60 minutes.

  3. Click Edit.

    The Edit Settings page appears.

    Description of snap_edit_set.gif follows
    Description of the illustration snap_edit_set.gif

  4. For Snapshot Retention, do one of the following:

    • Select Use Time-Based Retention Period (Days), and in the associated field enter the number of days to retain the snapshots.

    • Select Retain Forever to retain snapshots indefinitely.

    It is recommended that you increase the snapshot retention period to the maximum allowed by the available disk space.

    In this example, the snapshot retention period is changed to 30 days.

    Description of snapshot_retent.gif follows
    Description of the illustration snapshot_retent.gif

  5. For Snapshot Collection, do one of the following:

    • Select System Snapshot Interval, and in the Interval list, select the desired interval to change the interval between snapshots.

    • Select Turn off Snapshot Collection to disable snapshot collection.

    In this example, the snapshot collection interval is changed to 30 minutes.

    Description of snapshot_collect.gif follows
    Description of the illustration snapshot_collect.gif

  6. Click the link next to Collection Level.

    The Initialization Parameter page appears.

    To change the statistics level, select the desired value in the Value list for the statistics_level parameter. Click Save to File to set the value in the server parameter file.

    In this example, the default value of Typical is used.

    Description of snapshot_stat_lvl.gif follows
    Description of the illustration snapshot_stat_lvl.gif

  7. Click OK to apply the changes.

    The Automatic Workload Repository page appears and displays the new settings.

    Description of snapshot_sett_mod.gif follows
    Description of the illustration snapshot_sett_mod.gif

Reviewing the Automatic Database Diagnostic Monitor Analysis

By default, ADDM runs every hour to analyze snapshots taken by AWR during that period. If the database finds performance problems, then it displays the results of the analysis under Diagnostic Summary on the Database Home page.

Description of diagnostic_sum.gif follows
Description of the illustration diagnostic_sum.gif

The ADDM Findings link shows how many ADDM findings were found in the most recent ADDM analysis.

To view ADDM findings: 

  1. On the Database Home page, under Diagnostic Summary, click the link next to ADDM Findings.

    The Automatic Database Diagnostic Monitor (ADDM) page appears. The results of the ADDM run are displayed.

    Description of addm_page.gif follows
    Description of the illustration addm_page.gif

    On the Automatic Database Diagnostic Monitor (ADDM) page, the Database Activity chart shows the database activity during the ADDM analysis period. Database activity types are defined in the legend based on their corresponding colors in the chart. Each icon below the chart represents a different ADDM task, which in turn corresponds to a pair of snapshots saved in AWR.

    In this example, the two largest blocks of activity were 2:30 p.m. to 5:30 p.m. on January 20 and 9 a.m. to 11 a.m. the next day. The thick CPU and thin Wait bars in the hour after 4:30 p.m. indicate that CPU may have been a bottleneck during this period. In other areas of the chart, the Wait bar is thicker than the CPU bar, indicating that wait events had a greater performance impact than CPU.

    In the ADDM Performance Analysis section, ADDM findings are listed in descending order, from highest to least impact. The Informational Findings section lists areas that have no performance impact and are for information only.

    Description of information_findings.gif follows
    Description of the illustration information_findings.gif

  2. Optionally, click the Zoom icons to shorten or lengthen the analysis period displayed on the chart.

  3. To view the ADDM findings in a report, click View Report.

    The View Report page appears.

    You can click Save to File to save the report for later access.

Interpretation of Automatic Database Diagnostic Monitor Findings

The ADDM analysis results are represented as a set of findings. Each ADDM finding belongs to one of three types:

  • Problem

    Findings that describe the root cause of a database performance issue

  • Symptom

    Findings that contain information that often leads to one or more problem findings

  • Information

    Findings that are used to report areas of the system that do not have a performance impact

Each problem finding is quantified with an estimate of the portion of DB time that resulted from the performance problem.

When a specific problem has multiple causes, ADDM may report multiple findings. In this case, the impacts of these multiple findings can contain the same portion of DB time. Because performance problems can overlap, summing the impacts of the reported findings can yield a number higher than 100% of DB time. For example, if a system performs many read I/O operations, ADDM may report a SQL statement responsible for 50% of DB time due to I/O activity as one finding, and an undersized buffer cache responsible for 75% of DB time as another finding.

A problem finding can be associated with a list of recommendations for reducing the impact of a performance problem. Each recommendation has a benefit that is an estimate of the portion of DB time that can be saved if the recommendation is implemented. When multiple recommendations are associated with an ADDM finding, the recommendations may contain alternatives for solving the same problem. In this case, the sum of the benefits may be higher than the impact of the finding. You do not need to apply all the recommendations to solve the same problem.

Recommendations are composed of actions and rationales. You must apply all the actions of a recommendation to gain its estimated benefit. The rationales explain why the set of actions was recommended, and provide additional information for implementing them. An ADDM action may present multiple solutions. If this is the case, then choose the easiest solution to implement.

Implementing Automatic Database Diagnostic Monitor Recommendations

This section describes how to implement ADDM recommendations. ADDM findings are displayed in the Automatic Database Diagnostic Monitor (ADDM) page under ADDM Performance Analysis.

Description of perf_analysis.gif follows
Description of the illustration perf_analysis.gif

To implement ADDM recommendations: 

  1. On the Database Home page, under Diagnostic Summary, click the link next to ADDM Findings.

    The Automatic Database Diagnostic Monitor (ADDM) page appears.

  2. In the Database Activity section, click the icon for the ADDM to investigate.

    The data in the ADDM Performance Analysis section changes based on the ADDM run that you selected.

  3. In the ADDM Performance Analysis section, click the ADDM finding that has the greatest impact.

    In this example, the finding with the greatest impact is Top SQL Statements.

    The Performance Finding Details page appears.

    In the following example, three recommendations are shown. The first is estimated to have a maximum benefit of up to 39.5% of DB time in the analysis period. The second recommendation is estimated to have a maximum benefit of up to 25.6% of DB time, while the third has a maximum of 18.6%.

    Description of perf_finding_details.gif follows
    Description of the illustration perf_finding_details.gif

  4. Under Recommendations, click Show to review the recommendations and required actions for each recommendation.

    The Category column displays the category of the recommendation. The Benefit (%) column displays the estimated benefit of implementing the recommendation.

    Description of addm_rec.gif follows
    Description of the illustration addm_rec.gif

  5. If additional information is available about why the set of actions was recommended, then click Additional Information, or review the content displayed under Additional Information.

    For example, the Undersized Buffer Cache finding contains additional information to indicate the recommended value of the DB_CACHE_SIZE initialization parameter.

    Description of addm_addit_info.gif follows
    Description of the illustration addm_addit_info.gif

  6. To view the history of a finding, click Finding History.

    The Finding History page appears.

    Description of finding_history.gif follows
    Description of the illustration finding_history.gif

    The Finding History page shows how often a particular finding has occurred in a selected 3-hour interval. You can use this information to determine whether the finding was a transient or a persistent problem in the system. Based on this information, you can determine whether the actions associated with the finding should be implemented.

    The Active Sessions chart shows the impact of the finding and of the other loads on the system. You can change the display as follows:

    1. To move the 3-hour interval, click and drag the shaded box in the Active Sessions chart.

    2. To change dates, enter the desired date in the View field, and then click Go.

    3. To view details about a finding, under Detail for Selected 3 Hour Interval, click the link in the Finding Details column to display the Performance Finding Details page for the corresponding ADDM finding.

  7. Optionally, create a filter to suppress known findings that have been tuned or cannot be tuned further. To create filters for a selected ADDM finding:

    1. Click Filters.

      The Filters for Finding page appears.

    2. Click Create.

      The Create Filter for Finding page appears.

    3. In the Name field, enter a name for the ADDM filter.

    4. In the Active Sessions field, specify the filter criteria in terms of the number of active sessions.

      The database filters the ADDM finding for future ADDM runs if the number of active sessions for the finding is less than the specified filter criteria.

    5. In the % Active Sessions field, specify the filter criteria in terms of percentage of active sessions.

      The database filters the ADDM finding for future ADDM runs if the number of active sessions for the finding is less than the specified filter criteria.

    6. Click OK.

  8. Perform the required action of a chosen recommendation.

    Depending on the type of action you choose to perform, various options may be available, such as Implement or Run Advisor Now. These options enable you to implement the recommendation immediately with a single mouse click.

    In the example shown in Step 4, the simplest solution is to click Run Advisor Now to immediately run a SQL Tuning Advisor task on the SQL statement.

    See Also:

Viewing Snapshot Statistics

You can view the data contained in snapshots taken by AWR using Enterprise Manager. Typically, it is not necessary to review snapshot data because it primarily contains raw statistics. Instead, rely on ADDM, which analyzes statistics to identify performance problems. Snapshot statistics are intended primarily for advanced users, DBAs accustomed to using Statspack for performance analysis.

To view snapshot statistics: 

  1. From the Database Home page, click Performance.

    The Performance page appears.

  2. Under Additional Monitoring Links, click Snapshots.

    The Snapshots page appears with a list of the most recent snapshots.

  3. To view the statistics gathered in a snapshot, click the ID link of the snapshot you want to view.

    The Snapshot Details appears, showing the Details subpage.

    Description of snapshot_det.gif follows
    Description of the illustration snapshot_det.gif

    In this example, statistics gathered from the previous snapshot (snapshot 100) to the selected snapshot (snapshot 101) are displayed.

  4. To view a Workload Repository report of the statistics, click Report.

    The Workload Repository report appears.

  5. Optionally, click Save to File to save the report for later access.

See Also: