The Oracle Database File System Hierarchical Store is implemented in the DBMS_DBFS_HS package. This package provides users the ability to use tape or Amazon S3 Web service as a storage tier when doing Information Lifecycle Management for their database tables.
This chapter contains the following topics:
Overview
Security Model
Constants
Operational Notes
The DBMS_DBFS_HS package is a service provider underneath the DBMS_DBFS_CONTENT package that enables use of tape or Amazon S3 Web service as storage for data.
The data on tape or Amazon S3 Web service is part of the Oracle Database and can be accessed through all standard interfaces, but only through the database. The package allows users to use tape or Amazon S3 Web service as a storage tier when doing Information Lifecycle Management of their content.
The package initially stores all content files in level-1 cache. As the level-1 cache fills up, content files are moved to level-2 cache and then to an external storage device using bulk writes.
The DBMS_DBFS_HS package uses the constants shown in following tables:
DBMS_DBFS_HS Constants - Used by the SETSTOREPROPERTY Procedure and the GETSTOREPROPERTY Function
DBMS_DBFS_HS Constants - Used by the REGISTERSTORECOMMAND Function
Table 49-1 DBMS_DBFS_HS Constants - Used by the CREATESTORE Procedure
| Constant | Type | Value | Description |
|---|---|---|---|
|
|
|
|
Use tape as a storage tier |
|
|
|
|
Use Amazon S3 Web service as a storage tier |
Table 49-2 DBMS_DBFS_HS Constants - Used by the SETSTOREPROPERTY Procedure and the GETSTOREPROPERTY Function
| Constant | Type | Value | Description |
|---|---|---|---|
|
|
|
|
Specifies the AWS bucket to be used as a storage tier by the Hierarchical Store. Restrictions on bucket name are: 1) Bucket names can only contain lowercase letters, numbers, periods ( 2) Bucket names must start with a number or letter. 3) Bucket names cannot be in an IP address style ( 4) Bucket names must be between 3 and 63 characters long. 5) Bucket names should not end with a dash. 6) Dashes cannot appear next to periods. For example, |
|
|
|
|
Specifies the cumulative cache size used for the Hierarchical Store. This property is set by the CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. It cannot be modified by the SETSTOREPROPERTY Procedure, though its value can be queried by the GETSTOREPROPERTY Function. |
|
|
|
|
Use to enable compression of files stored in the DBFS hierarchical store. It specifies the compression level to be used for compressing the files |
|
|
|
|
If this property is set to ' |
|
|
|
|
Specifies the DNS name of the HTTP proxy, if any, that is needed to access the Amazon S3 storage service |
|
|
|
|
Specifies the license ID associated with the library |
|
|
|
|
Specifies fraction of the This property cannot be modified by the SETSTOREPROPERTY Procedurethough its value can be queried by the GETSTOREPROPERTY Function. Its value is set by CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. |
|
|
|
|
Specifies the media pool number to use for storing the content |
|
|
|
|
Indicates no compression |
|
|
|
|
Use to set the compression level to |
|
|
|
|
Use to set the compression level to |
|
|
|
|
Use to set the compression level to |
|
|
|
|
Specifies Multiple content files are bundled together into one archive file and then the archive file is transferred to tape or Amazon S3. This is because creating one file on tape or Amazon S3 for every content file in the store is a prohibitively expensive operation. This property cannot be modified by the SETSTOREPROPERTY Procedurethough its value can be queried by the GETSTOREPROPERTY Function. Its value is set by CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. |
|
|
|
|
Specifies the size used by the SBT protocol to transfer data from tape or S3.This chunk is allocated in memory per transaction for retrieval of content files from an archive store, so the value of this property should be conservative. The default size of 1MB is typically good for most users. |
|
|
|
|
Specifies the |
|
|
|
|
Specifies the path of the shared library used by RMAN to communicate with Amazon S3. It is named |
|
|
|
|
Indicates whether buffer-based |
|
|
|
|
The value of this property should be of the form:
Defines the Oracle Wallet which contains the credentials of the Amazon S3 account associated with the store under consideration.
The format of
In UNIX or Linux it is, for example:
When the package is executed in the Oracle database server, the wallet is accessed from the database server.
|
|
|
|
|
Specifies the size used by the SBT protocol to transfer data to tape or S3. This chunk is allocated in memory per transaction for The default size of 1MB is typically good for most users. |
Table 49-3 DBMS_DBFS_HS Constants - Used by the REGISTERSTORECOMMAND Function
| Constant | Type | Value | Description |
|---|---|---|---|
|
|
|
|
Specified operation must be performed before writing a |
|
|
|
|
Specified operation must be performed before a retrieval operation such as reading a |
Table 49-5 DBMS_DBFS_HS Package Subprograms
| Subprogram | Description |
|---|---|
|
Removes files created on the external storage device that hold no currently used data |
|
|
Creates an AWS bucket, associated with a store of type |
|
|
Creates a new hierarchical store |
|
|
Removes a command that had been previously associated with a store through the RECONFIGCACHE Procedure |
|
|
Deletes a previously created hierarchical store |
|
|
Flushes (writes out) dirty contents from the level-1 cache. |
|
|
Retrieves the values of a property of a store |
|
|
Reconfigures the parameters of the database cache used by the store |
|
|
Registers commands for a store with the Hierarchical Store to be sent to the Media Manager for the external storage device associated with the store |
|
|
Sends a command to be executed on the external storage device's Media Manager |
|
|
Stores properties of a store in the database |
|
|
Pushes locally staged data to the remote storage |
This procedure removes files created on the external storage device that hold no currently used data in them.
The action of removing files from external storage device can not be rolled back.
This method can be executed periodically to clear space on the external storage device. Asynchronously deleting content from the external storage device is useful because it has minimal impact on the OLTP performance. The periodic scheduling can be accomplished using the DBMS_SCHEDULER package.
This procedure creates an AWS bucket, associated with a store of type STORETYPE_AMAZONS3 into which the Hierarchical Store can then move data.
The PROPNAME_BUCKET property of the store should be set before this subprogram is called.
Once this procedure has successfully created a bucket in Amazon S3, the bucket can only be deleted using out-of-band methods, such as logging-in to S3 and deleting data (directories, files, and other items) for the bucket.
This procedure creates a new hierarchical store store_name of type STORE_TYPE (STORETYPE_TAPE or STORETYPE_AMAZONS3) in schema schema_name (defaulting to current schema) under the ownership of the invoking session user.
DBMS_DBFS_HS.CREATESTORE ( store_name IN VARCHAR2, store_type IN VARCHAR2, tbl_name IN VARCHAR2, tbs_name IN VARCHAR2, cache_size IN NUMBER, lob_cache_quota IN NUMBER DEFAULT NULL, optimal_tarball_size IN NUMBER DEFAULT NULL, schema_name IN VARCHAR2 DEFAULT NULL);
Table 49-8 CREATESTORE Procedure Parameters
| Parameter | Description |
|---|---|
|
|
Name of store |
|
|
|
|
|
Table for store entries |
|
|
Tablespace for the store |
|
|
Amount of space used by the store to cache content in given tablespace |
|
|
Fraction of the cache_size which is allocated for level 1 cache. The default value of this parameter is |
|
|
Maximum possible size of the archive file. Multiple content files are bundled together into one archive file, and then the archive file is transferred to tape or Amazon S3. This is because creating one file on tape or Amazon S3 for every content file in the store is a prohibitively expensive operation. The value of is set by default to 10GB for tape and to 100MB for Amazon S3. |
|
|
Schema for the store |
Store names must be unique for an owner. The same store names can be used for different stores owned by different owners.
Once a table space has been specified to store the store's content in a database, it cannot be changed later.
This subprogram will execute like a DDL statement, performing an automatic COMMIT before and after execution.
Stores using DBMS_DBFS_HS must not use singleton mount. This means that the singleton parameter should be FALSE and the store_mount parameter should have a non-NULL value in a call to the DBMS_DBFS_CONTENT.MOUNTSTORE Procedure.
This procedure removes a command that had been previously associated with a store through the RECONFIGCACHE Procedure.
If this subprogram successfully executes, its actions cannot be rolled back by the user. If the user wants to restore the previous state, the user must call the REGISTERSTORECOMMAND Procedure.
This procedure deletes a previously created hierarchical store specified by name and owned by the invoking session user.
Table 49-10 DROPSTORE Procedure Parameters
| Parameter | Description |
|---|---|
|
|
Name of store owned by the invoking session user |
|
|
User can specify optional flags. If |
The procedure executes like a DDL in that it auto-commits before and after its execution.
If CLEANUPBACKUPFILES is disabled during the procedure, the user must resort to out-of-band techniques to cleanup unused backup files. No further invocations of CLEANUPBACKFILES for a dropped store are possible through hierarchical store.
This subprogram will un-register the store from DBMS_DBFS_CONTENT package. All files in the given store are deleted from the store (Tape or Amazon S3 Web service). The database table holding the store's entries in the database, is also dropped by this subprogram.
This procedure flushes out dirty contents from level-1 cache, which can be locked, to level-2 cache, thereby freeing-up space in level 1 cache.
This function retrieves the values of a property.
DBMS_DBFS_HS.GETSTOREPROPERTY ( store_name IN VARCHAR2, property_name IN VARCHAR2, noexcp IN BOOLEAN DEFAULT FALSE) RETURN VARCHAR2;
This procedure reconfigures the parameters of the database cache being used by the store.
DBMS_DBFS_HS.RECONFIGCACHE ( store_name IN VARCHAR2, cache_size IN NUMBER DEFAULT NULL, lobcache_quota IN NUMBER DEFAULT NULL, optimal_tarball_size IN NUMBER DEFAULT NULL);
Table 49-13 RECONFIGCACHE Procedure Parameters
| Parameter | Description |
|---|---|
|
|
Name of store |
|
|
Cumulative cache size used for the Hierarchical Store |
|
|
Fraction of the cache size that are assigned to level 1 cache |
|
|
Maximum possible size of an archive file. Since creating one file for every content file in the store is a prohibitively expensive operation, multiple content files are bundled together into one archive file for transfer to tape or Amazon S3. |
The specified store must already have been created before reconfiguration.
The Hierarchical Store uses a level 1 cache and a level 2 cache. The level 1 cache subsumes most of the working set and the level 2 cache is used to perform bulk writes to the backend device.
If any of the last 3 parameters is NULL, its value specified during store creation is used. If the parameter was NULL when the call to the CREATESTORE Procedure was issued, the DBMS_DBFS_HS package assigns a default value.
The DBMS_DBFS_HS package optimistically tries to allocate more than 1 tarball's worth of size for level 2 cache to facilitate concurrency, though a minimum of 1 tarball size is necessary for level 2 cache.
The values for cumulative cache size and LOB cache quota decide allocation of space for the two caches. If values are not provided, a user might see an INSUFFICIENT_CACHE exception. In that case, it is better to revise the cache parameters in order to have a working store.
If this subprogram successfully executes, its actions cannot be rolled back by the user. In that case, the user should call RECONFIGCACHE again with new or modified parameters.
This procedure registers commands for a store with the Hierarchical Store. These commands are sent to the Media Manager for the external storage device associated with the store.
DBMS_DBFS_HS.REGISTERSTORECOMMAND ( store_name IN VARCHAR2, message IN VARCHAR2, flags IN NUMBER);
These commands are sent before the next read or write of content. When the Hierarchical Store wants to push (or get) data to (or from) the storage device, it begins a session (to communicate with the device). After beginning the session, it sends all registered commands for the to the relevant device before writing (or getting) any data.
If this method successfully executes, its actions cannot be rolled back by the user. To restore the previous state the user must call the DEREGSTORECOMMAND Function.
This procedure sends a command to be executed on the external storage device's Media Manager.
This procedure stores properties of a store in the database as name-value pairs.
DBMS_DBFS_HS.SETSTOREPROPERTY ( store_name IN VARCHAR2, property_name IN VARCHAR2, property_value IN VARCHAR2);
Table 49-16 SETSTOREPROPERTY Procedure Parameters
| Parameter | Description |
|---|---|
|
|
Name of store |
|
|
For a store using Tape device, there are three properties whose values must be set by the user, and four properties that have default values. Stores of type |
|
|
Stores using a Tape Device The values for the following properties must be set by the user:
The following properties, which have default values assigned to them when a store is created, benefit from tuning:
|
|
|
Stores of type It is mandatory that the following properties have assigned values, and default values are provided:
|
|
|
The following properties are optional:
|
The specified store must already have been created.
If this subprogram successfully executes, its actions cannot be rolled back by the user.
The same property can be set multiple times to the same or different values using this subprogram
This procedure pushes locally staged data to the remote storage.
Table 49-17 STOREPUSH Procedure Parameters
| Parameter | Description |
|---|---|
|
|
Name of store whose content the client writes from local cache to the external store |
|
|
A non-mount qualified (without mount point) path within the store. By default, its value is |
The Hierarchical Store caches the content files locally in database tables. When enough content is amassed in the cache to make it efficient to write to the external storage device (or the cache is completely filled), the Hierarchical Store creates a tarball out of the local content and writes these tarballs as files on the external device. The size of the tarball created by the Hierarchical Store is controlled by the store property PROPNAME_OPTTARBALLSIZE.
When the amount of free space in the cache is such that the caching of a content file will push the space used above cache_size, the Hierarchical Store will internally call STOREPUSH. The STOREPUSH Procedure creates tarball(s) out of the existing dirty or modified content files in the cache and writes them out to the external device. A STOREPUSH call is not guaranteed to write all the dirty content from local cache to the external storage, since some files may be locked by other sessions.
STOREPUSH has a built-in ability feature allowing it to automatically resume operation. If a STOREPUSH call is interrupted (say by a network outage) after it has transferred some tarballs to the external device, it can be restarted after the outage and will then resume transferring data from the point it was interrupted. In other words, work done before the outage is not lost. STOREPUSH can safely be restarted and the effect is such as if the outage never occurred.
If this method successfully executes, its actions cannot be rolled back by the user.
By default, when path is NULL, all files in the store are candidates for STOREPUSH. If path has a valid input value, all files which are under the namespace of given path are written from the local cache to the external store. If a given path is an existing file, it is pushed out again to the remote store.