BlackPearl Workflows
There are two types of configuration; Direct and Asynchronous.
- In Direct configuration, BlackPearl jobs will wait until files are written to tape before the job is completed. Once the job is complete, the asset is marked as Archived.
- In Asynchronous configuration, BlackPearl jobs will complete when the file is written to the BlackPearl cache. A further Search and Transfer is used to check the files are Archived to tape later.
Software
This will need a new version of some services:
- Xfer Agent v2.12.1.30
- Device Director v5.16.1.126
- Process Engine v2.15.1.93
- Workflow Module: BlackPearl v1.0.0.0
The BlackPearl workflow module will create or update default BlackPearl stores, but does not set up Device Director or Curator Server at present.
Curator Server Configuration
Metadata Names
The following Metadata Names should be created or should already exist. These are all for Audio, Video and Image asset types:
Name |
Type |
Description |
|---|---|---|
BlackPearlObject |
text |
The path to the root of the black pearl storage. This may already exist, but must be of type text. If needed, we could use a different name and migrate existing values. |
BlackPearlArchived |
boolean |
Stores whether an archive has been initated on BlackPearl, and in Direct configration, this is also if the archive is written to long term storage. |
BlackPearlArchiveComplete |
boolean |
Stores whether an archive as been completed on BlackPearl, as in the file has been written to long term storage. This is used for Asynchronous configuration only. |
BlackPearlArchiveManifest |
blob |
Stores information about which files are stored on BlackPearl. |
BlackPearlCopyCount |
int |
(optional) Stores the number of domains which have a tape copy. This is only used if the there is a workflow to update status configured. |
Device Director Configuration
Note that there are two BlackPearl devices; we should only be using the Xfer BlackPearl device type. The other BlackPearl device is no longer supported and will be made obsolete in future versions.
Devices
At least one Xfer BlackPearl will need to be created. For this:
- The host address on the first page is the address of Xfer
- On the second page, we fill in the address of the BlackPearl and the associated settings
Profiles
Two profiles are needed; one for Restore and one for Archive.
- For Archive
- Job Type = Write
- Direct: Transfer Wait Mode = Wait for job to complete
- Asynchronous: Transfer Wait Mode = Wait for data transfer only
- For Restore
- Job Type = Read
- Transfer Wait Mode = Wait for job to complete
Process Engine Configuration
MediaStores
Two main BlackPearl specific MediaStores are used. This is the default settings and some notes; see further below for required configuration steps.
BLACKPEARL
|
Key |
Value |
Notes |
|---|---|---|
ArchiveManifestMetadataName |
BlackPearlArchiveManifest |
This may have been known as BlackPearlManifestKey before, but is unlikely to have been configured. It is now required. |
AvailableMetadataKey |
BlackPearlArchived |
|
BlackPearlAccessKey |
Value from the BlackPearl interface |
|
BlackPearlHost |
The BlackPearl host name |
This can include a port and a protocol if needed. By default we use http on port 1080 |
BlackPearlJobRetries |
1 |
Number of retries if there is no cache space. |
BlackPearlJobRetryDelay |
00:30:00 |
Delay between retries if there is no cache space. |
BlackPearlProxy |
|
Optional |
BlackPearlSecretKey |
Value from the BlackPearl interface. |
This value should be encrypted when updated. |
BucketMetadataKey |
|
If this is used, the value from this field will override the generated path and folder pattern. |
DeviceGroup |
|
|
FolderPattern |
{asset.folder_path}/{asset.name}-{System.DateTime} |
This pattern is appended to the value in Path to make the complete archive location. |
HiResAccessible |
False |
This is to avoid the transfer asset workflow looking to see if files are present automatically. |
InputProfile |
BlackPearl-Archive |
The Device Director profile for Xfer BlackPearl. This should have job type of Write. |
OutputProfile |
BlackPearl-Restore |
The Device Director profile for Xfer BlackPearl. This should have job type of Read. |
Path |
Archive |
The name of the bucket, and optionally a prefix where the archived files will be stored. |
PathMetadataKey |
BlackPearlObject |
|
Source |
HI-RES |
The Hi-Res store must be at the same physical location as the BlackPearl. |
StoreType |
Archive |
Required for Transfer |
Workflow |
Spawn - SpectraS3 |
Required for Transfer |
BLACKPEARL-STATUS
This is used as a transfer target to set the BlackPearl status based on number of tape copies.
|
Key |
Value |
Notes |
|---|---|---|
ArchiveCompleteMetadataName |
BlackPearlArchiveComplete |
|
BlackPearlConfigStore |
BLACKPEARL |
|
SetArchiveCompleteMethod |
TapeDomainCount:2 |
If you have a single copy to tape, this should be set to 1 rather than 2. This specifies that archive complete is when there are two different storage domains in the list of tapes that contain parts of this file. |
SourceRequired |
False |
Required for Transfer. |
StoreType |
Dynamic |
Required for Transfer. |
TapeStorageDomainCountMetadataName |
BlackPearlCopyCount |
This can be cleared if you don’t want to store the total copy count. |
Workflow |
Spawn - Get Blackpearl Status |
Required for Transfer. |
Further MediaStore Configuration
The following will need to be done for both Direct and Asynchronous mode:
Update HI-RES to have BLACKPEARL as source.
-
Update BLACKPEARL with relevant configuration details
Make sure the secret key is encrypted.
Asynchronous Configuration
In addition to the above, you will need a Search and Transfer set up to update jobs that have had an archive started, but not yet completed.
Key |
Value |
Notes |
|---|---|---|
AssetTypes |
audio,image,media |
Search on archivable types. |
ScavengeEnabled |
True |
Search and transfer enabled. |
ScavengeMediaStore |
BLACKPEARL-STATUS |
Use this store for transfer. |
ScavengeRepeatAfter |
01:00:00 |
Repeat every hour. |
ScavengeSearchProcess |
Spawn - Search & Transfer |
Use standard search and transfer. |
ScavengeSearchString |
BlackPearlArchived:true AND -BlackPearlArchiveComplete:true |
Search for assets that have had the archive process started, but haven’t had complete set yet. |
Usage suggestions
In order to track assets that have failed to archive, you can search for all assets that have an archive date in the distant past and don’t have archive complete.
You can also create a search and transfer to Update Asset Metadata in order to add an additional metadata field for assets that have not been archived completely.
If there are many archived assets simultaneously, you may want to add a ScavengeSearchSize to the above scavenge, otherwise newer files may not get marked.