How to set up SQS ingest

Instructions

Prerequisites

Workflow - SQS Monitor Process

Transform file - SQSAssetTransform.xsl

Process Engine - Search and Transfer V3 running

To kick this workflow off, make sure that SQS-MONITOR-QUEUE is used.

Mediastores

An example of a generic SQS setup can be seen below:

AWS-S3-SQS - Contains the SQS URL and Access details. This config will be used in other media stores highlighted later on in this guide

 SQS-MONITOR-QUEUE - This is the media store that points to the AWS-S3-SQS config that has SQS access information and the AWSConfigStore should be populated with this. The SQS process monitors the queue and for each incoming message creates an asset XML. This then transfers to the next media store that creates the place holder (FILEINGEST-METADATAIMPORT). Note that keys such as ExcludeRegexFileName are used to ignore ._ mac hidden files. The PathMetadataKey by default is set to ArchivePath assuming the path in S3 is one. This is configurable.

 FILEINGEST-METADATAIMPORT - This media store is used to create the placeholder assets in Curator that will be picked up by the scavenge media store next. Note that the CuratorFolderPattern can be changed to suit your needs.

SCAVENGE-SQS-INGEST - The below is a pretty typical example of a scavenge. It will run constantly due to the ScavengeSchedule being set to * * * * *. It will process two at a time with a total of fifty in one search. The ScavengeSearchString is also pretty generic. It will only run on assets that do not have a webproxy. There is no HiRes path - the ArchivePath is set and there is no fail date within the last day (usually this is set to three days) which will stop this scavenge running on failed assets constantly. With regards to the ScavengeMediaStore, a good example may contain the following Media Stores: PROXY-V3, MEDIAINFO, COPY-HLS-PROXYFILES, DELETE-HIRES-PROXY. When running scavenge on a file that is on S3, the HiRes file will be download to help run proxy and analyse process. The proxy files are then uploaded to S3 bucket and the local HiRes and proxy files will be deleted. Please bear in mind that this is a specific use case and the requirements documented will highlight the expected behaviour; this is only an example.

 COPY-HLS-PROXYFILES - This will copy the proxy files created to an S3 location that is set in the Path key. Note that the OutputProfile is needed. It should be set to an Xfer AWS S3 profile in Device Director. This is mentioned below

 Profiles

AWS S3 - this should contain the Access Key information and this will be stored from the AWS media store. If that media store is correct, it will correctly update this profile

Devices

Xfer AWS S3 device is needed to copy the files, this should be present by default.

 Tips

If you are upgrading a system and there are already assets that are in the system, you will need to update the asset type of OriginalPath and possibly WebProxyPath. You will need to change this in the database of Curator in the Metadatanames table to type 1 (text) so that the workflow does not fail. Make sure that you complete a solr re-index too. Below is what the error will look like:

Make sure that Persistent Search & Transfer V3 is added to the SYSTEM-MONITOR media store so that the scavenge runs.