Downloads

• Scheduled_Watch_2015.fmw

There are special cases to be aware of when watching files and folders using FME Server. The following page describes these in detail and suggests some helpful tips.

Using ‘Create’ Triggers | Multi-File/Folder Datasets

The complexity of multi-file datasets (e.g. Shapefiles, Geodatabases) presents challenges when using the Directory Watcher. In these cases, you will want to ensure that all necessary files have been uploaded before processing the data. The following discusses the various approaches:

Zip Files

The easiest solution is to package your data as a .zip file. Since FME can read .zip directory for many formats, it makes sense to provide a packaged file into a watched location instead of multiple files.

Decelerator Transformer

If .zip is not an option or you are editing existing data in a watched resource, it makes sense to buffer the amount of time before processing. This can ensure that the data is complete before being utilized. A simple method for this process is to include a Decelerator transformer at the start of the workspace. The “Create” trigger will occur, running the workspace which will then sit for an allotted amount of time before processing. There is a downside to this approach as the job will tie up an FME Engine while it waits. This method is best suited for smaller size multi-file datasets.

Creating a Schedule

An even better approach to watching multi-file datasets is to setup a scheduled job on FME Server. To do this you would capture the Modify trigger from a multi-file dataset, filter for the desired extension (i.e. ‘.shp’), buffer the current time and then use the REST API to create a one-time schedule to process the workspace. The benefit of this approach is that this workflow does not tie up an engine while waiting for the files to finish uploading.

The attached workspace (Scheduled_Watch.fmw) demonstrates this process.

Using ‘Create’ Triggers | Processing Multiple Single File Datasets at Once

When working with single file datasets it is ideal to process many of them using one workspace/engine. For example, What if an image translation service is setup using FME Server and and the FTP Watch publication. When a GeoTIFF file is uploaded to the FTP directory, it gets reprojected and translated into a JPEG. Using this scenario, if 5 images are uploaded to the directory, then 5 separate jobs are created on the server. This can be made smarter with a little bit of intervention.

In the Directory Watch with Idle Time Delay: Advanced Tutorial there is a database used to store information from a Modify notification (i.e. filename). After a certain amount of time, a second workspace running by a scheduled task analyzes the database and processes appropriate files. We can use the same technique to process multiple single file datasets at one time using a single workspace.

To create this workflow you must set up two workspaces, the first stores the Create notification details along with a timestamp into a database while the second workspace is designed to analyze the database and process the data. This workspace should decide which files have been stored for an appropriate amount of time, read them into the workflow using a FeatureReader transformer and then process the entire set of files (i.e. reproject and translate).

Using ‘Modify’ Triggers | Large Files

A large file may cause the watch protocol to report multiple Modify triggers when the file is changed. This is due to the length of time it takes to save changes to the file when it is stored on a local drive. There is a special parameter within the Directory Watch publisher called Min Time Elapsed (Seconds). This setting will impede Modify triggers from being reported for a file until a minimum amount of time has been reached (Note: this setting is not necessary when the directory is using a UNC path as the Modify trigger will occur after the file has completed saving).