This article covers the many ways of performance-tuning FME, in different FME applications, and at different stages of project development.

On a computer, work is carried out using system resources. If performance is the time taken to carry out useful work, then excessive use of resources, or time lost on useless work, hinders performance.

Many such factors are diagnosed or reported via the FME log file, so this article covers how to read a log file in some detail. This particularly helps to find out whether there is an overall performance slow down, or a bottleneck at one stage of the process.

System Hardware and Configuration

The hardware upon which FME is run, and its configuration, can greatly affect performance.

Technical Specifications

Technical specifications for FME - and the system requirements for it - can be found on the Safe Software web site (under the Downloads page).

System Configuration and the FME Log File

The FME log file contains important information about the system configuration and is a good place to look when trying to fine-tune performance.

Temporary Directory Location

One of the first items of importance in the log file is the temporary directory:

INFORM|FME Configuration: Temporary folder is `C:\Users\imark\AppData\Local\Temp', 
set from environment variable `TEMP'

You'll also see a line commenting on the amount of disk space available in that directory...

INFORM|System Status: 46.20 GB of disk space available in the FME temporary folder 
(C:\Users\imark\AppData\Local\Temp)

When FME runs a large translation, it often requires a lot of temporary disk space. Available disk space is important, but so is the speed of any disk activity.

Tip: Point your temporary directory a fast disk with plenty of space. An SSD (solid-state drive) disk is preferred. See Setting the Temporary Directory for how to set FME_TEMP. Where possible, use a different disk to the one used by the operating system.

You could also try a RAM drive (also called a RAM disk) for your temporary directory, although the benefits are more apparent on memory-limited 32-bit systems. Modern systems should be using excess memory for caching anyway.

System Status

Other lines in the FME log tell us the status of the machine in terms of platform, hardware, and encoding:

INFORM|System Status: 42.69 GB of disk space available in the FME temporary folder
INFORM|System Status: 24.00 GB of physical memory available
INFORM|System Status: 95.99 GB of virtual memory available
INFORM|Operating System: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601)
INFORM|FME Platform: WIN64
INFORM|Locale: en_CA
INFORM|System Encoding: windows-1252

Physical memory is the amount of memory installed in the system. Here it is 24GB. Virtual memory is an aspect of the system's own memory management systems. It's a technique where the system swaps data around to compensate for low-memory scenarios. Here it has 95.99GB of space designated as virtual memory.

Tip: Obviously, the more physical and virtual memory available, the better performing the computer will be.

Virtual Machines

A Virtual Machine is a computer system that does not physically exist but is emulated on a physical computer.

Tip: A Virtual Machine can provide superior performance on a temporary basis, which is ideal for running FME workspaces with large amounts of data processing.

However, FME deployed on a Virtual Machine can sometimes run slower than on an equivalent desktop machine. This is often due to a poorly configured VM environment where several VM machines are sharing resources that are actually designed for a single-use machine, such as network bandwidth, memory resources, and shared drives.

Network latency on the VM cluster may be an issue when accessing a database from a VM. The host machine must be capable of supporting the number of guest VM machines, including the network bandwidth to support database access and network file system.

Tip: Past experience has shown that disabling dynamic memory management for the virtual machines' environment, may improve memory allocation and performance.

Machine Comparisons

Sometimes the same workspace performs differently on different computers, even when the specification looks identical. If you see different performance characteristics on different machines, then it might be because of one of these differences:

• Is one computer physical and the other virtual?
• Is the operating system the same, with the same version and the same OS updates (service packs)?
• Is there a larger or faster CPU cache to access data more quickly on one system?
• Is the version, build, and bitness (32-bit vs 64-bit) of FME the same?
• Is the FME temporary directory on the same sort of drive (local/network or HDD/SSD)?
• Are the data sources on the same sort of drive (local/network or HDD/SSD)?
• Does each install of FME have the same access permissions to the source/destination data?
• Is one system running extra software that might consume resources?
• Is one system running potentially blocking services like a firewall or antivirus?
• Does one machine have better network bandwidth?
• Is one machine physically closer to the database server than the other?
• Is there competition for memory, disk, network, CPU, or other limiting resources?

In general, to improve performance, many factors need to be static, and those that cannot be controlled need to be minimized. Using different machines destroys most controls.

32-bit Windows and 32-bit FME

At the time of writing (April 2020), fewer people use 32-bit Windows than when this article was first created. For those that still do use 32-bit, Windows restricts the memory available to a single process to 2Gbytes. This low number can negatively affect performance.

You may increase the amount of memory available by setting the /3GB Switch.

Sometimes 32-bit FME is used because it supports an older format that 64-bit does not. In this case, you can use 32-bit FME running on a 64-bit workstation, to access greater memory resources.

However, overall our recommendation is to use FME 64-bit running on a 64-bit processor.

Tip: Use 64-bit FME wherever possible. However, if 32-bit FME is unavoidable, either increase the amount of available memory on 32-bit Windows by using the /3GB switch or use a 64-bit computer.

Workspace Authoring

There are a few tools available in FME to help a workspace run faster, while it is being authored and debugged, thus speeding up the authoring process.

Feature Caching

Best Practice for authoring (and debugging) a workspace is to make small changes and run the workspace often to test the results. This way it's easier to pinpoint a problem when it occurs, rather than having a large number of untested transformers to check.

Frequently running a workspace can be a time-consuming process, but this is mitigated by Feature Caching.

Feature Caching (or Run with Full Inspection in FME 2017 and earlier) stores the results of a workspace, at every step of the translation. Having this data to hand helps you to inspect the results at each stage. However, it also means that running the translation does not mean having to run the entire workspace. Instead, cached results can be used in place of running prior parts of the workspace.

This technique saves time in re-running a workspace, although it does consume disk space and resources when writing the cache. Additionally, it can give a false impression of performance when a workspace is put into production without caching.

It is important that Feature Caching is turned off when the workspace is complete and put into production. Feature Caching can prevent performance tools like Parallel Processing from working, and in a production environment, it creates caches that are rarely used. Additionally, a workspace in production should not need to be started mid-way through. It's for this reason that Feature Caching is automatically turned off when running a workspace on FME Server.

Feature Counts

Feature Counts are the numbers that appear on connections when you run a workspace in FME Workbench. The real-time animation of these counts often gives an indication which transformers are blocking your data or slowing the workspace.

A combination of Feature Caching and Feature Counts both show the feature counts and caches the data at each link on the workspace, which can be particularly useful.

Workspace Design Patterns

It's important to design your workspace for maximum performance as you create it. There are various patterns of design that promote better performance.

Attribute Cleaning

During a translation, FME holds your data either in memory (physical or virtual) or cached to a disk. Obviously, performance is improved by reducing the amount of unnecessary data being held, both features and the components of those features.

One particular aspect is an excess of attributes. Often not all attributes on the source data are required for processing or output. In this scenario, it's better to avoid reading those attributes or to use a transformer to remove those attributes as soon as possible in your translation.

Design Pattern:

• Read Data
• Remove Excess Attributes
• Process Data

In other words, only carry through the translation any geometry and attributes you intend to be available on the output. An AttributeManager, AttributeRemover, AttributeKeeper, or GeometryRemover transformer can be used to remove excess components and should be used as early in the translation as possible.

In particular, lists (an attribute with multiple values) or geometry stored as attribute values, can take up resources, because they tend to carry larger amounts of data. For example, a feature joined to 1,000 records, storing those records as a list, is now the equivalent of 1,000 features!

Another method of reducing attributes is to not read them at all. See the section on databases (below) for more information.

Data Filtering

Similarly to excess attributes, excess features use valuable system resources and should be removed as early in the translation as possible.

Design Pattern:

• Filter Data
• Process Data

In other words, filter unwanted data so that it does not incur unnecessary processing.

In this example, the author measures the area of features and then filters the data:

This is entirely the wrong way around. The workspace wastes time measuring features that are later discarded (Tester:Failed).

Reducing Duplication

A common bad design pattern (sometimes called an Anti-Pattern) is a chain of duplicate transformers. This is often not as efficient as processing data entirely in a single transformer.

For example, chaining together ExpressionEvaluator transformers in this way, where each carries out a different step of one overall calculation, is not the most efficient way of processing data. It would be far better to condense the actions into a single AttributeManager transformer.

Tip: A chain of duplicate transformers is a sign of an anti-pattern, and you should investigate whether there is a better way to produce your goal.

Memory Reduction Strategies

The FME engine passes features through in a mixture of ways in order to maximize performance.

Some transformers in Workbench operate on one feature at a time. These are known as feature-based transformers. They can operate on one feature at a time because the process they carry out does not need different features to interact.

Other transformers work on groups of features. These are known as Group-Based Transformers. Group-based transformers are the ones that process multiple features simultaneously; for example, intersecting many line features to produce a topological network.

Obviously, any transformer that works on a group of features must hold them all in memory at a single time to do so, incurring processing costs. This is known as Feature Holding, and should ideally be considered when designing an overall FME strategy.

Tip: The FME transformers documentation contains information about whether a transformer is Group or Feature-Based, and whether it is a Feature Holding transformer.

Fortunately, many group-based transformers can reduce their memory footprint under certain conditions by holding onto fewer amounts of data. However, it is up to the workspace author to set a parameter to confirm that these conditions are being met.

For example, the DuplicateFilter transformer separates features containing a duplicate attribute key. The transformer can perform more efficiently if it knows that the data is already sorted into key order, but it is up to the workspace author to set the Input is Ordered parameter to confirm that this is the case.

Tip: Many feature-holding transformers have parameters that can improve performance under the right conditions. Group-By Mose is one such parameter and is common to many transformers. Clippers First is a parameter that only applies to one transformer (the Clipper). Inspect the documentation closely to look for ways to reduce the load on group-based transformers.

Writer Order

Multiple writers in a workspace have a set order in which they are executed. The first writer in the list is the first to write its data, while subsequent writers cache data until it is their turn.

Performance is hindered by caching data. Therefore it makes sense to promote the writer handling the most data to the top of the list so that its data is not cached.

Writers can be ordered by dragging them up and down the list in the Navigator window.

Additionally, the advanced Workspace Parameter Order Writers By, allows you to set the order that writers are executed (either the order in the Navigator or the order in which features arrive).

Tip: When you have multiple writers in a workspace, always ensure the one getting the larger amount of data is the first writer in the list. Need an example? This FAQ tells you all you need to know.

Tip: For peak performance, tiles output from the RasterTiler or WebMapTiler should be written in the order they are output from these transformers. Do not use a Dataset Fanout here as it re-orders the output and negatively affects performance.

Database Design Patterns

Reading from and writing to databases provide unique opportunities for performance tuning. In general, processing can be quicker in the database itself, rather than the FME engine.

When reading data from a database, the ideal pattern is to filter the data as it is read, rather than in FME:

Design Pattern:

• Filter Data
• Read Data
• Process Data

Or...

Design Pattern:

• Read Data (with a Filter)
• Process Data

A "filter" can be applied by using the SQL Statements and SQL WHERE clauses available in most FME database readers.

Another way to "filter" data is to not read unnecessary attributes from the database table. These can be turned off in FME feature types, by unchecking the attribute in the User Attributes tab:

When joining data, SQL Joins can be quicker than an FME FeatureJoiner transformer. Create a database materialized view for even better performance and to simplify your workspace (although sometimes DBAs won't allow you to do this).

For ArcSDE, SQL Statements are only supported for non-spatial tables. For spatial tables use: sdetable -o create_view to create a view that contains a spatial column in the join.

The ArcSDE help has useful tips on how to do this, see the ArcGIS Help - A quick tour of views in the geodatabase

You can create complex table joins using a combination of sdetable -o create_view followed by SQL ALTER VIEW.

Tip: Performance improves in some cases by handing off processing to a database.

Maximize Use of System Resources

It may seem strange to suggest using as many resources as possible, but this is acceptable as long as they are doing useful work. For example, if your computer has an 8-core CPU, it makes sense to split your work into eight parts, so that each core can do its share. Restricting the process to a single core is not making the best use of system resources.

For FME Server, the recommendation is to have one engine per core, although this can vary depending on the type and size of data being processed.

In terms of memory, it can make sense to read the entire contents of a database table in one query - regardless of whether every feature is required or not - in order to have the required records immediately available to FME. Reading individual records on demand, though it incurs lower network traffic, may overall be a slower strategy.

Partitioning and Load Balancing

An extremely large amount of data can be difficult to process in a single task. You may want to consider partitioning the data into groups (maybe on the basis of a geographic region) and processing each separately.

For example, you could divide the entirety of a Canada-wide dataset into a separate group per province, using a Where clause to select the required data. This way, the data is divided into ten different groups.

At this point, either each group is processed consecutively (so that approximately only 10% of the data is processed at any one time by the sole FME engine) or each group is processed concurrently over a number of engines.

Miscellaneous Design Tips

Tip: For maximum performance, in an arithmetic editor, use the functions @add(), @multi(), and @div() instead of using the equivalent operators (+ * and /). These functions work at a lower - much faster - level of processing. Plus, they have the added bonus of handling nulls better.

Post-Mortem Performance Tuning

Analyzing the results of a process in order to assess or improve future performance is called post-mortem debugging. In FME, the key result of a process is the log file.

Therefore, before being able to tune a workspace, it's vital to understand how to read an FME log file. Without this knowledge, a user will often jump to incorrect conclusions about a translation and start looking for performance issues in the wrong places.

Log Timestamps

When trying to speed up a translation, it is always beneficial to check the timings noted in the log file to determine exactly where the time was being spent. Timings are always written to the log file, but only optionally written to the log window.

Tip: Check that Tools > FME Options > Translation > Log Settings > Log Timestamp Information is checked. This will cause timestamps to appear in the FME Workbench log window.

Interpreting Log Timestamps

A log timestamp always shows three columns:

• An absolute datetime stamp (2020-04-22 14:43:13)
• A total elapsed time for the translation (8.8 seconds)
• An elapsed time since the last log message (0.3 seconds)

Tip: Elapsed Time (measured in seconds) is actual FME Processing time. This may not be the same as the overall length of the process.

For example, here the elapsed time shows the process took over 6 minutes, but the Total field reports FME used only 25.8 seconds.

Absolute Time | Elapsed Time Total| Incremental Elapsed Time
2020-04-22 14:43:06| 8.5| 0.0|
2020-04-22 14:43:13| 8.8| 0.3|
2020-04-22 14:46:29| 18.0| 9.1|
2020-04-22 14:49:29| 25.8| 7.9|

See How FME logs processing time for more information.

FME Session Duration

At the very end of your log file you'll see:

INFORM|FME Session Duration: 1 minute 27.2 seconds. (CPU: 13.7s user, 0.9s system)

The duration tells us how long the translation took (which should match the difference between the first absolute timestamp in the log, and the last absolute timestamp.

The CPU time will approximate the last "Elapsed Time Total" time in the log timings. It tells us how much CPU time was occupied by the FME process.

Tip: The difference between the session duration and CPU times is an indication that there are aspects of the translation outside of FME's control: slow database queries, network latency, or slow HTTP requests.

Key Stage Performance

A typical FME translation has a number of key stages, often reading, transformation, and writing. A preliminary assessment of performance involves timing each of these key stages.

However, because FME processes features in a mix of ways (feature-based, group-based, and bulk mode), it is difficult to assess these stages from a single translation. For example, FME starts to transform data before it has finished reading it.

The log message Emptying Factory Pipeline generally means reading data is finished:

2020-02-03 11:37:47| 342.7| 0.5|INFORM|Emptying factory pipeline

Here it took 342.7 seconds (about 6 minutes) to read the source data. But, as you now know, this may include time spent processing the features within the workspace. Disabling all but the readers gave this result:

2020-02-03 14:44:43| 66.5| 0.3|INFORM|Emptying factory pipeline

Only one minute instead of six. This tells us that 80% of the time was spent processing the data, and only 20% reading it. In this case, the user thought the reading was the bottleneck, but this shows it was the transformation.

Tip: To measure the performance of key stages, run the workspace with everything but readers disabled. Then run it again with just writers disabled. With the log timings from these runs (and the original run), you should be able to calculate just how long FME spends reading, writing, and transforming data, and this will show where to concentrate your performance efforts.

Database Performance Tuning

Databases are an important component of many datasets, and the log file will help us determine how good our database performance is, plus how well FME is interacting with the database.

Database Queries and Indexing

Here is one example of a query sent to a database:

2020-05-14 17:18:52| 476.1| 0.0|INFORM|Started SQL cache prefetch

2020-05-14 17:25:10| 476.2| 0.1|INFORM|Finished SQL cache

Note the difference in actual time on the left; you can see that the time between when we issue the SQL prefetch and until it’s done is roughly seven minutes. However, FME logs only 0.1 seconds of CPU time. From this, we can say that the remaining time was spent by the database in running the query it was given.

To improve performance, the user needs to look at how the database is structured and how the query is written. Perhaps the field being searched on isn’t indexed? Maybe the query supplied isn’t as efficient as it could be?

Tip: Check the log carefully to find out how much database-related time is spent outside of FME and see if you need to improve your database efficiency.

Database Indexing

Indexing is not only important when reading data, but it has a vast effect on writing data too.

Writing to an unindexed table is quick because the database has no overhead work to do.

Writing data to a table that is indexed takes a lot longer because - for each new record - the database has to index the data immediately.

As above, the reported CPU time doesn't change because it is the database server - and not FME - that is doing the indexing work.

Tip: Where possible, drop indexes before doing a bulk load into a table, then recreate them after the load is complete. It is often quicker than leaving the index in place during the data load.

Bulk Reads

Each query sent to a database incurs considerable expense in terms of time and performance. This overhead cannot be emphasized enough. The fewer the number of queries to a database, the faster a workspace will be. If necessary, read a larger-than-required set of data in a single query, rather than carrying out multiple, individual queries, on single features.

Truncating and Dropping Tables

Related to indexing is the difference between truncating a table and dropping it. FME has settings to do either, but when you truncate a table, the index remains, and subsequent data loading is slower. When you drop a table first, a side effect is that all indexes are also dropped; hence data can be written faster because no indexing is taking place.

Tip: Consider using the option to Drop a table, rather than Truncate, in order to get better performance from a bulk data load.

Chunk Sizes and Transaction Intervals

Data written to a database is held in two places. FME holds the data until it has a set amount of data (the Chunk Size). Then it passes the data to the database. The database holds the data until it has a set amount of data (the Transaction Interval). Then it commits the data.

A higher chunk size (also known as Features per Bulk Write) means that FME holds more data, but makes fewer requests to the database. Each request to the database comes at a performance cost, so are best avoided.

A higher transaction interval means that the database holds more data, but has to carry out fewer commit actions. Committing transactions is an expensive operation that it is best to avoid.

At their extreme low ends, smaller chunk sizes and transaction intervals mean you could be passing records to the database and committing them one at a time! This would be extremely costly.

But higher values can use more system resources and run the risk that a failed record means an entire set of data uploads need to be rolled back.

Tip: Chunk size and transaction intervals are parameters that can - with some experimentation - be optimized to pass and commit data in the most efficient manner.

FME Updates and Improvements

As FME is developed and refined over time, it gains a number of performance improvements. These can be classified as two different types of upgrades.

An automatic upgrade is one that is applied to the FME engine as you upgrade the FME version. These performance improvements are used automatically. They are backward compatible and won't change the behavior of a workspace, just its performance.

A premeditated upgrade is one that is initiated by the FME user. These performance improvements become available as you upgrade FME, but aren't used automatically. They may or may not be backward compatible, so it is important that you be aware of any differences before applying them.

Premeditated Upgrades

Whenever you upgrade to a newer version of FME, you can open a workspace and look for a section of the Navigator window labeled Upgradable Transformers.

These are optional upgrades that you can apply to a transformer. You should check the changelog for the transformer (available when you carry out the upgrade) to see if there are any changes that might affect the results of the transformer.

In general, if the changelog includes the phrase:

Added support for Bulk Mode

...then you should get a decent improvement in performance, like with this StatisticsCalculator in 2020:

In fact, the StatisticsCalculator in FME 2020 is on average 88% faster than prior versions, with some cases running 100 times faster! Those are official numbers from our QA team.

But as you can see, there are many other updates to the transformer, and you should check these to ensure they won't interfere with your expected results.

What's even better is that the performance benefit might even extend beyond that particular transformer, to the rest of the workspace. That's because a transformer that doesn't support Bulk Mode can cause the rest of the workspace to revert to an older, less efficient method of processing.

So upgrading one transformer to support bulk mode can cause all subsequent transformers to start operating in bulk mode too!

Monitoring Performance & Profiling

You can run a profile of your FME Workspace to obtain a detailed log of how much time is spent in each underlying factory or function. Use Tools -> Edit Header to add the directive: FME_PROFILE_RESULT_CSV i.e. FME_PROFILE_RESULT_CSV C:\TEMP\profile.csv

Note: Don't forget to remove the profile directive for your production workspace since profiling also has an impact on performance!

It's sometimes worth using a Performance Monitoring tool (such as PerfMon) to log the CPU and memory usage of a process.

FME Memory Management

Memory Management is mostly an automatic process that the FME engine carries out during a translation. However, there are log file messages that relate to memory management and can be useful to inspect.

FME Memory Configuration

At the start of the translation, FME determines the amount of memory available to it and plans how to use that memory to give optimum performance. Two lines in the log file indicate these plans:

INFORM|FME Configuration: Start freeing memory when the process exceeds 71.99 GB
INFORM|FME Configuration: Stop freeing memory when the process is below 53.99 

Note that this refers to virtual memory (the system's own mix of physical memory and disk space). In the above example, FME plans to use as many resources as required, up to a limit of 71.99GB. At that point, it will carry out its own swapping, moving data out of memory and on to the temporary disk folder.

When it has released enough memory to go below 53.99GB of usage, it will again start to make use of memory instead of caching to disk.

Because caching to disk is an expensive operation, it's best for performance if FME can stay within the maximum limits. If it cannot do so, then it will record its actions in the log.

Optimizing Memory Usage

If you see either of these messages in the FME log:

INFORM|ResourceManager: Optimizing Memory Usage. Please wait...

INFORM|Failed to free sufficient memory to reach the process usage limit

...they are evidence that FME has to optimize its memory use. The first message reports that FME has reached its upper limit and is now releasing memory by caching to disk.

The second message warns that FME has not been able to release enough memory to reach the lower limit. This may be because of extra applications starting and taking up the memory FME expected to use. More information can be found in this article.

In some cases (FME 2019.0 & earlier), the messages are spurious, and FME is running fine. If your FME task completes in the expected time frame, then this is likely the case.

However, real problems can occur for a number of reasons:

• Insufficient resources.
  • Insufficient Specification: Your computer may simply lack the resources (memory and disk space) required to handle the amount of data being processed.
  • Competing Applications: Check the task manager to see if there are other applications taking up resources, or you might also have other FME jobs running (particularly if you are seeing this on FME Server). Where possible, don't start new applications once FME is running. They make take up resources that FME was expecting to be able to use.
  • Memory Fragmentation: Fragmentation occurs over time as applications start and stop. If your system memory has become fragmented, it's harder for FME to allocate contiguous memory. You can identify this if your job runs fine after a system restart, but the same job starts to report memory optimization later in the day. Recourse: restart your computer.
• Workspace Issues
  • Workspace Complexity: Multiple blocking transformers (Group By) can cause extreme memory use. You might need to increase your resources or redesign your workspace, if possible.
  • Debugging Options: Remember to turn off feature caching or other debugging tools like Inspector or Logger transformers.

If you consistently see the Optimizing Memory Usage for the same FME workspace, then share the workspace and data with us (Safe Support), and we can try and fix the problem.

Memory Redline Parameter

WARNING: The value of this parameter should NOT be adjusted unless you have exceptional circumstances or have been directed by Safe Software. Changing this value can cause unexpected behavior and reduced performance. In most cases - particularly FME 2018 or newer - no benefit accrues from adjusting the memory redline parameter

The Resource Manager automatically determines the optimal total memory the FME Engine process should use. It also dynamically allocates this total memory optimally to the algorithms within FME requesting it.

The FME_ENGINE_MEMORY_REDLINE directive is a hint to the FME Engine on how aggressive it should be in consuming memory. It takes a value between 0 and 1 (0.5 is the default value). For more aggressive memory usage, a value above 0.5 should be used. For reduced memory usage, a value below 0.5 should be used.

The risk in being too aggressive is the process running out of memory or the machine "thrashing" (a state of constant memory errors). The risk in being too conservative is that the process may take disproportionately longer to complete.

To set this value in FME Workbench, select Tools > Edit Header from the menubar and enter:

FME_ENGINE_MEMORY_REDLINE 0.5

...as the very first line. For FME Server Engines, see How to Control FME Server Engine Memory Usage.