IntroductionPython is a programming language that can be used within FME to accomplish tasks either before or after FME runs or to perform tasks within FME which are not possible with standard FME tools and transformers*. For example you can use a python script to extract source data from a zip file before FME runs or to send an email after FME runs. In this article we will look at simple examples of python scripting in the 4 main places python is used in FME:
- Startup Python Script
- Scripted Parameter
- PythonCaller and PythonCreator Transformers
- Shutdown Python Script
For some basic resources on programming in python see the FMEPedia article: What is Python and How Can I Install It? http://fmepedia.safe.com/articles/FAQ/What-is-Python-and-How-Can-I-Install-It
Python and FME Basics
FME installs its own python interpreter which is used by any scripts running python in FME. If you want FME to use your own python interpreter there is a setting in Workbench. Please see this article for more information:http://fmepedia.safe.com/articles/How_To/Choosing-a-different-Python-Interpreter-installation
FME Objects and PythonMuch of FME's core functionality is available within python scripts by using FME classes and methods from the FME Objects python API. For example you can get at the FME log file, FME parameters and use any of the countless FME methods such as getArea or reproject etc. (more on this later). To use FME Objects in your python script with FME 2012 and newer you will need to import FME Objects with the statement:
For more information on the new FME Objects python API please see the article here:
FME Variables in Python Scripts
A number of FME related variables are available in FME Startup and Shutdown scripts. As of FME 2014 these variables are accessible in the fme module so you will need to include the statement import fme to access them. For example the fme.status variable can be true or false at the end of a translation depending on the success or failure of the workspace. For a complete list of FME variables available in Startup and Shutdown Python scripts please go to Help > Workspace Basics > User Interface Reference > Workspace Navigator > Workspace Parameters > Advanced > Startup and Shutdown Python Scripts.
Prior to FME 2014 these variables were named differently and it was not necessary to import the fme module. For example the status was returned in the variable FME_Status. For these older versions of FME the FME variables are documented in the FME Quick Translator Help.
FME Parameters in Python Scripts
One of the FME related variables described above returns a dictionary of FME parameters and their values when the workspace was run. These parameter values are often useful in a python script, for example when you want to know file path of the source dataset used in a reader or some other user defined parameter. In FME 2014 you can get the values of FME parameters by accessing the dictionary fme.macroValues. For example if you want to get the value of the source dataset parameter for your AutoCAD reader and the parameter is called SourceDataset_ACAD and you can access this parameter in python like this:
import fme SourceDataset = fme.macroValues['SourceDataset_ACAD']
Please note: if using an FME version older than FME 2014 use the syntax below:
SourceDataset = FME_MacroValues['SourceDataset_ACAD']
We can also set FME parameters by creating a scripted parameter and returning the value from the script. This is covered below in the section on Scripted Parameters.
Startup Python ScriptIn the Workbench Navigator pane under Workbench Parameters - > Advanced you will see a parameter called Startup Python Script. Here you can enter some Python code that will execute before the FME translation.
Note: If you want to set or return an FME parameter this is not place - see Scripted Parameters below.
Example of Startup Python ScriptAttached workspace: StartupPython.fmw
By default when FME writes to an existing Microsoft Excel file we insert new records, but what if you want to delete the Excel file each time? (See the article http://fmepedia.safe.com/articles/How_To/Example-scripts-for-deleting-Excel-files-prior-to-writing)
We can delete the destination dataset before running FME using the following Start Up Python script (workspace attached):
import os OutputFile = str(FME_MacroValues['DestDataset_XLS_ADO'])FME World Tour 2013 - 1Spatial & Star-Apic (London and Leeds) print OutputFile if os.path.exists(OutputFile): os.remove(OutputFile)
We are getting the value for the destination dataset parameter from the FME_MacroValues dictionary. Your destination dataset parameter may have another name and which you can find by right clicking on the parameter and selecting Edit User Parameter Definition where you can see the Name:
Scripted parameters are extremely useful when we want to set a parameter in FME based on something we derived or calculated from another parameter or parameters. For example you may want users to select themes or groups of layers and have your script set the the individual feature types to read within these groups.
Example Scripted Parameter
The FME Store has a good example of scripting the feature types to read parameter based on a group selected by the user. In FME Workbench you can access this workspace from the FME Store by using the menu File - New and browsing the Store on the bottom left. There is a group of workspaces called Scripting in WB - select the one called Python scripted parameters - layer groups and click Choose to download and open.
Use the File - Prompt and Run to see the published parameters. Notice that the layers you are promoted for are groups: Bus Information, All Transit, Road and Rail. Choose Bus Information and run the workspace. Notice that the workspace read both the BusRoutes and the BusStops feature types.
Lets look at the scripted parameter used to tell FME that choosing Bus Information should return both of these feature types to the Feature Types to Read parameter.
In the navigator pane expand User Parameters -> Private Parameters and you will see a python scripted parameter called feature_types_to_read. Notice that this parameter is linked to the actual Feature Types to Read parameter of the reader in the workspace. Whatever value the script returns will be used by the SDF reader to determine which feature type to read.
Go back to the scripted parameter and double click on it and click the ellipsis to open the editor. Here is the script used inside:
featureTypes = '' if FME_MacroValues['layers'].find('Bus Information') != -1: featureTypes += 'BusRoutes BusStops ' if FME_MacroValues['layers'].find('Road and Rail') != -1: featureTypes += 'metrorail Roads ' if FME_MacroValues['layers'].find('All Transit') != -1: featureTypes += 'metrorail Roads BusRoutes BusStops Labels ' #Debug #print featureTypes return featureTypes
You can see that we have a series of if statements which find out which layer the user chose using the FME_MacroValues[ ] dictionary and then sets the value of featureTypes which will be returned. The last line is the most important as this is where we "return" the actual value of the scripted parameter with statement
This return statement must always exist in a python scripted parameter because it is here that the parameter value is given to FME. Also notice the commented out print statement. You can use print to help you debug your scripts by returning variable values to the FME log. For more in logging with python see this article: http://fmepedia.safe.com/articles/Error_Unexpected_Behavior/Logging-with-Python-scripts
Example Scripted Parameter 2 (Complex)
If you understand the basic scripted parameter example above you should have no trouble understanding this more complex example. The attached workspace UnZipSource.fmw accepts a published parameter for a zip file and a parameter for a directory to unzip to. A scripted parameter unzips the file and returns the directory to the source dataset parameter of the reader.
PythonCaller TransformerThe PythonCaller transformer lets you do things to features or groups of features using a Python script. Before using this transformer please be sure there is not already an FME transformer which does the task you want to do. Feel free to ask us in support (www.safe.com/support) if there is a transformer that can help you.
The PythonCaller can call a Function which you can use to process one feature at a time or a Class where you may want to do things to groups of features. In either case the EntryPoint parameter of the transformer is where you name the function or class in the script to run. The PythonCaller can use the full range of FME Objects including numerous methods and classes.
PythonCaller Example 1 (Function)
Attached workspace: PythonCallerFunction.fmw
As with all transformers FME Workbench will display a detailed transformer description when you have added the transformer to the canvas and have it selected. In this case we will use the sample code included in the description to add a timestamp to features. This is not an ideal example because this task could be done with the standard FME transformer the TimeStamper.
1. Add a PythonCaller to your workspace and select it. You should see the Transformer Description in a window below. If you don't see it there maybe a tab for it beside the log window or you can open it from the menu - View -> Window -> Transformer Description.
2. Open the transformer properties and click on the ellipsis to open the code editor. Notice there is already a Template Function and Class there for you. Delete the class because we are just using a function for now. Rename the function timestamper and copy in the code from example in the transformer description or from below. You may need to add some line returns and indentation to the code so that it looks like this (remember indents matter in python):
import fmeobjects import time # Template Function interface: def timestampFeature(feature): curTime = time.ctime(time.time()) feature.setAttribute("timestamp", curTime)The script uses some FME objects methods so we need import fmeobjects and we are using the python time module so we need to import it as well. The function definition accepts FME feature as its only argument meaning all features will enter the function one by one for processing.
A new attribute is added to the feature with setAttribute method on the feature - this is actually an FME Objects method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the function timestampFeature. Because we have added a new attribute called timestamp we can expose it by entering its name in Attributes to Expose.
4. Use a creator to create some features to send you the PythonCaller and a Logger to look at the output. Every feature should have a timestamp attribute.
PythonCaller Example 2 (Function)The FME Store includes a custom transformer called the FuzzyStringComparer which uses the Python difflib module to compare two string attributes and calculate a similarity ratio. In some ways this is a better example because it is something we cannot do with a regular FME transformer.
1. Open a blank workspace and add the FME Store transformer FuzzyStringComparer by typing it on the canvas or by browsing the FME Store transformers in the Tranformer gallery. Right-click on the transformer and select Embed which will allow you to edit the transformer and see its contents in the same workspace.
2. Once the FuzzyStringComparer is embedded right click on it to Edit and see the contents. Locate the PythonCaller transformer and edit its properties to access the script editor. Notice again we are using a function - this time to compare string attributes on a feature by feature basis. Again we are setting a new attribute which is called FuzzyStringCompare.ratio. An AttributeExposer is used later in the workflow to access the attribute we have created.
string1 = feature.getStringAttribute('FuzzyStringCompare.string1').lower()
string2 = feature.getStringAttribute('FuzzyStringCompare.string2').lower()
ratio = difflib.SequenceMatcher(None,string1,string2).ratio()
In the above example the function used to get an FME attribute value is from the now deprecated pyfme. If you are using fmeobjects the function to use would be feature.getAttribute()
PythonCaller Example 3 (Class)Attached workspace: PythonCallerClass.fmw
Again we can use the sample provided in the Transformer description for the PythonCaller. This time we will use a class rather than a function to calculate the area of all features. Once again this is not an ideal example because this task could be done with standard FME transformers.
1. Start a new blank workspace and use a Creator to create a polygon feature. Add a PythonCaller transformer and a logger to see the output.
2. Open the PythonCaller and open the script in the code editor. Copy in the code from the example in transformer description or from below. You may need to add some line returns and indentation to the code so that looks like this (remember indents matter in python):
import fmeobjects class MyFeatureProcessor(object):
def __init__(self): self.featureList =  self.totalArea = 0.0 def input(self,feature): self.featureList.append(feature) self.totalArea += feature.getGeometry().getArea() def close(self): for feature in self.featureList: feature.setAttribute("total_area", self.totalArea) self.pyoutput(feature)
Again we are using a number of FME Objects methods so we need to import fmeobjects. On the input method we add each feature to a list and get the area from each feature which is added to the totalArea. On the close method we loop through each feature and add the total_area attribute to each one. It is important to note that if we want features to continue through the workspace they must be written out using the pyoutput() method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the Class MyFeatureProcessor. Because we have added a new attribute called total_area we can expose it by entering its name in Attributes to Expose.
4. Run the workspace to ensure you have the total_area attribute on your feature. Open the Creator transformer and increase the number of features. Run the workspace again to see that the total area of all features is being calculated.
Shutdown Python ScriptA python script can be added to a workspace which will run when the workspace completes. For example you may want to copy files somewhere when a workspace is done, zip data, or send an email. The script runs after all of the transformers reader and writers in a workspace are finished but the script is still run a part of the FME process. You can access the FME parameter values in the same way as in the examples above using the FME_MacroValues dictionary. A shutdown python script can also access a number of global variables which FME sets such as FME_FeaturesRead, FME_FeaturesWritten and many more. For a complete list of the global variables please see the FME Help under the topic FME_END_PYTHON.
Example of Shutdown Python ScriptAttached workspace: ShutdownPython.fmw
In this example we add a shutdown python script to a workspace which sends an email using gmail when the workspace finishes. You can edit the script to use other mail servers is you wish. The subject of the email will say whether the workspace was successful or not and include the number of features written. A similar example can also be found in FME Workbench -> Help -> Startup and Shutdown Python Scripts.
1. Open any workspace and in the Workbench Navigator pane under Workbench parameters - >Advanced you will see a parameter called Shutdown Python Script. Here you can enter the Python code below which will can send the email using a gmail account. The comments in the code below signified with # explain most of the script. Notice two key points:
a) We are getting the user parameters from the FME_MacroValues dictionary in the same way as in the other scripts above. For example the gmail user name can be set when the workspace is run and comes from the published parameter like this: FME_MacroValues['GmailUser'].
b) Secondly, notice we use two global variables from FME: FME_FeaturesWritten which is a python dictionary, and FME_Status which is a boolean and tells us if the workspace succeed or failed.
# The smtplib gives us the emailing tools import smtplib, fmeobjects # Get the FME parameter values set when workspace run to = FME_MacroValues['EmailTo'] gmail_user = FME_MacroValues['GmailUser'] gmail_pwd = FME_MacroValues['Password'] Subject = FME_MacroValues['SuccessSubject'] FailSubject = FME_MacroValues['FailSubject'] Emailfrom = FME_MacroValues['EmailFrom'] #Get number of features FME wrote FeaturesWritten = str(FME_FeaturesWritten) # If FME Fails then Change Subject # FME_Status tells if FME was successful status = FME_Status if status == 0: Subject = FailSubject Message = 'Workspace Failed' else: Message = 'Workspace Successful the following features were written: ' + FeaturesWritten # Sending the email # Using the gmail smtp server #You can edit this line to use your own email server smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(gmail_user, gmail_pwd) header = 'To:' + to + '\n' + 'From: ' + Emailfrom + '\n' + 'Subject:' + Subject +'\n' print header msg = header + '\n' + Message + '\n\n' smtpserver.sendmail(gmail_user, to, msg) print 'done!' smtpserver.close()
2. Run the workspace using File - Prompt and Run or the prompt and run button. Fill in the appropriate parameters. You will need your own gmail account user name (email address) and password and a from email address which should be the same as the user. You can also fill in the to email address and a subject line for failure and success. In the attached sample workspace the Fail parameter is used to simulate workspace failure by send the features to the Terminator transformer.
For a more complex shutdown python examples see the articles:
Notify an FME Server Topic on Success or Failure of a Workspace
Example Workflow using FME, Python and Oracle
Suggested Similar Articles
In FME 2012, Safe introduces a new Python FME Objects API which replaces the old pyfme.
Using startup and shutdown scripts in FME with Python and Tcl
The purpose of this article is to provide steps to license Schneider Electric (was Telvent or Miner&Miner) ArcFM from within FME so that FME can write to ArcFM extended Geodatabase feature classes.
Oliver’s Python Corner was a tutorial-style article accompanied by sample pyfme Python scripts which was a great resource in FME 2011 and older.