Interested in FME & Python? Checkout the recording of the June 2012 Python webinar on safe.com
Python is a programming language that can be used within FME to accomplish tasks either before or after FME runs or to perform tasks within FME which are not possible with standard FME tools and transformers*. For example you can use a python script to extract source data from a zip file before FME runs or to send an email after FME runs. In this article we will look at simple examples of python scripting in the 4 main places python is used in FME:
*Important Note: There are numerous FME transformers for performing almost every task you can think of without resorting to scripting. If you are thinking about writing a python script to do something in FME you should make absolutely sure there is not an existing transformer available already. Please feel free to ask us by contacting support at www.safe.com/support.
For learning Python, there are many Python books available from publishers such as O'Reilly, etc.
Three free resources are:
FME installs its own python interpreter which is used by any scripts running python in FME. If you want FME to use your own python interpreter there is a setting in Workbench. Please see this article for more information:
Much of FME's core functionality is available within python scripts by using FME classes and methods from the FME Objects python API. For example you can get at the FME log file, FME parameters and use any of the countless FME methods such as getArea or reproject etc. (more on this later). To use FME Objects in your python script with FME 2012 and newer you will need to import FME Objects with the statement: import fmeobjects For more information on the new FME Objects python API please see the article here:
Python-related transformers have excellent Help right in FME Workbench, which also includes Help for Startup and Shutdown scripts. For using FME Objects in Python, you can find complete documentation of the Python FME Objects API here: <FME>\fmeobjects\python\apidoc\index.html
A number of FME related variables are available in FME Startup and Shutdown scripts. As of FME 2014 these variables are accessible in the fme module so you will need to include the statement import fmeto access them. For example the fme.status variable can be true or false at the end of a translation depending on the success or failure of the workspace. For a complete list of FME variables available in Startup and Shutdown Python scripts please go to Help > Workspace Basics > User Interface Reference > Workspace Navigator > Workspace Parameters > Advanced > Startup and Shutdown Python Scripts.
Prior to FME 2014 these variables were named differently and it was not necessary to import the fme module. For example the status was returned in the variable FME_Status. For these older versions of FME the FME variables are documented in the FME Quick Translator Help.
import fme SourceDataset = fme.macroValues['SourceDataset_ACAD']Please note: if using an FME version older than FME 2014 please use the syntax below:
SourceDataset = FME_MacroValues['SourceDataset_ACAD']
We can also set FME parameters by creating a scripted parameter and returning the value from the script. This is covered below in the section on Scripted Parameters.
In the Workbench Navigator pane under Workbench Parameters - > Advanced you will see a parameter called Startup Python Script. Here you can enter some Python code that will execute before the FME translation.
Note: If you want to set or return an FME parameter this is not place - see Scripted Parameters below.
Attached workspace: StartupPython.fmw
By default when FME writes to an existing Microsoft Excel file we insert new records, but what if you want to delete the Excel file each time? (See the article http://fmepedia.safe.com/articles/How_To/Example-scripts-for-deleting-Excel-files-prior-to-writing)
We can delete the destination dataset before running FME using the following Start Up Python script (workspace attached):
OutputFile = str(FME_MacroValues['DestDataset_XLS_ADO'])FME World Tour 2013 - 1Spatial & Star-Apic (London and Leeds)
We are getting the value for the destination dataset parameter from the FME_MacroValues dictionary. Your destination dataset parameter may have another name and which you can find by right clicking on the parameter and selecting Edit User Parameter Definition where you can see the Name:
Scripted parameters are extremely useful when we want to set a parameter in FME based on something we derived or calculated from another parameter or parameters. For example you may want users to select themes or groups of layers and have your script set the the individual feature types to read within these groups.
featureTypes = '' if FME_MacroValues['layers'].find('Bus Information') != -1: featureTypes += 'BusRoutes BusStops ' if FME_MacroValues['layers'].find('Road and Rail') != -1: featureTypes += 'metrorail Roads ' if FME_MacroValues['layers'].find('All Transit') != -1: featureTypes += 'metrorail Roads BusRoutes BusStops Labels ' #Debug #print featureTypes return featureTypes
You can see that we have a series of if statements which find out which layer the user chose using the FME_MacroValues[ ] dictionary and then sets the value of featureTypes which will be returned. The last line is the most important as this is where we "return" the actual value of the scripted parameter with statement
This return statement must always exist in a python scripted parameter because it is here that the parameter value is given to FME. Also notice the commented out print statement. You can use print to help you debug your scripts by returning variable values to the FME log. For more in logging with python see this article: http://fmepedia.safe.com/articles/Error_Unexpected_Behavior/Logging-with-Python-scripts
If you understand the basic scripted parameter example above you should have no trouble understanding this more complex example. The attached workspace UnZipSource.fmw accepts a published parameter for a zip file and a parameter for a directory to unzip to. A scripted parameter unzips the file and returns the directory to the source dataset parameter of the reader.
The PythonCaller transformer lets you do things to features or groups of features using a Python script. Before using this transformer please be sure there is not already an FME transformer which does the task you want to do. Feel free to ask us in support (www.safe.com/support) if there is a transformer that can help you.
The PythonCaller can call a Function which you can use to process one feature at a time or a Class where you may want to do things to groups of features. In either case the EntryPoint parameter of the transformer is where you name the function or class in the script to run. The PythonCaller can use the full range of FME Objects including numerous methods and classes.
import fmeobjects import time # Template Function interface: def timestampFeature(feature): curTime = time.ctime(time.time()) feature.setAttribute("timestamp", curTime)
The script uses some FME objects methods so we need import fmeobjects and we are using the python time module so we need to import it as well. The function definition accepts FME feature as its only argument meaning all features will enter the function one by one for processing.
A new attribute is added to the feature with setAttribute method on the feature - this is actually an FME Objects method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the function timestampFeature. Because we have added a new attribute called timestamp we can expose it by entering its name in Attributes to Expose.
4. Use a creator to create some features to send you the PythonCaller and a Logger to look at the output. Every feature should have a timestamp attribute.
The FME Store includes a custom transformer called the FuzzyStringComparer which uses the Python difflib module to compare two string attributes and calculate a similarity ratio. In some ways this is a better example because it is something we cannot do with a regular FME transformer.
1. Open a blank workspace and add the FME Store transformer FuzzyStringComparer by typing it on the canvas or by browsing the FME Store transformers in the Tranformer gallery. Right-click on the transformer and select Embed which will allow you to edit the transformer and see its contents in the same workspace.
2. Once the FuzzyStringComparer is embedded right click on it to Edit and see the contents. Locate the PythonCaller transformer and edit its properties to access the script editor. Notice again we are using a function - this time to compare string attributes on a feature by feature basis. Again we are setting a new attribute which is called FuzzyStringCompare.ratio. An AttributeExposer is used later in the workflow to access the attribute we have created.
string1 = feature.getStringAttribute('FuzzyStringCompare.string1').lower()
string2 = feature.getStringAttribute('FuzzyStringCompare.string2').lower()
ratio = difflib.SequenceMatcher(None,string1,string2).ratio()
In the above example the function used to get an FME attribute value is from the now deprecated pyfme. If you are using fmeobjects the function to use would be feature.getAttribute()
import fmeobjects class MyFeatureProcessor(object):
def __init__(self): self.featureList =  self.totalArea = 0.0 def input(self,feature): self.featureList.append(feature) self.totalArea += feature.getGeometry().getArea() def close(self): for feature in self.featureList: feature.setAttribute("total_area", self.totalArea) self.pyoutput(feature)
Again we are using a number of FME Objects methods so we need to import fmeobjects. On the input method we add each feature to a list and get the area from each feature which is added to the totalArea. On the close method we loop through each feature and add the total_area attribute to each one. It is important to note that if we want features to continue through the workspace they must be written out using the pyoutput() method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the Class MyFeatureProcessor. Because we have added a new attribute called total_area we can expose it by entering its name in Attributes to Expose.
4. Run the workspace to ensure you have the total_area attribute on your feature. Open the Creator transformer and increase the number of features. Run the workspace again to see that the total area of all features is being calculated.
A python script can be added to a workspace which will run when the workspace completes. For example you may want to copy files somewhere when a workspace is done, zip data, or send an email. The script runs after all of the transformers reader and writers in a workspace are finished but the script is still run a part of the FME process. You can access the FME parameter values in the same way as in the examples above using the FME_MacroValues dictionary. A shutdown python script can also access a number of global variables which FME sets such as FME_FeaturesRead, FME_FeaturesWritten and many more. For a complete list of the global variables please see the FME Help under the topic FME_END_PYTHON.
#The smtplib gives us the emailing tools import smtplib, fmeobjects #Get the FME parameter values set when workspace run to = FME_MacroValues['EmailTo'] gmail_user = FME_MacroValues['GmailUser'] gmail_pwd = FME_MacroValues['Password'] Subject = FME_MacroValues['SuccessSubject'] FailSubject = FME_MacroValues['FailSubject'] Emailfrom = FME_MacroValues['EmailFrom'] #Get number of features FME wrote FeaturesWritten = str(FME_FeaturesWritten) #If FME Fails then Change Subject #FME_Status tells if FME was successful status = FME_Status if status == 0: Subject = FailSubject Message = 'Workspace Failed' else: Message = 'Workspace Successful the following features were written: ' + FeaturesWritten #Sending the email #Using the gmail smtp server #You can edit this line to use your own email server smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(gmail_user, gmail_pwd) h eader = 'To:' + to + '\n' + 'From: ' + Emailfrom + '\n' + 'Subject:' + Subject +'\n' print header msg = header + '\n' + Message + '\n\n' smtpserver.sendmail(gmail_user, to, msg) print 'done!' smtpserver.close()
2. Run the workspace using File - Prompt and Run or the prompt and run button. Fill in the appropriate parameters. You will need your own gmail account user name (email address) and password and a from email address which should be the same as the user. You can also fill in the to email address and a subject line for failure and success. In the attached sample workspace the Fail parameter is used to simulate workspace failure by send the features to the Terminator transformer.
For a more complex shutdown python examples see the articles:
Notify an FME Server Topic on Success or Failure of a Workspace
Example Workflow using FME, Python and Oracle
4 People are following this .