Idea by pm · Nov 07, 2016 at 05:53 PM · python cloud ssl

Allow Different FME Cloud Python Interpreter

Similar to https://knowledge.safe.com/content/idea/19407/update-to-python-3x.html

FME Cloud currently loads python v. 2.7.6, however quite a few use cases exist that require 2.7.9+ (or even better 3.x - but that appears to be on the roadmap). Our issue involves SNI limitations with the urllib library that are hampering our ability to connect to SSL enabled REST endpoints. It would be great to have more control over the interpreter in the cloud similar to what we expect in Desktop and Server versions.

Thanks!

Add comment

10 |4000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 4.0 MB each and 4.0 MB total.

sigtill commented · Apr 30, 2018 at 08:19 AM

It seems that when you use parallell processing on a custom transformer it fires up:

FME Configuration: Command line arguments are `C:\Program Files\FMEServer\Server\fme\fmeworker' `d:\FMETEMP\fmeengines\localhost_Engine4\childProcMap1525074270579_31724.fme' `-WORKER_CAPABILITY' `247L1228U1' `LOG_FILENAME' `D:\FMEServerSystemShare\resources\logs\engine\current\jobs\2000\job_2228_ffs\job_2228_7.log' `-LOG_PREPEND_ID' `7' `--SOURCE_PORT_OUT' `54529' `--SOURCE_PORT_IN' `54528'

However the fmeworker.exe process does not inherit the python-interpreter. It starts the default 2.7 :(

Add comment

10 |4000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 4.0 MB each and 4.0 MB total.

GerhardAtSafe commented · Apr 03, 2018 at 03:17 PM

With FME Server 2018.0 on FME Cloud, you could also create 2 different Job Queues, one for Python 2.7 and one for Python 3.x and assign engines to both of them. If you route all jobs using a specific version of python to the same engine (e.g. single jobs, or a repository) you would not need to restart the engine. Also on FME Cloud, the pricing is not tied to the number of engines so usually it does not matter if you need a few more engines just for proper routing.

Add comment

10 |4000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 4.0 MB each and 4.0 MB total.

sigtill commented · Mar 22, 2018 at 01:04 PM

david_r : I wonder what happens if you have a Custom Transformer in Parallell mode - will it consider each running process a transaction or not, and if you have a jobsubmitter ... Well, thanks for the workaround.

Add comment

10 |4000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 4.0 MB each and 4.0 MB total.

stewartharper commented · Nov 08, 2016 at 10:26 PM

We just checked and Ubuntu 14.04, which is what FME Cloud runs on, does actually have Python 3.4 installed. You just have to get the FME Engine to use it. To do this you need to set it in FME Workbench before you upload it to FME Server. In FME Workbench, open up FME Options, click on the Translation section and set the preferred python interpreter to FME Python 3.4. Now publish this workspace to FME Cloud.

Things get a little bit tricky now. Engines only load the Python interpreter the first time they start up. The way the engine decides which interpreter to load is whichever interpreter is defined in the first workspace that runs on each engine.

So if you want to run multiple workspaces on the same FME Server and some of them use Python 2.7.6 and others 3.4 then you are going to have a problem. If all of your workspaces are 3.4 then just update the interpreter in all of them, restart the FME Server and publish them up there.

Note if you are using formats that are written in Python, they may not behave correctly in 2016 when using Python 3.4.

Add comment · Show 8

10 |4000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 4.0 MB each and 4.0 MB total.

sigtill · Mar 20, 2018 at 02:36 PM -1

@stewartharper / this is a large issue now. We have started to move some workspaces to 36 and we have some on 27 which cant be upgraded. I have contacted support - but I just wanted to add information on this also.

david_r sigtill · Mar 20, 2018 at 02:45 PM 1

I agree, we've stumbled upon this as well. Our workaround was to tell FME Server to restart each engine after each translation, but it's far from optimal.

sigtill david_r · Mar 20, 2018 at 10:27 PM 0

Hm, cant find the documentation for this, I know I have seen it somewhere. Do you remember?
I was thinking about that - but that WILL create a delay I assume?

Show more comments

stewartharper ♦♦ sigtill · Mar 22, 2018 at 03:00 AM 1

Glad to see you have an idea. As you alluded to in the idea, this dev work would need to be done on FME Server and then there shouldn't be any modifications required when we come to deploy on FME Cloud.

sigtill stewartharper ♦♦ · Apr 30, 2018 at 08:21 AM 0

This workaround does not work when a workspace failes within a Custom Transformer in Parallell mode (as it is the fmeworker.exe failing - not fme.exe) - so it will not restart these processes.

Allow Different FME Cloud Python Interpreter

4 Comments

Your Opinion Counts

IDEA DETAILS

Related Ideas