• EN
Question

Load balancing using WorkspaceRunner transformers

erik_jan
Rank
Userlevel 2
Badge +16

Hello FME users,

I hope somebody can help me with this situation:

I have a workspace that reads a list of feature ID's.

For each feature I want to run the F020 workspace (connecting to a database and creating a file).

To create 4 parallel processes, I split up the list and pass an ID to each of the 4 WorkspaceRunners (each using their own database connection).

The WorkspaceRunners are set to wait, but that results in all processes being run in sequence.

If I change the setting to "Wait = No", the file is not created in time for the next step in the process.

Any idea how I can run the 4 F020 WorkspaceRunners in parallel, but run the FileExistenceChecker for each process after the F020 has created its output file?

All suggestions are appreciated.

 

6 replies

ebygomm
Rank
Userlevel 1
Badge +21

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

jdh
Rank
Badge +22

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

erik_jan
Rank
Userlevel 2
Badge +16

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

Thanks @egomm I did vote for David's idea.

erik_jan
Rank
Userlevel 2
Badge +16

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

Thanks @jdh I think I will give that a try.

erik_jan
Rank
Userlevel 2
Badge +16

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

I implemented this suggestion and it works great!

Thanks @jdh

jdh
Rank
Badge +22

I implemented this suggestion and it works great!

Thanks @jdh

Your welcome.

 

Just make sure to keep the modulo under the number of concurrent licenses -1 (usually 7).

Reply


Community Stats

28,868
Posts
109,994
Replies
36,681
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.