Question

Load balancing using WorkspaceRunner transformers


Userlevel 2
Badge +16

Hello FME users,

I hope somebody can help me with this situation:

I have a workspace that reads a list of feature ID's.

For each feature I want to run the F020 workspace (connecting to a database and creating a file).

To create 4 parallel processes, I split up the list and pass an ID to each of the 4 WorkspaceRunners (each using their own database connection).

The WorkspaceRunners are set to wait, but that results in all processes being run in sequence.

If I change the setting to "Wait = No", the file is not created in time for the next step in the process.

Any idea how I can run the 4 F020 WorkspaceRunners in parallel, but run the FileExistenceChecker for each process after the F020 has created its output file?

All suggestions are appreciated.

 


6 replies

Userlevel 1
Badge +21

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

Badge +22

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

Userlevel 2
Badge +16

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

Thanks @egomm I did vote for David's idea.

Userlevel 2
Badge +16

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

Thanks @jdh I think I will give that a try.

Userlevel 2
Badge +16

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

I implemented this suggestion and it works great!

Thanks @jdh

Badge +22

I implemented this suggestion and it works great!

Thanks @jdh

Your welcome.

 

Just make sure to keep the modulo under the number of concurrent licenses -1 (usually 7).

Reply