Hello I'm running an FME workbench that will basically clip the Oracle Spatial data and transform into DWG format. I've used clipper to clip the larger boundary to smaller areas and running the transformation on individual smaller areas. FME Desktop 2016 is throwing Insufficient memory available when i'm running on 32GB RAM and 64 bit PC. I'm happy to attach the FME workbench if needed. Any help would be much appreciated. please find the attached log
FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.
Userlevel 5
+26
- Influencer
- 3357 replies
-
8 November 2018
When clipping large amounts of data you can set the Clipper type to "Clippers First", if you then make sure the clipper features all enter before the clippee features it'll use less memory.
But without seeing the workspace it's hard to come up with any specific suggestions. Any chance you can share it with us?
Hello @redgeographics, yes, i'm aware of it. I've set the Clipper type to "Clippers First. Still no luck, i've attached the workbench let me know if you need any more details. Appreciate your help on this request.FME_WORKBENCH.zip I believe this issue because of more data coming in via the Spatial Database for a specific grid
Userlevel 1
+21
- Contributor
- 3079 replies
-
8 November 2018
If you have spatial indexes on your Oracle tables you are likely to see better performance by batching with a workspace runner and only reading in the features within your search envelope (defined by your grid) each time
Userlevel 4
+25
- Safer
- 2399 replies
-
8 November 2018
Well it shouldn't fail (obviously). The log shows enough memory and even if there isn't, the temporary disk has a lot of space on it.
I don't think it's related, but I do notice you have a separate reader for each Oracle table. Is there a reason for that? Can you have one Oracle reader that reads multiple tables? It may help performance, even if it's not the root cause of your error.
One question is are you using Feature Caching? Because you have 115 transformers, so caching would create 100+ copies of the full data. That would put a strain on the system for sure. So turn caching off it you are using it.
I do notice you have both a feature type and a dataset fanout set. This can cause issues because of the way those work (dataset fanouts I believe can be particularly heavy on memory use). I have two suggestions there. Firstly replace the Writer with a FeatureWriter transformer. It operates in a different way and so it might not consume the same resources. Secondly, add a FeatureHolder just before the writer, to ensure all transformation is over and now using up memory before the writer starts. As a final alternative, write to an FFS file with no fanout. Then in a separate workspace read the FFS file and write it to AutoCAD with a fanout. I hate to suggest that, but it might be worth trying just to see if it helps. If it does, then at least we have an idea of where the issue is and what is causing it.
The other thought is that I have seen this error occur because of a faulty or corrupt geometry. The error occurs just when FME is reading feature 965000. So you could try running the workspace, starting at feature 965000 (set it in the reader parameters). Of course, this is less easy because you have separate readers per table. One other option might be to read one table at a time (disable the others) and see what happens. If one table fails by itself, then that would be somewhere to look for bad geometry. If no tables fail individually, then we know it's more likely to really be a memory issue.
Anyway, I hope that helps. There's a lot of experimentation in there I'm afraid. But I'm confident that we can help find where the problem occurs and implement a workaround.
Well it shouldn't fail (obviously). The log shows enough memory and even if there isn't, the temporary disk has a lot of space on it.
I don't think it's related, but I do notice you have a separate reader for each Oracle table. Is there a reason for that? Can you have one Oracle reader that reads multiple tables? It may help performance, even if it's not the root cause of your error.
One question is are you using Feature Caching? Because you have 115 transformers, so caching would create 100+ copies of the full data. That would put a strain on the system for sure. So turn caching off it you are using it.
I do notice you have both a feature type and a dataset fanout set. This can cause issues because of the way those work (dataset fanouts I believe can be particularly heavy on memory use). I have two suggestions there. Firstly replace the Writer with a FeatureWriter transformer. It operates in a different way and so it might not consume the same resources. Secondly, add a FeatureHolder just before the writer, to ensure all transformation is over and now using up memory before the writer starts. As a final alternative, write to an FFS file with no fanout. Then in a separate workspace read the FFS file and write it to AutoCAD with a fanout. I hate to suggest that, but it might be worth trying just to see if it helps. If it does, then at least we have an idea of where the issue is and what is causing it.
The other thought is that I have seen this error occur because of a faulty or corrupt geometry. The error occurs just when FME is reading feature 965000. So you could try running the workspace, starting at feature 965000 (set it in the reader parameters). Of course, this is less easy because you have separate readers per table. One other option might be to read one table at a time (disable the others) and see what happens. If one table fails by itself, then that would be somewhere to look for bad geometry. If no tables fail individually, then we know it's more likely to really be a memory issue.
Anyway, I hope that helps. There's a lot of experimentation in there I'm afraid. But I'm confident that we can help find where the problem occurs and implement a workaround.
Hi @Mark2AtSafe No, I'm not using Feature Caching. Will try your options and will come back with the proper root cause. Thank you so much for your time in going through the workbench and providing best possible directions. It definitely helps me, i'll give a try.
Regards
Anil Kumar K B
Well it shouldn't fail (obviously). The log shows enough memory and even if there isn't, the temporary disk has a lot of space on it.
I don't think it's related, but I do notice you have a separate reader for each Oracle table. Is there a reason for that? Can you have one Oracle reader that reads multiple tables? It may help performance, even if it's not the root cause of your error.
One question is are you using Feature Caching? Because you have 115 transformers, so caching would create 100+ copies of the full data. That would put a strain on the system for sure. So turn caching off it you are using it.
I do notice you have both a feature type and a dataset fanout set. This can cause issues because of the way those work (dataset fanouts I believe can be particularly heavy on memory use). I have two suggestions there. Firstly replace the Writer with a FeatureWriter transformer. It operates in a different way and so it might not consume the same resources. Secondly, add a FeatureHolder just before the writer, to ensure all transformation is over and now using up memory before the writer starts. As a final alternative, write to an FFS file with no fanout. Then in a separate workspace read the FFS file and write it to AutoCAD with a fanout. I hate to suggest that, but it might be worth trying just to see if it helps. If it does, then at least we have an idea of where the issue is and what is causing it.
The other thought is that I have seen this error occur because of a faulty or corrupt geometry. The error occurs just when FME is reading feature 965000. So you could try running the workspace, starting at feature 965000 (set it in the reader parameters). Of course, this is less easy because you have separate readers per table. One other option might be to read one table at a time (disable the others) and see what happens. If one table fails by itself, then that would be somewhere to look for bad geometry. If no tables fail individually, then we know it's more likely to really be a memory issue.
Anyway, I hope that helps. There's a lot of experimentation in there I'm afraid. But I'm confident that we can help find where the problem occurs and implement a workaround.
Hello @Mark2AtSafe, I'm using one of option of adding FeatureHolder just before the writer. Solution seems to be working, but DWG transformation is taking long time to process from FeatureHolder to AUTOCAD DWG writer. 60 Tiles have been process from past 3 hours is there a way to make the process quicker after everything is available in FeatureHolder transformer.
Any help would be much appreciated.
Hello @redgeographics, yes, i'm aware of it. I've set the Clipper type to "Clippers First. Still no luck, i've attached the workbench let me know if you need any more details. Appreciate your help on this request.FME_WORKBENCH.zip I believe this issue because of more data coming in via the Spatial Database for a specific grid
Hello @redgeographics hope you had a time to look at the workbench, any feedback would be much appreciated.
Userlevel 4
+25
- Safer
- 2399 replies
-
13 November 2018
Hello @Mark2AtSafe, I'm using one of option of adding FeatureHolder just before the writer. Solution seems to be working, but DWG transformation is taking long time to process from FeatureHolder to AUTOCAD DWG writer. 60 Tiles have been process from past 3 hours is there a way to make the process quicker after everything is available in FeatureHolder transformer.
Any help would be much appreciated.
I think the first thing to try is the FeatureWriter transformer, in place of a writer. That may well speed things up. In fact, once you have it in place I would try it both with and without the FeatureHolder. I expect that would make things quicker.
+21
- Contributor
- 955 replies
-
13 November 2018
Hm, have experienced something similar when writing DWG-files sometimes. I`ll try t write all possible things to try that I can think of. Threat them as individual tips and suggestions. Sorry if they are difficult to understand, feel free to ask more if they are a bit vague
- Large geometries: This often happens when working with larger geometries which are 3D.
- Disable writers: To see if it is due to the writer. Disable all Writers in the Navigator pane - and then try to run the workspace. If it succeeedes without issues, there is something happening during writing.
- Two DWG-writers: Remember to try both the DWG - writers (there are two different) and perhaps some older versions and settings of those writers also.
- Corrupt geometry: I suspect that some geometry from a reader is corrupt. Or it gets corrupt - or too large - when writing to DWG. Try to add the GeometryValidator right before the DWG-writer to see if that helps.
- Geometryfilter: You can also try to use a GeometryFilter right before the DWG-writer and first try to write only points, then only lines etc. And see if there is certain kind of geometry that gets "stuck" in the writer.
- Reproject to correct coordinate system before writer: Try to reproject to the correct coordinate system that you want to have in the DWG-file - BEFORE - the writer using a REPROJECTOR - as I suspect that some readers sometimes reproject anyway.
- Need to wrap-up all geometries: When writing to for instance SKETCHUP (or another format, dont remember right now) it seems that FME needs to wrap all the geometry into one master-geometry before writing. So that it needs to have all the geometry in memory before starting to write. Even when using a FeatureWriter it will not start to write data untill at the end. To check this. Figure out WHEN your writer starts to write. And see if the outputfilesize increases every 10seconds, or if it stays at 0 kb for a long time. Typically when it needs to wait for "wrapping all geometries" it stays for 0 kb for 40 minutes, and then writes the whole chunk and the whole file within 2 minutes after that.
Good luck - and let us know how it goes!
+1
Hello FME Users,
even Adjusting Memory Resources .
I am facing termination while running
FME Workbench 2019.2.3.2 , 64 bit, AWS
“FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.“
Following are the system configurations and done 'Adjusting Memory’
RAM: 64 GB
FME_TEMP Folder allocated: 1 terabyte
Can any once help me how can I solve issue attached log file as well
Regards
Amol
+21
- Contributor
- 955 replies
-
20 January 2021
Hello FME Users,
even Adjusting Memory Resources .
I am facing termination while running
FME Workbench 2019.2.3.2 , 64 bit, AWS
“FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.“
Following are the system configurations and done 'Adjusting Memory’
RAM: 64 GB
FME_TEMP Folder allocated: 1 terabyte
Can any once help me how can I solve issue attached log file as well
Regards
Amol
Hi! I would suggest to improve the workspace.
Have a look at the Featuremerger and make sure you can change to "Suppliers first: Yes) so all suppliers enters first.
Before the AreaGapAndOverlapCleaner - make sure to use a GeometryValidator to remove any bad geometry.
Difficult to see which other improvements to suggest since the workspace is not available.
+1
Hi! I would suggest to improve the workspace.
Have a look at the Featuremerger and make sure you can change to "Suppliers first: Yes) so all suppliers enters first.
Before the AreaGapAndOverlapCleaner - make sure to use a GeometryValidator to remove any bad geometry.
Difficult to see which other improvements to suggest since the workspace is not available.
@Sigbjørn Herstad First Of All thank you for help , i trid with "Suppliers first: Yes setting still facing same error. @mark2atsafe does any one has any other solution ?
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.