• EN
Solved

Optimising use of RasterCellCoercer

  • 26 November 2014
  • 10 replies
  • 16 views
J
Badge
Hi All,

 

    I've been tasked with optimising a workbench that's quite slow.

 

    Basically I take in a very large DEM raster 2m granularity, and need to clip it against a vector dataset of 50m grid squares. The vector dataset then pulls the underlying raster data (RasterCellCoercer), and calculates statistics (StatisticsCalculator) using a group-by from the clipping.

 

    

 

    The clipping part I've got down to about 30minutes.

 

    

 

    The problem is the RasterCellCoercer. That part takes 15hrs! It creates about 300,000,000 points (yes, 300 million!) from the raster (there are about 4.5 Billion points in the source raster, so the other 4.2billion are no "noData" or were culled completely). I can't think of a way to get the number lower because that's how many the features require.

 

    

 

    Any suggestions for a faster way to get the DEM values in a way that they can be grouped for a statisticsCalculator pass?

 

    Would somehow trying to coerce it into a PointCloud be faster? (And if so, how?).

 

    

 

    Thanks,

 

    Jonathan
icon

Best answer by fmelizard 4 June 2018, 22:15

View original

10 replies

owen
Badge +1
Hi, me again. cant you use the RasterBandMinMaxExtractor and/or RasterPropertiesExtractor? Would be speedy if it tells you what you need.

 

 

Or might the Tiler be quicker than the Clipper?
J
Badge

Hi Owen,

 

Thanks for the thoughts. Alas the two extractors names don't tell me what I need. I need things like count and stand deviation as well as max/min. Cunning plan though.

 

 

I did consider tiler, but in theory Clipper should be quicker; that part of the process is only about 30mins. It's the RasterCellCoercer that's slow, and if I were to use the tiler, that would have many more cells to coerce!

 

Ah well, I guess this is just a "big data" problem and thus inherently going to be slow.

Thanks though!

Jonathan

david_r
Rank
Userlevel 5
Hi,

 

 

I've been in the exact same situation. Unfortunately, the RasterCellCoercer is rather slow because it has to create all those features. Feature creation is rather expensive in FME (case in point: ListExploder), performance wise.

 

 

In the end, I ended up reading the raster using the GDAL libraries in a Python script. There you can access all the cell values numerically in using arrays and it's really extremely fast compared to the RasterCellCoercer. Unfortunately, it does means quite a bit of Python coding...

 

 

You should write Safe and tell them you want to be added to my feature request for better raster support in the fmeobjects API, that would make it a lot easier :-)

 

 

David
sunsilk11
Rank
Badge +6

Hi I am also looking to generate points from raster using RasterCellCoercer. It takes about 8 hours to generate point for 1 tile, and I have about 1500 tiles. Is there any alternative option available in FME 2016.

Regarding the GDAL libraries, are there any sample code I could start with?

Thanks

david_r
Rank
Userlevel 5

Hi I am also looking to generate points from raster using RasterCellCoercer. It takes about 8 hours to generate point for 1 tile, and I have about 1500 tiles. Is there any alternative option available in FME 2016.

Regarding the GDAL libraries, are there any sample code I could start with?

Thanks

Remember to upvote my suggestion to give it more visibility: https://knowledge.safe.com/content/idea/19483/raster-cell-support-in-fmeobjects.html

Regarding GDAL, it will depend entirely on which format you're reading. Google is your friend, there is quite a lot of info out there.

jdh
Rank
Badge +22

Coercing to point cloud is significantly faster.

See http://knowledge.safe.com/content/idea/22245/spatial-statistics-on-raster-data.html for a workflow.

fmelizard
Rank
Userlevel 4
Badge +13

Coercing to point cloud is significantly faster.

See http://knowledge.safe.com/content/idea/22245/spatial-statistics-on-raster-data.html for a workflow.

Definitely try going to point cloud. Then manipulate the points using the PointCloud transformers. Coerce to points only at the end. Drastic speed improvements for large amounts of data.

sunsilk11
Rank
Badge +6

Hi, I am reading 2 ASCI files convert these to rasters and clip each to a polygon feature class, then generate points using RasterCellCoercer so that I end up with a point feature class with DTM and DSM height values for each point. This process is taking far too long. See attached workbench.

Any suggestion how the performance can be improved?

jdh
Rank
Badge +22

Hi, I am reading 2 ASCI files convert these to rasters and clip each to a polygon feature class, then generate points using RasterCellCoercer so that I end up with a point feature class with DTM and DSM height values for each point. This process is taking far too long. See attached workbench.

Any suggestion how the performance can be improved?

I suggest you start a new thread for this question

fmelizard
Rank
Userlevel 4
Badge +13

The RasterCellCoercer supports feature tables in FME 2018 builds 18256+, meaning that its performance has been improved and it processes faster now.

Reply


Community Stats

28,866
Posts
109,982
Replies
36,671
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.