Questions in topic: "clustermodeller"
https://knowledge.safe.com/questions/topics/single/1257.html
The latest questions for the topic "clustermodeller"Clustering Points in FME (Cluster Size)
https://knowledge.safe.com/questions/73841/clustering-points-in-fme-cluster-size.html
<p>Hi All,</p><p>I have a total number of <em>n</em> (points) spatially divided over a given area. I want to group them into <em>x</em> clusters using <em>y</em> members per cluster. This has to be as spatially optimal as possible.</p><p>So I would like set the following parameters in the transformer:</p><p>- The number of clusters</p><p>- The size of the clusters or the min. and max. size of cluster</p><p>Anyone got any ideas? I already tried working with the cluster modeller, K-means & RClusterCalculatowith no success. </p><p>Thanks in advance!</p><p></p>cluster analysisclustermodellerclustersk-means cluster analysis fme python pythoncaller pointclusteringThu, 05 Jul 2018 18:21:26 GMTbishoymfDBSCAN Density point clustering
https://knowledge.safe.com/idea/59759/dbscan-density-point-clustering.html
<p>There are numerous implements of the various cluster algorithms within open source libraries including DBSCAN and Kmeans. An FME transformer for clustering would be really useful in trying to map more complex datasets. I know that people have used 3D terrains and distance matrices to do something similar but there doesn't appear to be any out of the box FME solutions and to create clustered areas in FME is quite a challenge. I guess python running scikit-learn would probably be the easiest way? But a transformer would be very useful and would help make FME more of a mapping tool.</p>mapnikrasterizerclustermodellerTue, 12 Dec 2017 10:32:53 GMTmarkoCluster or group n points based on location
https://knowledge.safe.com/questions/36955/cluster-or-group-n-points-based-on-location-1.html
<p>Hi,</p><p>I have 100,000+ points I would like to group into a fixed number of members as spatially optimal as possible.</p><p>The data comes delivered to me in either text or csv containing a variety of attributes including postcode. I have geo-referenced each row to the nominal centre of each postcode using OS Code Point. This gives me the easting and northing for each row. I have tried to convert this to an xyz to then follow the suggestions in this post <a target="_blank" href="https://knowledge.safe.com/questions/24064/cluster-points-based-on-location-k-means-method.html">https://knowledge.safe.com/questions/24064/cluster-points-based-on-location-k-means-method.html</a>. However, I suspect due to my limited working knowledge of FME, I'm struggling to create the correct workflow.</p><p>I have tried another approach in FME creating voronoi polygons from the points and then using PointOnAreaOverlayer to add the overlap/count of points in the polygons. I thought I could then look to dissolve the polygons based on nearest neighbour locations until the desired count is reached.</p><p>Can anyone provide any advice, tips or instructions on how to best come up with a solution?</p><p>Thanks for taking the time to look</p><p>B</p>point cloudneighborfinderclustermodellerSat, 26 Nov 2016 12:50:21 GMTbieahartCluster points based on location (K-MEANS method)
https://knowledge.safe.com/questions/24064/cluster-points-based-on-location-k-means-method.html
<p>Hi, </p><p>I am trying to do the following:</p><p>I have a total number of <em>n</em> adresses (points) spatially divided over a given area. </p><p>I want to group them into <em>x</em> clusters using <em>y</em> members per cluster. This has to be as spatially optimal as possible.</p><p>As far as I know, it should be something in the lines of a k-means algorithm.</p><p>Actually, the Neighbourhood Aggregator comes close, but I want to set a fixed number of members per cluster, not a minimal.</p><p>Anyone got any ideas? I already tried working with the cluster modeller, to no avail. </p><p>Thanks in advance!</p><p>Flip</p>neighborfinderclustermodellerWed, 02 Mar 2016 16:39:16 GMTflipvandervalkCluster or Density Modelling
https://knowledge.safe.com/articles/1258/cluster-or-density-modelling.html
<h1>Downloads</h1><p><a id="5153" href="/storage/attachments/5153-clustermodeller.zip">clustermodeller.zip</a></p><h1></h1><h1>Overview</h1><p>
This transformer arose because of a question on the <a target="_blank" href="https://groups.google.com/group/fmetalk?pli=1">FMETalk</a> user group.</p><p>
<em>I'm trying to render a hot spot map of crimes. Using other GIS systems, I can easily do a Kernel Density type analysis which will create contours of dense areas. I'd like to do this in FME, but am struggling! Any thoughts please? </em></p><p>
FME does have a <strong>DensityCalculator </strong>transformer, but this is a little different. Basically we need to calculate density (or clustering) by taking each point and assessing the density of similar points within the same area.</p><h2>Input</h2><p>
This transformer accepts any set of point features as inputs.</p><p></p><p><img src="/storage/attachments/5154-adresses.png"></p><p> <em><strong>Above</strong>: In my test case I used a set of address points. </em></p><p>
You can see where there are clusters of addresses, but can we write this to a format for better visualization?</p><h2>Output</h2><p>
Yes we can! The output from this transformer can be either a set of contours, or a surface feature. You could write this to many formats, including ones capable of true 3D visualization.</p><p>
<img alt="User-added image" src="https://cdn.safe.com/community/attachments/http_download_1444329978536_6032.png">
<em><strong>Above</strong>: 3D PDF output opened in Adobe Viewer. This is showing both surface and contours, though of course you could just use one or the other.</em></p><p>
The one issue about the output is that it can look fairly flat, even in the densest areas. To counter this, the Z values are all exaggerated by a factor of 10.</p><h1>Methodology</h1><p>
The method is very simple (<em>See attachment: </em><a target="_blank" href="https://knowledge.safe.com/storage/attachments/5153-clustermodeller.zip">ClusterModeller.zip</a>). I just find the number of neighbors within a given radius for each point, and use that value as the Z in a DEM/Surface Model. The trick is to find the best radius to give a good result. This transformer offers the user the ability to enter that value, or will calculate one automatically based on the average distance between the input points.</p><h2>Detailed Description</h2><p>
Here's the transformer as a whole. You can see how it is made up of two main parts.</p><p><img src="https://knowledge.safe.com/storage/attachments/5156-transformer.png"></p><p><em><strong>Above</strong>: The custom transformer definition.</em></p><p>
Let's take a closer look at the section in the blue <strong>bookmark</strong>, as this is where all the real work takes place.</p><p><img src="/storage/attachments/5158-bluebookmark.png"><em><strong>Above</strong>: The processing section.</em></p><p>
This starts out with a <strong>NeighborFinder</strong><strong> </strong>transformer. Rather than the normal Base/Candidate use, we're using the NeighborFinder in the candidate only mode. We set a maximum radius for finding neighbors (what I call 'tolerance') and set a list attribute to store the results.</p><p>
The <strong>ListElementCounter</strong><strong> </strong>is then used to find how many entries are in the list - i.e. how many neighbors cluster around this point, within the defined tolerance. Unmatched points (those with no neighbors) are set to 0.</p><p>
The <strong>3DForcer</strong><strong> </strong>transformer simply sets the Z coordinate of each point to this value, and the <strong>Scaler</strong><strong> </strong>multiplies it by 10 to emphasize the results.</p><p>
Finally, the SurfaceModeller turns the points and their Z values into a digital elevation model (DEM) and uses that to create a set of contours and a TIN surface.
The contours are simple line features compatible with almost any format. The surface is a true 3D feature, compatible with formats supporting 3D (PDF, Geodatabase, 3DS, etc).</p><p>
You might be asking, "what did the red section of workspace do?". That's where we calculate a tolerance if the user did not set one.</p><p><img src="/storage/attachments/5159-redbookmark.png"></p><p><em><strong>Above</strong>: The tolerance calculation section.</em></p><p>
Here the <strong>NeighborFinder</strong><strong> </strong>is again used, but this time we want to find the distance between each point and its nearest neighbor.</p><p>
Then a <strong>StatisticsCalculator</strong><strong> </strong>and an <strong>ExpressionEvaluator</strong><strong> </strong>are used to calculate a tolerance value based on the average distance between points.</p><p>
The <strong>FeatureMerger</strong><strong> </strong>then attaches that tolerance value back onto the original features.</p><h2>Limitations</h2><p>
There are some obvious limitations, but I should probably point them out:</p><ul><li>I've no idea if this method is really suitable for modelling Clusters. It's just something I thought of that seems to produce a good result.</li><li>One thing this definitely does not do is prove whether the clusters are statistically significant. They could just be random patterns.</li><li>The tolerance calculation is convenient when you aren't sure about your data, but again it's just a method which seems to produce an acceptable result. You could experiment with values to get a better result.</li></ul><h2>Updates</h2><p>
Since I took the screenshots above, I made the following updates:</p><ul><li>Get user input on the Z exaggeration factor.</li></ul>
<img alt="User-added image" src="https://cdn.safe.com/community/attachments/http_download_1444329979937_6032.png"><p></p><ul><li>Write to the log window a message about what tolerances are being calculated.</li></ul>
<img alt="User-added image" src="https://cdn.safe.com/community/attachments/http_download_1444329980126_6032.png"><p></p><ul><li>The <a target="_blank" href="https://hub.safe.com/transformers/clustermodeller">ClusterModeller</a> transformer is now available for download from the <a target="_blank" href="https://hub.safe.com/">FME Hub</a> here. The version on FME hub may be updated so be sure to check it out.</li></ul><h2>Future Updates</h2><p>
As this isn't for a particular project I don't really plan to make further updates. However I could....</p><ul><li>Calculate a contour interval in a similar way to the automatic tolerance calculation.</li><li>Calculate a statistical significance figure. I'm thinking...</li><li>Create the same number of points as the input, in the same X/Y extents, but in random locations.</li><li>Create a surface from the random locations.</li><li>Calculate the volume of the clustered surface minus the random surface</li><li>The closer the value is to zero, the less clustered the original data is.</li></ul>
<p></p><p></p>gisneighborfinderclustermodellerdensitycalculatorThu, 08 Oct 2015 19:15:56 GMTstewart.harper