Questions in topic: "precision"
https://knowledge.safe.com/questions/topics/single/44512.html
The latest questions for the topic "precision"Converting component 'x' from type 'Real64' to 'Int32' - should I care?
https://knowledge.safe.com/questions/102882/converting-component-x-from-type-real64-to-int32-s.html
<div class="fr-view clearfix"><p>I'm reading and writing LAS files. There are warnings in the log files about X,Y,Z and being converted from Real64 to int32. How do I tell (with confidence) if this is important? Are the numbers being changed in a way that meaningfully changes the result?</p><pre>LAS writer: Converting component 'x' from type 'Real64' to 'Int32'. If this conversion is undesired, consider changing the type explicitly
LAS writer: Converting component 'y' from type 'Real64' to 'Int32'. If this conversion is undesired, consider changing the type explicitly
LAS writer: Converting component 'z' from type 'Real64' to 'Int32'. If this conversion is undesired, consider changing the type explicitly</pre><p><a rel="noopener noreferrer" href="https://knowledge.safe.com/questions/47749/combine-tilling-and-compress-las-files-without-con.html" target="_blank">https://knowledge.safe.com/questions/47749/combine-tilling-and-compress-las-files-without-con.html</a> has a recipe for counter-acting the conversion, but it's not clear to me if that is needed. That discussion links to another page that says only int32 supported in the output format. If that's the case then why go through the extra work? </p><p>I guess the question behind the question is: why is FME using Real64 at all if it's not supported?</p><p>ArcMap considers the points identical.</p><p><em>ArcMap before and after overlay of a LAS point. View scale 1:0.13. Distance from center to exterior of symbol is 0.006 meters. No measurable difference in point position.<br /></em></p><p><em><img src="/storage/attachments/32001-1574273998482.png" style="width:378px" /></em></p></div>precisionWed, 20 Nov 2019 18:25:29 GMTmaphewPrecision Code
https://knowledge.safe.com/questions/101295/precision-code.html
<div class="fr-view clearfix"><p>Hello, I am looking for a way to generate a precision code for GPS coordinates of parcel fabric. Is there any way to generate a code for the accuracy of coordinates like a range of number for example 1-7 where 1 signifies most accurate while 7 signifies least accurate.. I am re projecting my data from to LL84. Any suggestion?</p></div>transformerscoordinate systemsprecisionFri, 25 Oct 2019 18:08:13 GMTvakanshaForce coordinates and cell sizes to be integer
https://knowledge.safe.com/questions/95068/force-coordinates-and-cell-sizes-to-be-integer.html
<div class="fr-view clearfix"><p>I have a workflow with some raster resampling and also some raster coercing (back to point data) and I end up having coordinates like this:</p><p><img src="/storage/attachments/28419-bildschirmfoto-2019-07-09-um-134406.png" /></p><p><br /></p><p>Also, when I want to write a raster (using Aircom Enterprise writer) I get this warning.</p><pre>Enterprise Plugin: Invalid raster cell size: 99.99999999888921 by 99.99999999339829</pre><p>All my input coordinates and raster cell sizes are Integer numbers, so there must be some internal stuff going on that makes these numbers like this. I found this article: <a rel="noopener noreferrer" href="https://www.safe.com/blog/2018/09/fme2018-tolerance-evangelist179/" target="_blank">https://www.safe.com/blog/2018/09/fme2018-tolerance-evangelist179/</a></p><p>But I couldn't find out how to makes all coordinates integer. Any idea how this could be achieved?</p></div>rasterprecisionmath64 bitinternalTue, 09 Jul 2019 11:48:56 GMToiramDefine json precision?
https://knowledge.safe.com/questions/58039/define-json-precision.html
<p>Hello,</p><p>I am wondering if there is a way to define precision for json files generated with the ESRIJSON writer. I am reading from a 10.1 ESRI File Geodatabase, writing to ESRIJSON and get 4 digits after the decimal, where I'm looking for 10. The coordinate system of the reader is web mercator (meter) - I know 10 digits is unnecessary but am trying to match the format of other files. Using the ArcGIS geoprocessing tool 'Features to json' gives me the 10 digit precision.</p><p>FME 2012, ArcGIS 10.1 SP1.</p><p>Any help would be appreciated.</p><p>Thanks</p><p></p>jsonprecisionesri file geodatabaseWed, 15 Nov 2017 21:15:04 GMTtimboberooskyWhy Do Numbers Change From Float to String?
https://knowledge.safe.com/articles/54633/why-do-numbers-change-from-float-to-string.html
<h1>Introduction</h1><p>With each new release in FME, we are working hard to handle the rounding errors inherent in binary floating point conversions, which is considered a loss in precision when you are expecting 0.2000000000000000000000 but end up with 0.2000000000000000111022 when there is a float-to-string conversion or vice versa. These conversions occur while using various transformers. This is because some transformers have a specific data type assigned to them, so at some point throughout the translation, the data will go through a “Float To String” or a “String to Float” conversion in the background. There are situations where this conversion cannot happen without the user being very aware of it. </p><p></p><h1>Within FME</h1><p>Within FME for a float-to-string example, when the numbers are logged or displayed, they are not normally displayed in hex but are converted to strings. For a string-to-float example, the user is asked to type numbers as parameters in transformers, they are not asked for hex numbers but instead strings. Inside FME, these strings are then converted to an <a target="_blank" href="https://en.wikipedia.org/wiki/IEEE_754">IEEE 754</a> floating point number for mathematical processing. Think of a string as a number as a decimal (base 10) value, and a float is a number as a binary (base 2) value. </p><p> </p><p>FME does, however, work very hard ensuring that round-trip values are the same. So the input of float-to-string-to-float or string-to-float-to-string values is identical to the output value. This means that transformers can take a value, convert it for use within the transformer and then output it back in the original format.</p><p> </p><h1>The Mathematics Behind This</h1><h2>String to Float</h2><p>Strings can be stored as an array of characters, but if the string represents a number, it can not be used for calculations, it must be converted into a 64-bit binary floating point number or a decimal floating point number. If you require precision, decimal floating point systems is the way to go, but if you require fast performance you should use 64-bit binary floating point systems in FME. </p><p>Commonly Strings are converted into Double data types which are 64 bits. 1 bit is for the sign (positive or negative), 11 bits are for the exponent, and 52 bits are for the mantissa (significand). All math values are done with binary values, with only 52 bits for a number, not all numbers can be converted precisely.</p><p> </p><p>Consider two cases: the values "0.25" and "0.2"</p><p> </p><p>"0.25" can be stored as a Double precisely:</p><p><img src="/storage/attachments/11617-025-precision.png" style="width: 304px;"></p><p><em>The denominator of the fraction is a power of 2</em></p><p>binary: 0 01111111101 0000000000000000000000000000000000000000000000000000</p><p>hex: 0x3FD0000000000000</p><p>decimal: 0.25</p><p> </p><p>"0.2" looks simple, but cannot be stored as a Double precisely. Here are the Double values that it lies between:</p><p><img src="/storage/attachments/11618-02-precision.png" style="width: 165px;"></p><p><em>The denominator of the fraction is a prime number, which is not a power of 2</em></p><p>binary: 0 01111111100 1001100110011001100110011001100110011001100110011001</p><p>hex: 0x3FC9999999999999</p><p>decimal (rounded off): .199999999999999983346654630623...</p><p> </p><p>binary: 0 01111111100 1001100110011001100110011001100110011001100110011010</p><p>hex: 0x3FC999999999999A</p><p>decimal (rounded off): .200000000000000011102230246252...</p><p> </p><p>In this case, the mathematically closest value is the second one, 0x3FC999999999999A.</p><p> </p><p></p><p>The data is changing, even with a simple number like “0.2” when converting from String to Double. Those numbers at the end are not random, but the true value that the number has changed into when stored as a Double. The decimal is rounded, but it is the computer rounding the binary which is resulting in the change in the decimal number when it is converted back. </p><p></p><h2>Float to String</h2><p>The same point is in reverse. If we have a Double value of 0x3FC999999999999A, this can only be exactly represented as a String (base 10) if we use an infinite number of digits. Once we decide to limit the number of digits, we are CHANGING the value. Limiting to 17 digits is conventionally used, as it has the essential property of giving each unique IEEE floating point value its own unique string representation. If you use less than 17 digits in the string you cannot guarantee exact round-tripping back to Double. You also cannot guarantee comparisons (like equality) with the Strings. They will mirror the same comparisons with the Double values.</p><p><br></p><p> As of FME 2017.1, FME will use the fewest number of decimal digits needed to make sure that if the value goes back to a Double afterwards the value is the original Double. This means that there are cases where FME will use less than 17 digits of decimal precision. However, FME will never use more than 17 digits, for reasons previously stated.</p><p></p><h1>Conclusion</h1><p>Even though the loss of precision is improving with each FME release, some precision loss might still occur. It is best to be aware of it, and know why it is occurring. Running all of your math calcuations through an AttributeRounder transformer or the using the @round function is a good way to avoid this. </p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>precisionTue, 03 Oct 2017 17:46:24 GMTlizsandersonValidating dwg coordinate precision
https://knowledge.safe.com/questions/47951/validating-dwg-coordinate-precision.html
<p>Might be a long shot, particularly as I know there is an issue with binary storage of numbers but here goes - is there a way to check that a dwg file's coordinate system has at least 3 decimal places?</p><p></p>autocad dwgcoordinatesdecimalprecisionFri, 14 Jul 2017 04:50:11 GMTchrisw84Point cloud writer support precision control
https://knowledge.safe.com/idea/44514/point-cloud-writer-support-precision-control.html
<p>FME need to be able to set precision control when writing point cloud data (xyz). </p><p>Without precision control:</p><p>435263.4 6036064.7 15.08<br>435263.6 6036064.7 15.08<br>435263.2 6036064.9 15.08<br>435263.4 6036064.9 15.08<br>435263.6 6036064.9 15.08<br>435263 6036065.1 15.08<br>435263.2 6036065.1 15.08</p><p>Using precision control with 2 decimals:</p><p>435263.40 6036064.70 15.08<br>435263.60 6036064.70 15.08<br>435263.20 6036064.90 15.08<br>435263.40 6036064.90 15.08<br>435263.60 6036064.90 15.08<br>435263.00 6036065.10 15.08<br>435263.20 6036065.10 15.08</p><p>Since the point cloud writer is converting numeric values to string, one way to achivie this would be to control number of characters for X Y and Z. </p><p>For example, easting value would be 6 integers followed be 2 decimals and 1 decimal point would correspond to 9 characters. So one string statement could be; If less than 2 then RightPad else leave as is..</p><p></p><p>Best regards</p><p>André</p>point cloudwritingprecisioncontrolWed, 17 May 2017 06:51:36 GMTandreAdding more decimal places to geodatabase feature class.
https://knowledge.safe.com/questions/43467/adding-more-decimal-places-to-geodatabase-feature.html
<p>Currently, the geodatabase is rounding to six decimal places and I need the latitude and longitude decimal places to go to 10 places behind the decimal as shown in the original data. Is this possible? The geodatabase writer that I am using is uses dynamic properties. </p><p></p>geodatabaseprecisionThu, 27 Apr 2017 23:04:43 GMTshanezentner