span8

span4

span8

span4

- Home /
- *FME Desktop /

- Home /

Article created with
FME Desktop - applicable to all versions

With each new release in FME, we are working hard to handle the rounding errors inherent in binary floating point conversions, which is considered a loss in precision when you are expecting 0.2000000000000000000000 but end up with 0.2000000000000000111022 when there is a float-to-string conversion or vice versa. These conversions occur while using various transformers. This is because some transformers have a specific data type assigned to them, so at some point throughout the translation, the data will go through a “Float To String” or a “String to Float” conversion in the background. There are situations where this conversion cannot happen without the user being very aware of it.

Within FME for a float-to-string example, when the numbers are logged or displayed, they are not normally displayed in hex but are converted to strings. For a string-to-float example, the user is asked to type numbers as parameters in transformers, they are not asked for hex numbers but instead strings. Inside FME, these strings are then converted to an IEEE 754 floating point number for mathematical processing. Think of a string as a number as a decimal (base 10) value, and a float is a number as a binary (base 2) value.

FME does, however, work very hard ensuring that round-trip values are the same. So the input of float-to-string-to-float or string-to-float-to-string values is identical to the output value. This means that transformers can take a value, convert it for use within the transformer and then output it back in the original format.

Strings can be stored as an array of characters, but if the string represents a number, it can not be used for calculations, it must be converted into a 64-bit binary floating point number or a decimal floating point number. If you require precision, decimal floating point systems is the way to go, but if you require fast performance you should use 64-bit binary floating point systems in FME.

Commonly Strings are converted into Double data types which are 64 bits. 1 bit is for the sign (positive or negative), 11 bits are for the exponent, and 52 bits are for the mantissa (significand). All math values are done with binary values, with only 52 bits for a number, not all numbers can be converted precisely.

Consider two cases: the values "0.25" and "0.2"

"0.25" can be stored as a Double precisely:

*The denominator of the fraction is a power of 2*

binary: 0 01111111101 0000000000000000000000000000000000000000000000000000

hex: 0x3FD0000000000000

decimal: 0.25

"0.2" looks simple, but cannot be stored as a Double precisely. Here are the Double values that it lies between:

*The denominator of the fraction is a prime number, which is not a power of 2*

binary: 0 01111111100 1001100110011001100110011001100110011001100110011001

hex: 0x3FC9999999999999

decimal (rounded off): .199999999999999983346654630623...

binary: 0 01111111100 1001100110011001100110011001100110011001100110011010

hex: 0x3FC999999999999A

decimal (rounded off): .200000000000000011102230246252...

In this case, the mathematically closest value is the second one, 0x3FC999999999999A.

The data is changing, even with a simple number like “0.2” when converting from String to Double. Those numbers at the end are not random, but the true value that the number has changed into when stored as a Double. The decimal is rounded, but it is the computer rounding the binary which is resulting in the change in the decimal number when it is converted back.

The same point is in reverse. If we have a Double value of 0x3FC999999999999A, this can only be exactly represented as a String (base 10) if we use an infinite number of digits. Once we decide to limit the number of digits, we are CHANGING the value. Limiting to 17 digits is conventionally used, as it has the essential property of giving each unique IEEE floating point value its own unique string representation. If you use less than 17 digits in the string you cannot guarantee exact round-tripping back to Double. You also cannot guarantee comparisons (like equality) with the Strings. They will mirror the same comparisons with the Double values.

As of FME 2017.1, FME will use the fewest number of decimal digits needed to make sure that if the value goes back to a Double afterwards the value is the original Double. This means that there are cases where FME will use less than 17 digits of decimal precision. However, FME will never use more than 17 digits, for reasons previously stated.

Even though the loss of precision is improving with each FME release, some precision loss might still occur. It is best to be aware of it, and know why it is occurring. Running all of your math calcuations through an AttributeRounder transformer or the using the @round function is a good way to avoid this.

thub.nodes.view.add-new-comment

025-precision.png
(9.9 kB)

02-precision.png
(7.3 kB)

© 2019 Safe Software Inc | Legal

- Anonymous
- Sign in
- Create
- New Question
- New Article
- New Idea
- Spaces
- 3D (and BIM)
- Attribute Handling
- Automations (FME Server)
- CAD
- Cloud
- Coordinate Systems
- Custom Transformers
- Database
- Dynamic Workspaces
- FME Cloud API
- FME Cloud Administration
- FME Cloud Getting Started
- FME Desktop 3rd Party Integrations
- FME Desktop Administration
- FME Desktop Administration & Configuration
- FME Desktop Development
- FME Desktop Getting Started
- FME Desktop Installation
- FME Desktop Licensing
- FME Desktop Plug-In SDK
- FME Desktop Workbench Scripting
- FME Server 3rd Party Integrations
- FME Server Administration
- FME Server Administration & Configuration
- FME Server Development
- FME Server Getting Started
- FME Server Installation
- FME Server Licensing
- Fanouts
- Ideas FME Cloud
- Ideas FME Desktop: Data Inspector
- Ideas FME Desktop: Formats & Systems
- Ideas FME Desktop: Transformers
- Ideas FME Desktop: Workbench
- Ideas FME Server
- Indoor Mapping
- KML
- Lists
- Performance Tuning
- Point Cloud
- Published Parameters
- Raster
- Real-Time
- Running Multiple Workspaces
- Tabular
- Troubleshooting Techniques
- Vector / GIS
- Web
- Workflow Design
- XML / GML
- Zip Files
- *FME Desktop
- *FME Server
- *FME Cloud
- *Other
- Explore
- Topics
- Questions
- Articles
- Ideas
- Users
- Badges