ListKeyValuePairExtractor - encoding error

I'm not sure if this gives the same result, but I gave it a try to make the whole thing a little bit shorter.

It will be easier to investigate if the result is close.

listkeyvalueextractor.fmx

(Thanks for tagging me, @david_r!)

Hi @nmeriotdev/Nicolas,

Sorry to hear about the trouble you are experiencing with "my" ListKeyValuePairExtractor!
I did test it on some special characters/encodings, but with this stuff, it's hard to catch all cases...

Would you mind sending me (a part of) your workspace and some test data, so I can hopefully reproduce the issue and improve/fix the ListKeyValuePairExtractor? As I'm quite busy at the moment, I might need 1-2 weeks or so...

You could try David's suggestions, but I doubt if they will work. The decode() function converts a string into a unicode object, but the prefix and suffix objects are probably of type unicode already. Instead, I expect that the attr object should be decoded as a unicode object, so the startswith() and endswith() functions will be called on objects of the same type.

The first thing we could try, is to switch FME from Python 2.7 to 3.x (Workspace Parameters > Scripting > Python Compatibility). If you have 3.x installed and have the option to use it, that is. The advantage of Python 3.x, is that the unicode type no longer exists and that it always compares bytes objects.

If 3.x doesn't work for you, you could change the code as follows:

def get_list_attribute_names(self, feature, pattern):
    prefix, suffix = pattern
    attributes = feature.getAllAttributeNames()     # get all attribute names
    result_list = []
    for attr in attributes:
        # filter out the attributes we need
        attr = attr.decode('cp1252')
        if attr.startswith(prefix) and attr.endswith(suffix):
            index = self.regex1.findall(attr)[-1]   # gets the last number (= list index)
            result_list.append((index, attr))
    return sorted(result_list)

As David already mentioned, you might need to play with the exact encoding here.
What does FME say in the log file around line 40? For instance, on my machine, it says:

Operating System: Microsoft Windows 10 64-bit (Build 16299)
FME Platform: WIN32
Locale: en_GB
Code Page: 1252 (ANSI - Latin I)

Alternatively, you can also change this line (from the code example above):

attr = attr.decode('cp1252')

into:

attr = attr.decode('cp1252', errors='ignore')

This will simply tell Python to ignore any bytes it cannot decode. Actually a bad solution, but at least your workspace will no longer crash here.

If this also doesn't work (i.e. the workspace still crashes), you could replace the get_list_attribute_names() function by this piece of code:

def get_list_attribute_names(self, feature, pattern):
    prefix, suffix = pattern
    attributes = feature.getAllAttributeNames()     # get all attribute names
    result_list = []
    for attr in attributes:
        # filter out the attributes we need
        try:
            if attr.startswith(prefix) and attr.endswith(suffix):
                index = self.regex1.findall(attr)[-1]   # gets the last number (= list index)
                result_list.append((index, attr))        
	except UnicodeError:            
            pass
    return sorted(result_list)

This is just a temporary solution though. I would love to properly fix the ListKeyValuePairExtractor...
I hope you are able to share your (partial) workspace + data here.

If you are the author of the code I would recomend changing python interpreter to 3.6+. it is default to unicode characters.

Remote debugging is hard :-)

Try the following instead:

prefix, suffix = pattern
prefix = prefix.decode('cp1252')
suffix = suffix.decode('cp1252')

You need to right-click on the custom transformer and select Edit.

The custom transformer will open in a new instance of FME Workbench, where you can edit the PythonCaller. When saving the custom transformer, the easiest is to not change the transformer version number (you will be asked before saving).

thanks for your reply @david_r but I'm not an expert on FME (self trained on it), so how can I modify Python code of that transformer ?

A small thing to test: modify line 113 of the PythonCaller to read:

prefix, suffix = pattern.decode('cp1252')

You may have to play around with the code page number

Hopefully hope @sander can help us out, he seems to be the developer :-)