Working with Hadoop HDFS: Preview FME 2018

Connecting to HDFS is now in FME Desktop 2018

As of the FME 2018.0 release, we have added the HDFSConnector transformer as a technology preview. This means that while it’s not quite ready for prime time yet, it is available to take for a test run.

We’d like to call on anyone interested in this new transformer and especially anyone who voted on any HDFS, Hadoop, Hive enhancements for this to try this transformer and let us know what you think.

Give us your feedback

We’re actively working on developing more integration with Hadoop systems and we want to hear from you!

How would you like to use FME in a Hadoop setting? How are you using Hadoop and what formats are you using to store your data? Does the HDFS connector meet your needs or do you need some more functionality like a Hive, Hbase, Oozie, or Spark connector? Would you like to be able to trigger a Hadoop command or execute a query once an upload was complete? If so, what would that look like?

Please post your feedback as a comment back to this idea.

How does it work?

The HDFSConnector uses your HDFS account credentials (either via a previously defined FME web connection, or by setting up new FME web connection right from the transformer) to access the file storage service.

Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; or delete items from the service. On uploads, path attributes are added to the output features. On List actions, file/folder information is added as attributes.