How To

Using the HTTPFetcher

Article Number: 000002134 -  Last Modified: Jul 21, 2015

Fetches a target by performing a HTTP GET operation on a specified URL, storing the results in the specified target attribute.

Description

Fetches a target by performing a HTTP GET operation on a specified URL, storing the results in the specified target attribute.

Fetching Images

The ImageFetcher transformer can be used to retrieve images on an FTP site. Note that this transformer does not have authentication parameters, so the only way to provide a username/password is to enter it in the URL.

FTP Sites

Q) Can the HTTPFetcher be used to download a file from an FTP site?

A) Yes (Note: FME 2015 introduced the FTPCaller transformer which allows downloading data from a FTP server to an attribute or file)

Processing Results

When the attribute is retrieved, it may be used for further processing, however, it can be quite hard as it contains all the HTML tags and data not displayed in the browser such as content within <head></head> tags and <script></script>.

In order to extract the visible content you can use either some HTML parser such as BeautifulSoup, which also requires installing Python, or you can try to get rid of the tags with Regular Expressions.

Example

The attached example shows how to bring the attribute containing a multiline HTML code to a clean text visible in the browser. As a fun conclusion, it combines results with the map of the USA correlating state areas with page popularity (the 50 most popular pages correspond well with 50 states).

User-added image


Suggested Similar Articles

FME can be used to geocode address records using the open source Gisgraphy service (uses geonames and openstreetmap data).

Discussion of how to use XQuery and how it works with FME. Gives examples of element and attribute extraction, FLWOR queries, simple conditions, and join expressions.

Recent concerns over the now publicly exposed vulnerabilities with the Open SSL libraries has prompted many software venders to provide information related to this issue. Here is Safe Software's breakdown of what you need to be aware of.

This example shows how to build an xfMap to read complex XML or GML.

This example shows how to build a workspace that consumes JSON results from the Yahoo GeoPlanet web service. It consists of three main parts: getting user input, building and submitting a URL, getting some response from the service, and exploring results and creating geometries.