• EN
Question

Writing LARGE datasets to ArcGIS Online

runneals
Rank
Badge +14

Working with esri and the developers for ArcGIS Online, we came up with this script that can be used on shutdown. You will need to write data out to a zipped file geodatabase (or other formats), then configure the script.

Only works with 2019.x+ as it requires you to use ArcGIS Pro (tested in 2.3). It does take a while to run (as it has to use the ArcGIS Python library), but is SUPER efficient when trying to write LARGE datasets to ArcGIS Online. It is much preferred over using the FME writer because that one uses the traditional database insert for each record while this is similar to bulk inserts. Can probably be modified for writing to Portal as well.

https://github.com/runneals/GISProjects/tree/master/agol/Append

cc @carsonlam @bruceharold @mmccart

8 replies

bruceharold
Rank
Badge +16

Very cool David. Also check it out @1spatialdave

bruceharold
Rank
Badge +16

@carsonlam Hi, this question could be refactored as an idea, namely to support bulk appends when the arcgis Python API is available, with and without truncate. Personally I use Pro to populate feature services and use DI/FME for maintenance operations but others may like a bulk writer.

C
Badge

@carsonlam Hi, this question could be refactored as an idea, namely to support bulk appends when the arcgis Python API is available, with and without truncate. Personally I use Pro to populate feature services and use DI/FME for maintenance operations but others may like a bulk writer.

Is there a similar bulk append mechanism in the ArcGIS REST API?

bruceharold
Rank
Badge +16

Is there a similar bulk append mechanism in the ArcGIS REST API?

https://developers.arcgis.com/rest/services-reference/truncate-feature-layer-.htm

https://developers.arcgis.com/rest/services-reference/append-feature-service-.htm

LizAtSafe
Rank
Userlevel 2
Badge +14

@carsonlam Hi, this question could be refactored as an idea, namely to support bulk appends when the arcgis Python API is available, with and without truncate. Personally I use Pro to populate feature services and use DI/FME for maintenance operations but others may like a bulk writer.

Hi @bruceharold,

Unfortunately we cannot convert questions to ideas, but I have created the idea for you to vote on. Please see Support bulk appends when the ArcGIS Python API becomes available

- Liz

cc: @runneals, @mmccart, @1spatialdave, @carsonlam

bruceharold
Rank
Badge +16

Hi @bruceharold,

Unfortunately we cannot convert questions to ideas, but I have created the idea for you to vote on. Please see Support bulk appends when the ArcGIS Python API becomes available

- Liz

cc: @runneals, @mmccart, @1spatialdave, @carsonlam

Thanks Liz! Maybe Carson will make an end run around the idea and do it with the REST API so there is no ArcGIS dependency. I'm seeing a lot of action with bulk work to AGOL so it will be popular.

runneals
Rank
Badge +14

Is there a similar bulk append mechanism in the ArcGIS REST API?

Great minds think alike @carsonlam :) I was going to go that way with the next enhancements for improving the script, as using the ArcGIS Pro API requires quite a bit of overhead and increases the run time quite a bit. However my workload exceeds my free time, so probably won't be happening for a while on my end.

If you need a use case, we have a workspace that runs every week that updates historic rail lines in Iowa that takes almost 30 minutes to run, and almost 20 minutes of that is just writing data up to ArcGIS online, so using this bulk append method would only take a couple minutes and speed it up quite a bit.

runneals
Rank
Badge +14

I was successfully able to get over the major issue when the file geodatabase would get stuck on ArcGIS Online due to the delete portion of the script not being executed. See the sample below for the best implementation that works now (using feature writer and a python caller)!

Sample_Workspace_AGOL_Append_PUBLIC.fmw

Reply


Community Stats

28,866
Posts
109,981
Replies
36,669
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.