Delete files and sub-folders from folders not starting with a specific name?
I really don't know if this is something that FME can do, but it has done a lot of things so far so just seeing if this might be possible!
I have a directory of 'D:\data' which has many folders containing data from map layers (D:\data\parks, D:\data\buildings, D:\data\rivers etc).
What I am trying to do is automate the deletion of the contents of these sub-folders (which include both files and more folders), except any folders within 'data' that start with the name 'raster' (for instance I don't want to delete D:\data\raster_aerial2016 and all of its contents).
Is this something that FME Desktop can do? I don't have experience in python / batch files and FME has solved many problems so just wondering if there are any transformers that can help me out?
Nice question. Of course - and I know I'm stating the obvious - you'll be sure to test your chosen technique (including mine) before letting it loose on your real data. I'd hate to be the cause of your entire filesystem going up in smoke!
My thought is that you could use the "Directory and File Pathnames" reader to read a list of files, use a Tester to drop all those that contain the word 'raster' and then use the "File/Copy" writer to move the remaining files to the /dev/null equivalent for your operating system.
Attached is a workspace that shows the technique. It searches for txt files in C:\FMEOutput and moves it to "nul" in another folder. Windows doesn't allow a file called "nul", so it is deleted.
Hope this helps.
Just to make sure my understanding of this is correct...
"nul" isn't explicitly specified in the FileCopy Writer, it ends up being "nul" because no values are supplied for the filecopy_dest_dataset and file_copy_dest_filename Writer parameters?
Although in the log it says...
"File Copy Writer: Received move request from 'C:\FMEOutput\ test.txt' to 'C:\FMEInput\nul'"
Where is C:\FMEInput coming from?
Hi all.BEES Archiver V4.fmw
I realise I'm very late (years!) to this thread. I was searching for something else on the forum and came across this thread.
I had to perform a similar task earlier this year (2019) and thought I'd share my answer nonetheless, especially because it doesn't use Python code to remove folders, as per CartoChris' question - and hopefully it might provide inspiration to others out there.
My problem was similar - as part of an archiving task I had to both move (copy) and remove particular files and folders on our network.
My solution (attached) basically takes the 'path_windows' attribute and feeds it into a SystemCaller transformer containing the rmd command to remove the necessary folders but not the folders above these folders.
The workbench is 'hard-coded' to the directory structure of my organisation, but I'm sure you will work it out. If not, get in touch.
I find the SystemCaller transformer referenced above by jeroenstiers to be great for file system operations: if you can do it in a CMD window, you can do it in FME!
Hi,
Yes, copying the required files to another folder like E:\Data to E:\Data1 with filecopy writer and delete the folder E:\Data in normal windows is the best option.
Pratap
Hi @cartochris,
I created a small example of workspace you can use for this functionality. It does make use of a small python-script that lists all folders in the parent DATA-folder. It loops through this list and to check the name of every folder. If the folder doesn't start with 'raster', a feature is created (with the url of the folder) and send to a SystemCaller that will use the path and the RMDIR "PATH_TO_FOLDER" \S \Q command to remove it and all files / folders inside it automatically.
You might want to remove the \Q in the statement above so you will have to confirm before removing a folder.
I don't think I have to specify how dangerous this code can be if the parent-folder is pointing to a drive... So take watch out with the use!
The PythonCode:
import fme
import fmeobjects
import glob
class RemoveData(object):
def __init__(self):
self.parentUrl = 'D:/DATA/'
def input(self,feature):
# Don't do anything for every feature entering
pass
def close(self):
# List all folders in the parentfolder (self.parentUrl)
folders = glob.glob('{}{}'.format(self.parentUrl, '*') )
# Loop throught all returned folders
for cFolder in folders:
# Check if the name starts with 'raster'
if cFolder.split('\\')[1][0:6] == 'raster':
# Skip this folder
continue
# Create feature, add attribute, and send it to workspace
newFeature = fmeobjects.FMEFeature()
# Convert the slashes to windows-notation
newFeature.setAttribute('url', '"{}"'.format(cFolder.replace('\\\\', '\\').replace('/', '\\')))
self.pyoutput(newFeature)example-removingallsubfolders.fmw
Hopefully this helps you!
Jeroen
Hi @cartochris, as far as I know, there is no way to remove files unless you write a script. However, I think you can copy or move only files which are saved under the "D:\data\raster_*" folder to a backup folder in the disk system using the Directory and File Pathnames Reader and File Copy Writer. Once you created the backup, it's easy to remove remnant files manually.
Thanks for your reply @takashi. My raster folders are very large in size so wouldn't like to move them ideally, but this is no problem I think I might be able to do what I need using the robocopy command in Windows.
I don't think moving files in the same disk drive takes so long time, since it is equivalent to renaming. However, I didn't know that moving a file to 'null' works to delete it. It would be a good tip as @Mark2AtSafe suggested.
Anyway, be careful not to delete necessary files. I always create a backup for safe before cleaning up a folder, even if it could take a long time.
