+1-888-319-3663

COMMUNITY FORUM

File Names in a Folder Data Source

tomHamill asked 8 years ago

Is there any way to pull the file name into a SmartConnect script when running a map that uses a folder data source? I have several maps that I run daily. All of them use folder data sources. All of them can, and do, import multiple files in a single run. I’d like to be able to use those file names in scripts running before and after the map to log basic data like what file was imported, how long it ran, did it succeed or not. Most importantly, I want to be able to check if a file has been imported already and stop the import of that file if it has. Any suggestions?

Answers

0 Vote Up Vote Down

patrick answered 8 years ago

tom, While a good suggestion, No, this isn’t possible. In a folder data source – the data isn’t processed one file at a time. Instead, all the files are read at once and processed together. So when it is processing the data, at that point it is just one big pile data of the conglomerated data and so SC doesn’t know where it came from. At the end of the integration, all of the source data files are then moved in the success/fail folder regardless if the data from a particular file was successful or not. Patrick eone

John replied 8 years ago

Drat!

patrick replied 8 years ago

Update- actually I’ve learned that it does know which file the data came from.
But that doesn’t really help us because SC doesn’t expose a variable that tells us this info. So double-drats.

However, if we wanted to be clever, it wouldn’t be that hard to do this.
You’d only have to create a map “pre” script that runs.
You would use a Folder/File class to loop through the files in a directory that will be a new “import from here” directory.
You could then read the contents of each file and determine if you need to integrate it or not. If so, then you make a new file that has the file name as a column and then each data line from the source file.
now write that file into the “Source data” for the map.
loop through each file and write them back into the source data folder.
now when the map runs, the original file name is a column in the source and you can do whatever.

If you would like to submit an answer or comment, please sign in to the eOne portal.