Tech Tuesday: SmartConnect Processing Sequence
Tuesday, April 15, 2014
Our Director of Professional Services, Lorren Zemke, will be taking you through the SmartConnect processing sequence for this week’s Tech Tuesday!
In this article I am continuing my series of discussions on how SmartConnect processes data. Previously I’ve used examples on summing values and working with line sequence numbers. Today I will diagram the process flow of SmartConnect, using the process of sending data to Microsoft Dynamics GP but the same process is used for all maps.
Transformation
This section of data is processed before it is sent to the destination. All of the column calculations are done at this time and the data is prepped for sending to the destination. In the case of Dynamics GP, SmartConnect is creating the XML file that will eventually be sent to eConnect. In a call to a web service destination, SmartConnect is generating the object needed to make the call and send data to that web service.
Use the tasks in this section for setting global variable data that will need to be used for Post Document or Post Map tasks.
If you are using Global Variables in your map, you can clear them out at this time to get them ready for the next document unless you need them for a Post Document or Post Map Task.
Document Send
SmartConnect creates a connection to the destination and sends a single document.
Post Document Tasks
Once a document has been sent to the destination, based on whether the document was successful will determine which set of Post Document Tasks will execute. Tasks can run if the document was successfully integrated or not. If a task needs to run regardless of whether the document was successful or not, the task needs to be placed.
The Pre-Document Tasks, Document Send and Post Document Tasks will continue to loop based on the value set for the Map Keys on the Data Source section of the map.
Post Map Tasks
After all data has been sent to the destination, SmartConnect determines if the map was successful or not to determine which set of tasks to execute. For all destinations but Dynamics GP it will depend if even one document or task is listed as having failed, the tasks that are part of the failed document section will execute. Only if 100% of the documents are successful will the map success tasks execute.
Happy Integrating!
Interested in learning more about SmartConnect? Reach out to our Sales Team at sales@eonesolutions.com for more information!
In this article I am continuing my series of discussions on how SmartConnect processes data. Previously I’ve used examples on summing values and working with line sequence numbers. Today I will diagram the process flow of SmartConnect, using the process of sending data to Microsoft Dynamics GP but the same process is used for all maps.
The basic steps fall into the following categories:
- Pre-Map Tasks
- Data Read
- Transformation
- Pre-Document Tasks
- Document Send
- Post Document Tasks
- Post Map Tasks
Pre-Map Tasks
Pre-Map tasks are executed before the data is read from the data source. This can be document processing tasks such as moving files, running scripts to ask the user for values, running another map and reading data from a SQL table.
Data Read
Pre-map tasks have finished executing and the data is being read from the data source.
One specific task within SmartConnect, after the data is read, is called SQL Validation. This task is used to query SQL tables based on data from the data source. The Validation logic can verify if data exists or does not exist allowing the map to stop processing if the logic is not met.
Transformation
This section of data is processed before it is sent to the destination. All of the column calculations are done at this time and the data is prepped for sending to the destination. In the case of Dynamics GP, SmartConnect is creating the XML file that will eventually be sent to eConnect. In a call to a web service destination, SmartConnect is generating the object needed to make the call and send data to that web service.
Pre-Document Tasks
The tasks created in this section happen immediately before each document is sent to its destination. At this point all data has been retrieved and all calculations with the map have taken place.Use the tasks in this section for setting global variable data that will need to be used for Post Document or Post Map tasks.
If you are using Global Variables in your map, you can clear them out at this time to get them ready for the next document unless you need them for a Post Document or Post Map Task.
Document Send
SmartConnect creates a connection to the destination and sends a single document.
Post Document Tasks
Once a document has been sent to the destination, based on whether the document was successful will determine which set of Post Document Tasks will execute. Tasks can run if the document was successfully integrated or not. If a task needs to run regardless of whether the document was successful or not, the task needs to be placed.
The Pre-Document Tasks, Document Send and Post Document Tasks will continue to loop based on the value set for the Map Keys on the Data Source section of the map.
Post Map Tasks
After all data has been sent to the destination, SmartConnect determines if the map was successful or not to determine which set of tasks to execute. For all destinations but Dynamics GP it will depend if even one document or task is listed as having failed, the tasks that are part of the failed document section will execute. Only if 100% of the documents are successful will the map success tasks execute.
Happy Integrating!
Interested in learning more about SmartConnect? Reach out to our Sales Team at sales@eonesolutions.com for more information!
It appears SmartConnect fires the post map logic prior to creating it’s log entries for the run. Kinda hard to query the TaskHeader and TaskDetail tables from a script. Am I doing it wrong?
“Am I doing it wrong?”
I doubt it – how could you?
I would suspect it would work this way because a POST map task could also fail and therefore should be logged as well. And if SC has already written the entries, it would then need to write more.
But yes sometimes it would be nice if SC would write the detail entries as it goes vs all at once when finished.
So does this mean before SmartConnect sends the document to the GP server, it has to ingest the data to complete the transformation and create the document? Or does SmartConnect essentially perform a read query on the source server…with some kind of agent on that server…?…create the document, then send to the GP server?
I’m trying to figure out if SmartConnect is reading a large dataset and has to make any transformation, is that data set moved to the SmartConnect server, then effectively pushed to the target server.
Thanks!
Andrew,
SmartConnect processes the source data from wherever it executes from.
So typically the “local machine”
It reads the data from wherever it comes from – CRM Query, SQL, Excel, Text Files.
From there, it transforms the data however you need it to via calculations, etc and turns it into a “document”.
And then it calls the appropriate destination method. So that might be a SQL table directly, it might be CRM on prem or the cloud, might be BC or Salesforce, or might be GP.
Thank you for the explanation…can you clarify if this document is created on the SmartConnect server?
We are looking at hosting our SmartConnect server offsite. We will be charged for outgoing data transfer.
I am trying to calculate, given the number of maps and their source & destinations, whether running the maps will add to our outgoing data usage.
So…in my example…SmartConnect server A is in AWS…Servers B & C are on the home network. If I am moving data from B to C, is the “document” generated by SmartConnect generated on server A in AWS, then moved to server C as the load…?
Or does SmartConnect somehow direct server C to pull from server B using instructions from the “document”…?
Thank you in advance for your help!
To me, this means there is not a true pre-document task. I need to write a script that gets a next transaction number from GP (and I cannot use the built-in GP rolling column function because the source dynamically indicates the target company) and placing that script on the pre-document tasks means my first document does not get a next number.
Think of “before document” to be “Before Document Submit” which means “I have my document, here is the final “send it/stop it” option.
I haven’t had an issue with using a GP Rolling Column not getting the “next number” from an integration regardless if I used the “Define” capability or else hard coding the company destination.
So You might want to double check that this isn’t working for you.
But if that doesn’t work for you or if we just want to do it ourselves then we could.
You are correct there is no task for “I’m starting a new document now”.
Easiest solution is to use the “Restriction” script of whatever your first transaction mapped node is.
In that restriction script, make a .NET connection to your database and run your proc to get the next number.
in that script, you would have something like:
if GBL_NEXTDOCNO = “” then
GBL_NEXTDOCNO =
end if
return true
if the global is empty, then we execute the .net sql code to pull that next number. If not, we leave it set as-is.
Now make a calculation that just returns the global
return GBL_NEXTDOCNO
and map that everywhere you need a document number for the document.
Now the issue is, we need to clear it so that on the NEXT document, we find the next number again.
And we can either use:
BeforeDocument task (because it is ready to do)
or instead
AfterDocumentSuccess & AfterDocumentFailure
clear the global in both
GBL_NEXTDOCNO = ""
return true
So as we go, we find the next global and then use it in the map.
and then before/after we've finished the document, clear it out so that we re-get it during the next cycle.
Hmmm. Not sure why that posted in Red like that. But I do like that it looks authoritative.