Tech Tuesday: Importing Large Documents Using SmartConnect
When running large transactions from an eConnect application, SmartConnect would include any application referencing eConnect including the Direct Document Sender sample application included with eConnect. It is possible to get a number of different errors all related to various possible timeouts that eConnect uses.
The various errors that SmartConnect can give you are:
There was an error writing to the pipe: The pipe is being closed. (232, 0xe8).
There was an error reading from the pipe: The pipe has been ended. (109, 0x6d)
The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.
To understand how this could occur, it is helpful to know about a bit about how the architecture of eConnect works.
I “borrowed” the following screenshot of the eConnect architecture from MSDN and added 3 items into the list to show where SmartConnect fits in as well as the technology between the components.
At the top at 1, you have an eConnect application such as SmartConnect.
As we know, SmartConnect can take pretty much any source data and then transform it into an eConnect formatted XML document.
But where does it go from there?
SmartConnect then calls the local eConnect assembly Microsoft.Dynamics.GP.eConnect.dll and passes it the XML document created.
At this point, this is where it gets interesting.
In #2 above, we have the connection between the “.NET assemblies” and the “eConnect Integration Service”.
The eConnect Integration Service is a Microsoft Windows Service that runs on the local machine and listens for a WCF connection to be made to it.
When the call to the local eConnect assembly is made, we then go to step #2 above where the local eConnect assembly opens a WCF connection to the eConnect Integration Service and then passes it the XML document to process.
Because we want a document to be processed as “all or nothing”, the eConnect Integration Service starts a SQL transaction (#3 in the chart) and then proceeds to process each node in the XML as a call to an eConnect stored procedure.
Once the entire XML document is processed, the eConnect Integration Service either commits the transaction or rolls it back along with any error messages.
At that point, handling is returned back to the caller – the local eConnect assemblies– which in turn returns the results from the eConnect Integration Service back to the calling application.
Now you might be asking “OK, but how does that solve my problem?”
Well, it doesn’t directly but the specific error you are getting tells us what needs to be adjusted.
The two “pipe” error messages initially are WCF errors. Because we know that eConnect is making the WCF connection to the eConnect Integration Service – we know that we could be running across a WCF timeout of some type and so we need to adjust the WCF settings.
For a WCF connection, we need to make sure both the “sender” (SmartConnect) and the “receiver” (the eConnect Integration Service) both need to be on the same page for their WCF settings.
For the SQL transaction error, that actually is caused by a SQL Timeout happening although the error doesn’t specifically mention that happened. But it is.
You can download a zip file containing my Microsoft.Dynamics.GP.eConnect.Service.exe.config and also a new eOne.SmartConnect.UI.External.exe.config file that contains the client SmartConnect client eConnect settings HERE.
In both of the files, configuration lines have been added and/or modified and are commented to explain the changes.
You likely will not have an eOne.SmartConnect.UI.External.exe.config file so you can just copy that into your C:\Program Files (x86)\eOne Solutions\SmartConnect folder.
You WILL have a Microsoft.Dynamics.GP.eConnect.Service.exe.config file in your C:\Program Files\Microsoft Dynamics\eConnect XX.0\Service folder.
There isn’t anything unique in this configuration file (unless you manually changed/added it), so you can rename your existing Microsoft.Dynamics.GP.eConnect.Service.exe.config file to Microsoft.Dynamics.GP.eConnect.Service.exe.config.original. Then copy in the new Microsoft.Dynamics.GP.eConnect.Service.exe.config file.
Once you do this, make sure to restart the eConnect Integration Service on the machine.
Lastly, one other thing you will also need to change isn’t in those configuration files.
In the configuration files, we are changing the SQL Timeout settings. However there is a “maximum timeout” value that is set in your machine.config file that supposedly overrides our maximum value. On my own machine, I don’t seem to need it. Don’t know if that is because I’m in a VM or something but I’ve seen other customers needing to add it.
Per this blog article and others, we also need to add some lines into your machine.config.
We would only need to update the .NET 4.0 versions of this and both the 32bit & 64bit configurations just in case.
At the bottom of the machine.config, we will need to add the new lines for our settings. You won’t have these unless you modified the files already.
Below, I’ve added the new <system.transactions> section near the bottom of both machine.config files with the new maxTimeout value of 2 hours.
<machineSettings maxTimeout="02:00:00" />
This will match the eConnect & SmartConnect timeout changes that I’ve added as I’ve set them to also be 2 hours (using values of 120 minutes and 7200 seconds as appropriate).
Have any questions? Feel free to leave a comment below or email into firstname.lastname@example.org!
Leave a Comment
Integrate & Automate without Any Code.
SmartList Data has Never Been Faster.
The Easiest Way to Report on GP Data.
I have a client with a large number of G/L transaction including AA transactions. They were receiving a time out and I had them apply the changes to the eConnect and the machine config files, then reboot the machine. They are now receiving the following message:
There was an error reading from the pipe: The pipe has been ended. (109, 0x6d).
Journal Entry with AA (xlsx format) (JOURNAL_ENTRY) processed with errors. Not all journal entry records uploaded correctly, as noted below.
Company: BABET JE: Test Big SC Entry User: 427090
Company: BABET JE: Test Big SC Entry User: 427090
Would this error be related to the time out/pipe original error or is there something else we need to check?
As noted in the blog article “pipe” errors are WCF timeouts. So you’d want to focus on the eConnect config file changes typically.
But there are also some sc exe changes (the other file) that you can create – just copy in mine.
We have run into this issue with large imports using SmartConnect.
I have attempted to modify my machine.config but when I do this it causes GP to crash and i assume other applications might not work either.
I have obviously done something wrong in adding the system.transaction section.
Below is the end of my machine.config file. Does this look correct?
When I add the code to the machine.config files GP crashes for me too.
James & Dave,
You modified the config file incorrectly in some way. This would cause any application that has .NET 4.x components to fail due to a bad config file.
I modified this article to show a screenshot of the changes in context of my existing machine.config.
I have tried copied the config files shared in your link and have updated the two machine.config files for .Net 4.0, but all that did was change the error I was seeing from “There was an error reading from the pipe: The pipe has been ended. (109, 0x6d)” to “The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.” Is there something I am missing?