Skip to content
+1-888-319-3663

COMMUNITY FORUM

REST datasource slow

Justin asked 12 months ago

I have a rest datasource that is retrieving about 3500 records, 1.15mb of data.  I can use postman and retrieve the results in 400ms.  If I ping the same endpoint with smartconnect, it takes 14 hours to process.  is there some kind of limit on smartconnect causing this?

 

by the way, my destination is a simple sql table.  

 

here is an example of the json from the api:

{
  "data": [
    {
      "LocationId": 1666,
      "CustomerId": 1225,
      "Customer": null,
      "DateAdded": "2021-12-20T00:00:00",
      "LocationName": "DUN",
      "AddressCode": "DUN",
      "AddressLine1": "1000 LITTLE RD",
      "AddressLine2": null,
      "City": "COLUMBIA",
      "ZipCode": "99999",
      "StateId": 1,
      "State": null,
      "TerritoryId": 3,
      "LocationTerritory": null,
      "IsNonTax": false,
      "LastModified": "2021-12-20T16:12:26.3663785"
    },
Answers
Ethan Sorenson Staff answered 3 months ago

Justin,

This is a limitation of the legacy REST Connector in SmartConnect 2018 and older. The connector was not efficient at parsing large data steps.

I encourage you to upgrade to SmartConnect 21 to take advantage of the new REST Connector which is optimized for large data sets and supports more types of authentication and paging. You can find full documentation and download links here.


If you would like to submit an answer or comment, please sign in to the eOne portal.