Overview

The Data Lake Upload Tool is designed to help customers move their on-premises data into the cloud. It can be used to perform regular data backups or migrate legacy data to cloud storage for faster access through Popdock. Once data has been transferred to the data lake, it is stored, managed, and accessed from this central repository.

This installation guide will support eOne Partners and services consultants through setting up and using the Data Lake Upload Tool to migrate customer data from Dynamics GP. This guide includes instructions on preparing for the installation, installing the tool, and using the tool to port data from Dynamics GP to an Azure data lake or an Amazon S3 storage account.

Prepare for the installation

To ensure a successful install and migration of your systems using the tool, you will want to:

• Complete pre-installation tasks
• Meet minimum hardware/software guidelines
• Follow network recommendations
• Create an account credentials list

Pre-installation tasks

There are two pre-installation tasks the Data Lake Upload Tool requires and should be completed before installing it. An existing Azure Data Lake or Amazon S3 setup is required along with an Azure Data Lake or Amazon S3 connector configured in Popdock which connects to it.

The tool needs a running Data Lake with 2 storage containers/buckets. One will store your table information and the other will contain list information. You can use this article to set up an Azure Data Lake or this article for setting up Amazon S3.

Once the data lake containers or buckets have been created, you will add an Azure Data Lake Connector or an Amazon S3 Connector to your Popdock configuration. After either your Azure Data Lake or Amazon S3 Connector is created, you will have the necessary pre-installation tasks completed.

Hardware/Software Guidelines

The Data Lake Upload Tool requires that the computer running the tool is connected to the same network as the SQL database server, has enough processing power, storage space and memory for the tool’s data transfer processes and meets the minimum supported Operating System version guidelines.

Minimum hardware requirements:

Minimum software requirements:

Minimum database requirements:

NOTE: For technical support with earlier versions of SQL Server, contact support@eonesolutions.com.

Network Recommendations

The internet connection should be at least 5 Mbps upload to support the file transfer processes of the tool. The network and internet connection play a key role in the data transfer experience because low bandwidth internet and poor network connections can lead to slower transfer speeds and connection timeouts. While the 5 MBPs internet speed is a minimum guideline, a faster internet connection may not always guarantee faster data transfer speeds.

Account Credentials List

The installation process requires login information for multiple accounts to provide access to resources and systems involved in the upload process.

Use the list below to populate the necessary account information and have it available during the Data Lake Upload Tool installation.

Popdock Credentials – This is login information for the Popdock account.

Data Lake Connector Credentials – This information can be gathered from either the Azure Data Lake Connector or the Azure Console if using an Azure Data Lake. If using Amazon S3 storage, this information can be gathered from the Amazon S3 Connector.

For Azure Data Lake:

For Amazon S3:

Database Credentials – This information is from your database server.

Run the setup wizard installation

The Data Lake Upload Tool Setup Wizard is an installer that places the files necessary to run the upload tool on the computer.

To install the Data Lake Upload Tool:

1. This is the welcome page, select Next to continue.

2. In Select Installation Folder, a default installation folder for the Data Lake Upload Tool files is selected.

3. Select the default installation folder location, then Next to continue.

4. The Setup Wizard will install the software required by the upload tool.

5. If the Windows Desktop Runtime 6.0.13 is not installed, on the computer that will run the Data Lake Upload Tool, it will be installed when you select Install.

6. You will know the .NET Runtime installation is complete when you see this window, select Close.

7. The setup will resume the remaining steps in the installation automatically.

8. When you see this window, the setup wizard has successfully completed the installation, select Finish.

Copying Smartlists from Dynamics GP

Once the setup wizard is completed successfully, the files necessary to run the Data Lake Upload Tool are installed on the computer and the tool is ready to use.

When porting your legacy data, it is recommended that you migrate both the Smartlists and the tables and views on your Dynamics GP servers. In this section you will migrate the Smartlists and in the next section the tables and views will be selected.

Go to the Start Menu and select Popdock – Data Lake Upload Tool.

Welcome Page

On the welcome page, you will have the option to copy from different systems. Here we will look at options that apply to Dynamics GP.

To get started, you will want to make a copy of the Smartlists from your Dynamics GP Server. These should be the lists you use most frequently, so copy them first. Once you have a copy of these Smartlists, you will re-run the tool to copy all the tables and views.

1. Select the dropdown arrow on the right, to view the available options.

2. Select Dynamics GP- Copy SmartLists, and then select Continue.

Log in to your Popdock account

At the Log in to Popdock screen, you will provide the Popdock username, password, and account information for your account, which will be validated by the tool. The Popdock login entered should be the admin of the account.

1. At the log in screen, enter the Username and Password for the Popdock account admin, then select Log in to validate the credentials.

2. If the login information cannot be confirmed as entered, you will see the error message below. Check your Popdock login username/password, then enter the information and select Log in.

3. If you have more than one account, an Account field will appear on a new page. Select the dropdown arrow below account to view the list of available accounts.

4. If the default account populated in the Account dropdown is the Account you want, select Continue.

5. If the default account populated is not the account you want to continue with, select the new Account from the account dropdown menu and then select Switch account to move forward.

6. Once the login information and account are successfully validated, the installation process will automatically move to the next configuration page.

Connect to your storage account

At the Data Lake screen, you will provide connection information for your data lake and connector in Popdock. The installer will verify the information entered then proceed to the next step in the installation.

For Azure Data Lake (for Amazon S3 storage go here):

1. Here you will select a data lake connector, enter the storage account, enter the container, and paste the shared key.

2. Select the dropdown arrow under Data Lake connector to display the list of Azure data lake connectors configured in your Popdock account.

3. Select a Popdock Azure Data Lake connector.

4. Enter the name of your Storage account.

5. Enter the name of your List container.

6. Copy and paste the contents of your Shared key.

7. After entering the Azure Data Lake information, select Connect to validate.

8. If the storage account, container, or shared key information is entered incorrectly, the installation will not move to the Database page and the error message below will be displayed.

9. Check your credential list Azure data lake information, then re-enter information, and select Log in to validate.

10. Once the connection to your Azure Data Lake connection is successfully validated, the installation process will automatically move to the next page.

For Amazon S3:

1. Here you will select a data lake connector, enter the AWS region, enter the bucket, and paste both the access key and the secret key. 

2. Select the dropdown arrow under Data Lake connector to display the list of Data Lake connectors configured in your Popdock account.

3. Select a Popdock Amazon S3 Data Lake connector.

4. Enter the AWS region of your storage bucket (example format: us-east-2).

5. Enter the name of your AWS S3 Bucket.

6. Paste the contents of your Access key.

7. Paste the contents of your Secret key.

8. Select if you want to Use gzip. If selected, this will convert each list or table into a gzip file type. Gzip will provide performance gains when accessing the data via Popdock.

9. After entering your Amazon S3 information, select Connect to validate.

10. Once the connection to your Amazon S3 connection is successfully validated, the installation process will automatically move to the next page.

Connect to your database

At the Database screen, you will provide connection information for the Dynamics GP database. The installer will use the server and login information to test the connection to your database server.

1. Select the dropdown arrow under Version to display the list of versions available for your database application.

2. Select the software Version that matches your Dynamics GP server.

3. Under Server, enter the hostname of your database server.

4. Enter the Port being used by your SQL server to provide database services. The default port for database services is 1433, but before using it, confirm to find the TCP Port number your SQL Instance is listening on.

5. Enter the name of the system database.

6. Enter the SQL Server user credentials, under Username and Password.

7. Once all the information for the database server is entered, select Connect to validate.

8. If any of the database server connection information is not entered correctly, an error message will appear, and the installer will not move to the next page.

9. To find additional information on common error messages, go to the troubleshoot the tool section.

10. If a successful connection is made, the lists are loaded, and the installer will move to the next page.

Select Companies

The Select Companies screen displays the available company databases to choose from. You will select the companies here, and in the next screen, you will be able to choose which lists to download.

1. Choose the companies by selecting the box to the left of the company name.

2. Select the box next to the companies you want to copy, and then select Next: Select lists.

Select Lists

On the Select Lists page, you will choose the lists you want to copy from the selected companies.

1. Choose the Lists you would like to upload, by selecting the box to the left of each list.

2. After choosing your lists, select Next: Run.

Run

The Data Lake Upload Tool will start the import process immediately. At the Run screen, the upload tool will make the necessary connections, download the data to the computer running the data lake upload tool, then upload it to the data lake.

Once the tool has finished running, you will see the Report screen.

1. The Report screen shows the final status of the import process.

2. Now that the Smartlists have been copied, proceed to copying the Dynamics GP tables and views.

3. You can select Run Again on the Report page, to run the tool without having to re-enter your login credentials.

4. Select the Run Again button, the installer will return to the Welcome screen and start the configuration process again.

Copying Tables/Views from Dynamics GP

When porting your legacy data, you will migrate both the Smartlists and the tables and views on your Dynamics GP servers. You uploaded the Smartlists in the previous section, now the tables and views will be selected for migration.

Welcome Page

1. Select the dropdown arrow to view the Application/Process options.

2. Select Dynamics GP- Copy tables/views, then select Continue.

Log in to your Popdock account

The Log in to Popdock screen will display the Popdock username, password, and account information you logged in with previously. You have two options at this screen, update your Popdock credentials or use the same authenticated credentials by selecting the Next: Edit data lake settings button.

1. To make changes to your Popdock Username and Password, re-enter your credentials, then select Log in to validate.

To use the same Popdock credentials, simply select Log in.

2. The Account used in the prior process displays by default. Select Continue to move forward with this account.

Connect to your storage account

The Data Lake screen will display the Storage account information you logged in with previously.

1. Select your Data Lake (Azure Data Lake or Amazon S3) connector from the dropdown list. Update the available fields outlined if applicable. See Connect to your Storage Account section if needed.

2. Select Connect to continue.

Connect to your database

The Database screen will display the Dynamics GP database information you logged in with previously. At this screen, you can update your DB server and/or login information, or you can use the same authenticated information by selecting Next: Select databases.

1. To update your database server information, make changes to your settings then select Connect to re-validate.

2. To continue with the same database information, select Connect to re-validate and move to the next screen.

Select Databases

The Select Databases screen displays the available company databases to choose from. You will select the databases here, and in the next screen, you will be able to choose which views to download.

1. Choose the Databases by selecting the box to the left of the company name.

2. Select the box next to the companies you want to copy, and then select Next: Select views.

Select Views

At the Select Views page, you will choose the views you want to copy from the selected databases. By default, no views will be selected.

1. Choose the Views you would like to upload, by selecting the box next to each view.

2. Select Next: Run.

Run

The Data Lake Upload Tool will start the import process again.

Below are the options available on the Run page:

Cancel the download

Select Cancel to stop the upload and make changes to the settings in the upload tool, i.e., Data Lake, Database, etc.

Track upload progress

1. The Progress section shows the overall status of the upload process.

You can view the statuses below in the Progress section:

• “Running” is the total number to upload.
• “Pending” is the remaining number to upload.
• “Completed” is the number that uploaded successfully.
• “Failed” is the number that failed to upload due to an error.
• “Zero rows” is the number that were not uploaded due to having zero rows of data.

2. The Current process section displays the ongoing tasks the upload tool is performing and their status.

3. The Report screen will appear at the end of the upload.

You can view the statuses below in the Progress section:

• “Success” means the upload was successful.
• “Failed” means the upload failed due to an error.
• “Skipped” means the upload was skipped due to zero rows of data.
• “Cancelled” means the upload was cancelled.

4. Select Run again to execute the tool, without having to re-enter login credentials.

5. Select View logs to access more detailed information about the upload or to investigate any copy failures.

6. The View logs button will open the folder containing the log files. You can open the log files in a text editor to view detailed information on failures. You can use the log files to fix the upload errors or send them to support@eonesolutions.com for additional support.

7. After fixing the errors in the log file, you can select Re-run selected failures to retry copying the uploads with a failed status.

Troubleshoot the tool

Database “Login Failed” Error

If you get the “login failed for user” error, check that your username and password are:

• Typed correctly
• SQL server login credentials
• Have access to the system database

Database “Server not found” Error

The “Server is not found or was not accessible” error will appear, if the computer running the data lake upload tool, cannot connect to the database server.

In that case, you should:

• Make sure the server name is spelled correctly.
• Confirm that the server port entered matches the port on the SQL server.
• Check that the SQL server is up and running.

Questions about the Data Lake Upload Tool? Email support at support@eonesolutions.com