Overview

The Data Lake Upload Tool is designed to help customers move their on-premises data into the cloud. It can be used to perform regular data backups or migrate legacy data to cloud storage for faster access through Popdock. Once transferred, your data is stored in this central repository and can be accessed through Popdock.

This installation guide will helps eOne Partners and services consultants set up and use the Data Lake Upload Tool to migrate customer data from Dynamics NAV. This guide includes instructions on preparing for the installation, installing the tool, and using the tool to copy data from Dynamics NAV to an Azure Data Lake or an Amazon S3 storage account.

Prepare for the installation

To ensure a successful install and migration of your systems using the tool, you should:
• Complete pre-installation tasks
• Meet minimum hardware/software guidelines
• Follow network recommendations
• Create an account credentials list

Pre-installation tasks

There are two pre-installation tasks you must complete before installing the Data Lake Upload Tool. An existing Azure Data Lake or Amazon S3 environment is required along with an Azure Data Lake or Amazon S3 connector configured in Popdock that connects to it.

The tool needs a running data lake with a storage container (Azure)/bucket (Amazon S3) to store your list information. You can use this article to set up an Azure Data Lake or this article for setting up Amazon S3.

Once the data lake container or bucket has been created, add an Azure Data Lake connector or an Amazon S3 connector to your Popdock configuration. Once the connector is created, the pre-installation tasks are complete.

Hardware/Software Guidelines

The Data Lake Upload Tool requires that the computer running the tool is connected to the same network as the SQL database server, has enough processing power, storage space and memory for the tool’s data transfer processes and meets the minimum supported operating system version guidelines.

Minimum hardware requirements:

Minimum software requirements:

Minimum database requirements:

Network Recommendations

The internet connection should be at least 5 Mbps upload to support the file transfer processes of the tool. The network and internet connection play a key role in the data transfer experience because low bandwidth internet and poor network connections can lead to slower transfer speeds and connection timeouts. While 5 Mbps is a minimum guideline, a faster internet connection does not guarantee faster data transfer speeds.

Account Credentials List

The installation process requires login information for multiple accounts to provide access to resources and systems involved in the upload process.

Use the list below to gather the necessary account information and have it available during the Data Lake Upload Tool installation.

Popdock Credentials – This is login information for the Popdock account.

Data Lake Connector Credentials – Gather this information from the Azure Data Lake Connector or the Azure portal (Azure), or from the Amazon S3 Connector (for Amazon S3).

For Azure Data Lake:

For Amazon S3:

Database Credentials – Gather this information from your Dynamics NAV database server.

Run the setup wizard installation

The Data Lake Upload Tool Setup Wizard is the installer that places the files necessary to run the upload tool on the computer.

To install the Data Lake Upload Tool:

1. On the welcome page, select Next to continue.

2. Accept the license agreement and then click Next.

3. In Select Installation Folder, a default installation folder for the Data Lake Upload Tool files is pre-selected.

Select the default installation folder, then select Next.

4. The Setup Wizard will install the software required by the upload tool.

5. If the Windows Desktop Runtime 6.0.13 is not installed, on the computer running the Data Lake Upload Tool, install it by selecting Install.

6. Once the .NET Runtime installation is complete, select Close.

7. The setup will automatically resume the remaining steps.

8. When the setup wizard completes the installation successfully, select Finish.

Copying Lists from Dynamics NAV

Once the setup wizard completes successfully, the tool is installed and ready to use. You will need the account credentials list you gathered earlier to use the Data Lake Upload Tool.

In this section, you will use the tool to import your legacy data from Dynamics NAV by migrating the tables as lists.

Go to the Start Menu and select Popdock DataLake Exporter.

Welcome Page

On the welcome page, you can choose to copy data from different systems. The only option available for Dynamics NAV is copying lists, which creates lists for all your NAV tables.
Proceed with making a copy of all the tables as lists from your server.

1. Select the dropdown arrow on the right, select Dynamics NAV – Copy lists, and then select Continue.

Log in to your Popdock account

At the Log in to your Popdock account screen, provide the Popdock username, password, and account information for your account. The information will be validated by the tool. The Popdock login entered should be the admin of the account.

1. At the Log in to your Popdock account screen, enter the Username and Password for the Popdock account admin, and then select Log in to validate the credentials.

2. If the login information cannot be verified, the tool displays an error message. Check your Popdock login username and password, re-enter the information and select Log in.

3. If you have more than one account, an Account field appears on a new page. If the account populated in the Account dropdown is the account you want, select Continue.

4. If the populated account is not the one you want, select the account from the Account dropdown menu, and then select Switch account to continue.

5. Once the login information and account are successfully validated, the installation automatically moves to the next configuration page.

6. Select your Popdock Data Lake connector from the dropdown list.

7. Once you have selected your Data Lake connector, you are automatically taken to the Connect to your storage account screen.

Connect to your storage account

At the Connect to your storage account screen, provide connection information for your Azure Data Lake or Amazon S3 storage account and its respective connector in Popdock. The installer will verify the information entered, then proceed to the next step in the installation.  

For Azure Data Lake

(for Amazon S3 storage see the next section):

1. Select a data lake connector, enter the storage account, enter the list and table containers, and paste the shared key. 

2. If the Data lake connector field is not populated from the previous steps, select your connector from the Data lake connector list.

3. Enter the name of your Storage account.

5. Enter the name of your List container.

6. Enter your Table container.

7. Copy and paste your Shared key.

8. After entering the Azure Data Lake information, select Connect to validate.

9. If the storage account, container, or shared key information is entered incorrectly, the installation will not move to the Database page, and an error message is displayed.

10. Check your credential list for Azure Data Lake information, re-enter information, and select Connect to validate.

11. Once the connection to your Azure Data Lake is successfully validated, the installation process automatically moves to the next page.

For Amazon S3:

1. Select a data lake connector, enter the AWS region, enter the bucket, and paste both the access key and the secret key. 

2. Select the dropdown arrow under Data Lake connector to display the list of data lake connectors configured in your Popdock account.

3. Select a Popdock Amazon S3 Data lake connector.

4. Enter the AWS region of your storage bucket (example format: us-east-2).

5. Enter the name of your AWS S3 Bucket.

6. Paste the contents of your Access key.

7. Paste the contents of your Secret key.

8. Select if you want to use gzip. If selected, the tool converts each list or table into a gzip file. Gzip provides performance gains when accessing the data via Popdock.

9. After entering your Amazon S3 information, select Connect to validate.

10. Once the connection to your Amazon S3 is successfully validated, the installation process automatically moves to the next page.

Connect to your database

At the Connect to your Database screen, provide connection information for the Dynamics NAV database. The installer will use the server and login information to test the connection to your database server.

1. Under Server, enter the hostname of your database server.

2. Enter the Port used by your SQL server for database services. The default port is 1433, but before using it, confirm the TCP port number your SQL instance is listening on.

3. Enter the name of the System Database.

4. Enter the SQL Server user credentials, under Username and Password.

5. The Schema defaults to dbo. If you are using a different schema, enter it here.

6. Once all the information for the database server is entered, select Connect to validate.

7. If any of the database server connection information is entered incorrectly, an error message appears, and the installer will not move to the next page.

8. To find additional information on the common error messages, see the troubleshoot the tool section.

9. If a successful connection is made, the lists begin loading and the installer moves to the next page.

Select Companies

The Select Companies screen displays the available company databases to choose from. Select the companies here, then choose the lists to download in the next screen.

1. Choose the companies by selecting the checkbox next to the company name. Then, select Next: Select lists.

Select Lists

On the Select Lists page, choose the lists you want to copy from the selected companies. The lists are copies of every NAV table.

1. Select the Lists you want to upload. Then, select Next: Run.

Run

The Data Lake Upload Tool begins the import process. The upload tracking information below is available on the Run page:

Cancel the download

Select Cancel to stop the upload and make changes to the settings in the upload tool (for example, Data Lake or Database settings).

Track upload progress

1. The Progress section shows the overall status of the upload process.

The following statuses are displayed in the Progress section:
• Running is the number currently being uploaded.
• Pending is the remaining number to upload.
• Completed is the number that were uploaded successfully.
• Failed is the number that failed to upload due to an error.
• Zero rows is the number that were not uploaded because they contain zero rows of data.

2. The Current process section displays the ongoing tasks the upload tool is performing and their status.

3. The Report screen appears at the end of the upload.

You can view the following statuses in the Progress section:
• Success means the upload was successful.
• Failed means the upload failed due to an error.
• Skipped means the upload was skipped due to zero rows of data.
• Cancelled means the upload was cancelled.

4. Use Run again to execute the tool, without re-enter login credentials.

5. Select View logs to access more detailed information about the upload or to investigate any upload failures.

6. The View logs button opens the folder containing the log files. Open the log files in a text editor to view detailed information on failures. Use the log files to fix the upload errors or send them to support@eonesolutions.com for additional support.

7. After fixing the errors in the log file, select Re-run selected failures button to retry copying the uploads with a failed status.

Troubleshoot the tool

Database “Login Failed” Error

If you get the “login failed for user” error, check that your username and password meet the following criteria:
• They are typed correctly.
• They are valid SQL server login credentials.
• They have access to the system database.

Database “Server not found” Error

The “Server is not found or was not accessible” appears, if the computer running the Data Lake Upload Tool, cannot connect to the database server.

In that case, check the following:
• Make sure the server name is spelled correctly.
• Confirm that the server port entered matches the port on the SQL server.
• Check that the SQL server is up and running.

Questions on the Data Lake Upload Tool? Email support at support@eonesolutions.com