Notice

Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government, and shall not be used for advertising or product endorsement purposes.

Overview

This document provides instructions on the use and installation of Aspera Connect for high throughput file transfer with NCBI. As the sizes of the datasets have increased, we have found that the traditional methods of ftp or http do not have the performance characteristics needed to support this load of data.

Requirements for large scale data transfer over the internet include high bandwidth, auto checksum, recursive copy, and security based on strong keys. NCBI has chosen to use a product from Aspera, Inc (Emeryville, CA) because of improved data transfer characteristics. FTP and HTTP access will continue to be available and are the default options for users without Aspera installed. Instructions are provided below for investigators to use this data transfer technology. NCBI also is open to using additional products with the appropriate performance characteristics.

Aspera

Using ascp to Download by Command Line

The command line program ascp is a utility delivered along with the Aspera Connect product.

ascp -i <asperaweb_id_dsa.openssh with path> -k1 -Tr –l100m 
anonftp@ftp.ncbi.nlm.nih.gov:/<files to transfer> <local destination>

• -i <asperaweb_id_dsa.openssh with path> = fully qualified path & file name where

this public key file is located. This file is part of Aspera Connect distribution and is usually located in the ‘etc’ subdirectory.

• –T to disable encryption
• –k 1 enables resume of partial transfers
• –r recursive copy
• –l (maximum bandwidth of request, try 100M and go up from there)

Experiment with transfers starting at 100 Mbps and working up to 400 Mbps. Select the bandwidth setting that gives good performance with unattended operation.

• <files(s) to transfer> = names of files to transfer (including path)
• <local destination path> = location to store the downloaded data

Using ascp to Upload by Command Line

In order to use the Aspera upload service you will need to use a private SSH key, individual users can contact us at vog.hin.mln.ibcn@ars to request an Aspera private key.

Upload Command

ascp -i <private key file> -T -l 100m <file(s) to transfer>
asp-****@upload.ncbi.nlm.nih.gov:<destination directory> 

• -i < private key file > = fully qualified path & file name of the private SSH key
• –T to disable encryption
• –k 1 enables resume of partial transfers
• –l (maximum bandwidth of request, try 100M and go up from there)

Experiment with transfers starting at 100 Mbps and working up to 400 Mbps. Select the bandwidth setting that gives good performance with unattended operation.

• <files(s) to transfer> = names of files to transfer (including path)
• <destination directory> = deposit location of the uploaded data (typically either ‘test’ or ‘incoming’)

For password protected private keys, it is possible to run ascp in an autonomous, unattended manner that does not require repeated login. The environmental variable ASPERA_SCP_PASS can be used to store the private key path for a scripted series of bulk uploads.

Requirements

Troubleshooting

Here are some example commands demonstrating a test download.

Mac OS X:

ascp -T -l640M -i "/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh" anonftp@ftp.ncbi.nlm.nih.gov:1GB /tmp/

Linux:

ascp -T -l640M -i /opt/aspera/etc/asperaweb_id_dsa.openssh anonftp@ftp.ncbi.nlm.nih.gov:1GB /tmp/

MS Windows:

C:\TEMP>"C:\Program Files (x86)\Aspera\Aspera Connect\bin\ascp.exe" -T -l640M -
i "C:\Program Files (x86)\Aspera\Aspera Connect\etc\asperaweb_id_dsa.openssh " anon
ftp@ftp.ncbi.nlm.nih.gov:1GB C:\Temp\

For additional assistance, please contact the NCBI Help desk at vog.hin.mln.ibcn@ofni

When you are about to contact the NCBI Help desk please provide them some basic information like operating system, version of aspera connect, type of disk storage used for transferring files and the type of network connection your organization has to the internet.

If you have a Linux or MacOS X operating system you may run these commands and show us their output:

curl -o /dev/null ftp://ftp.ncbi.nlm.nih.gov/1GB
curl -o /dev/null http://www.ncbi.nlm.nih.gov/staff/beloslyu/large.tar
traceroute ftp.ncbi.nlm.nih.gov

First two commands download a 1GB file from NCBI using ftp and http protocols, the content is dumped to /dev/null. The third command will let us see the latency in your internet connection and possible congestions on the way to NCBI.

Another possibility is to make some test downloads from Aspera’s demo server, for Linux the command line is:

env ASPERA_SCP_PASS=demoaspera ascp -L- -T -l100m aspera@demo.asperasoft.com:aspera-test-dir-large/1GB /tmp/

Aspera Connect is a commercial product and program specific support is available from the manufacturer at http://asperasoft.com/support/

The currently up-to-date documentation for ascp can be found at http://downloads.asperasoft.com/en/documentation/8