Data Transfer Using Globus
Contents
NOTE
22 November 2023
The Globus Connect Server at University of Auckland has been upgraded from version 4 to version 5.4.
Access your research drive on the Data Transfer Node (DTN) by searching the FileManager for the collection University of Auckland Research Data Collection.
Transfer data between UoA and other NZ locations
-
You can use globus to transfer data between your research drive and NeSI and other associated NZ public globus endpoints
Transfer data between an institutional globus endpoint and your workstation
- Using GlobusConnectPersonal, you can create a personal endpoint on your workstation or Nectar vm, and transfer data between that endpoint and any NZ institutional endpoint in exactly the same way: Nectar vm data-transfers
University of Auckland Research Drive Access
If you have transferred files to or from a previous University of Auckland Globus endpoint (prior to November 2023), you should find that all research-drives you had Globus access to then are now visible from your home-directory of the current UoA data collection
If you have not previously transferred files to or from a University of Auckland Globus endpoint, you need to request access from Centre for eResearch. Please submit a research-drive request, specifying “existing” storage, providing
* the name of the research drive
* the usernames (UPI) of those on your team who need globus access to that drive.
Access your research drive on the Data Transfer Node (DTN) either by using the link University of Auckland Research Data Collection or . searching the FileManager for the collection “University of Auckland Research Data Collection”.
Once you have logged in (UoA Single-Sign-On), navigate to your research drive from “/home/<your_upi>”, or “/~/” for short.
or to “/~/”:
Transfer Errors
You can inspect the logs for your globus transfers from the console in globus:
-
Select the Activity tab on the LHS of the globus page and then select the job that you want to investigate.
-
Select the Event Log tab (highlighted) and view the Fault Events for that job. You may be able to see the problem with your transfer here. If this raises issues you can’t control, please get in touch with Centre for eResearch.
Performance
From https://portal.xsede.org/data-management:
Beyond the capabilities of the network and storage systems involved, the single biggest factor in your transfer performance will be the size of the files being transferred, as there is time spent on the network setting up and tearing down the connection for each file transfer, and for high bandwidth networks like XSEDE this is relatively costly. For example, going from file sizes of 10MB to 1GB can improve your average transfer performance from a few MB/sec to over 1GB/sec. If you need to transfer a large number of files you will get the best performance by first bundling them into a single tar file and copying the single file.
Archive (bundle) your files prior to transfer
cd <name_of_directory_containing_files>/..
tar czvf <bundle_name>.tar.gz <name_of_directory_containing_files>/
now transfer the file <bundle_name>.tar.gz
Unpack your files after transfer
tar xzvf <bundle_name>.tar.gz #extract into directory named <name_of_directory_containing_files>