TRANSFERRING FILES

For transfers from your desktop or home computer, or another computer external to Research Computing, to one of the Research Computing clusters there are several methods:

• Globus Online: Globus Connect File Transfer. To get started with Globus Online see the Getting Started page, see Getting Started With Globus Connect.  Click for video example.

• In addition to Globus Online, there are a number of SFTP (Secure File Transfer Protocol) tools available for medium or small file transfers on both Mac and Windows platforms.  Click for video example.
  (Big file transfers should be done with GLOBUS).

  • Some common file transfer commands include sftp, scp, and rsync.
  • For file transfer clients (GUI Interfaces):
    • Mac users can also use Fetch which is a ftp client which has a graphical user interface.
    • Window users, file transfer clients are available at UNC Software Distribution Shareware: http://software.sites.unc.edu/shareware/#s. Some examples include:
      • WinSCP, CoreFTP, or the ftp client that comes with MobaXterm. SecureCRT also has built-in file transfer capabilities.
      • CyberDuck (https://cyberduck.io/) is available for both Mac and Windows platforms.
      • CoreFTP (http://coreftp.com/) is another possibility for Windows platforms.
      • FileZilla (https://filezilla-project.org/) is also an option for Mac, Windows, and Linux platforms.

  For the SFTP tools, although it is possible to connect directly to the cluster log in nodes, the cluster log in nodes are a shared resource and long sftp processes degrade performance for everyone else on that login node. Use one of our specialized data mover nodes instead. There are four data mover nodes: rc-dm1.its.unc.edu, rc-dm2.its.unc.edu, rc-dm3.its.unc.edu, and rc-dm4.its.unc.edu. Connecting with the host address: rc-dm.its.unc.edu will connect you to the least busy of the four. This will generally result in the best performance.

• If transferring files between your local machine to a Research Computing cluster, the file browser tool in Open OnDemand is the recommended approach for files less than 10 GB in size.

WHEN IS FILE TRANSFER SO BIG TO BE CONSIDERED A BIG FILE TRANSFER?

The file transfer size required to make it count as a big transfer is rapidly increasing as technology and network bandwidth improves. Data transfer times are fastest for copying a file between two UNC research clusters, then copying between a research cluster and your on-campus computer and significantly slower between a research cluster and your off-campus computer (due to the VPN and your internet connection). The slowest is copying a file from a media (ex: CD, DVD) onto your computer hard drive [to then copy to a research cluster]. Because of this, listing any explicit size determination for a big file is problematic. One solution is to treat every file as a large file and use GLOBUS FOR EVERY COPY. THIS WILL WORK, BUT REQUIRES MORE OVERHEAD THAN SOME USERS FIND PRACTICAL FOR THEIR WORK FLOW. INSTEAD, HERE IS A GUIDELINE BASED ON TIME: IF THE COPY COMMAND (CP, SCP, SFTP, ETC.) is going to take more that 10 minutes, then treat the copy as a big file transfer. If it is less than a minute, then it is a small transfer. Medium-sized transfers are, therefore 1-10 minutes long. As of this writing, this roughly translates to the following cutoff size between medium & big files:

• Between 2 research clusters:
  • any directory to mass storage: Use
  • SLURM or GLOBUS. - home directory to home directory: 200g.
  • home directory to/from scratch or project directory: 500g.
  • scratch or project directory to scratch or project directory: 1T.
• Between a research clusters and your on-campus computer:
  • to/from mass storage: Use
  • GLOBUS - or first copy the file to scratch and then use SLURM to copy from scratch to mass storage to scratch. - to/from home directory: 50g.
  • to/from scratch or project directory: 50g.
• Between a research clusters and an off-campus computer: This is highly variable depending on the speed of your internet connect. We commend using GLOBUS because it is also harder to hold a long connection off-campus.