TRANSFERRING FILES

For transfers from your desktop or home computer, or another computer external to Research Computing, to one of the Research Computing, there are several methods:

WHEN IS FILE TRANSFER SO BIG TO BE CONSIDERED A BIG FILE TRANSFER?

The file transfer size required to make it count as a big transfer is rapidly increasing as technology and network bandwidth improves. Data transfer times are fastest for copying a file between two UNC research clusters, then copying between a research cluster and your on-campus computer and significantly slower between a research cluster and your off-campus computer (due to the VPN and your internet connection). The slowest is copying a file from a media (ex: CD, DVD) onto your computer hard drive [to then copy to a research cluster]. Because of this, listing any explicit size determination for a big file is problematic. One solution is to treat every file as a large file and use GLOBUS FOR EVERY COPY. THIS WILL WORK, BUT REQUIRES MORE OVERHEAD THAN SOME USERS FIND PRACTICAL FOR THEIR WORK FLOW. INSTEAD, HERE IS A GUIDELINE BASED ON TIME: IF THE COPY COMMAND (CP, SCP, SFTP, ETC.) is going to take more that 10 minutes, then treat the copy as a big file transfer. If it is less than a minute, then it is a small transfer. Medium-sized transfers are, therefore 1-10 minutes long. As of this writing, this roughly translates to the following cutoff size between medium & big files:

  • Between 2 research clusters:
    • any directory to mass storage: Use
    • SLURM or GLOBUS. - home directory to home directory: 200g.
    • home directory to/from scratch or project directory: 500g.
    • scratch or project directory to scratch or project directory: 1T.
  • Between a research clusters and your on-campus computer:
    • to/from mass storage: Use
    • GLOBUS - or first copy the file to scratch and then use SLURM to copy from scratch to mass storage to scratch. - to/from home directory: 50g.
    • to/from scratch or project directory: 50g.
  • Between a research clusters and an off-campus computer: This is highly variable depending on the speed of your internet connect. We commend using GLOBUS because it is also harder to hold a long connection off-campus.

 

Last Update 7/15/2024 7:10:52 PM