2. File Transfer
Using scp is the easiest method to use when transferring single files.
Local File to Remote Host
% scp localfile user@remotehost:/destination/dir/or/filename
Remote Host to Local File
% scp user@remotehost:/remote/filename localfile
One may find this mode very similar to the interactive interface offered. A login session may look similar to the following:
% sftp user@remotehost (enter in password) ... sftp>
The commands are similar to those offered by the outmoded ftp client programs: get, put, cd, pwd, lcd, etc. For more information on the available set of commands, one should consult sftp the man page.
% man sftp
One may use sftp interactively in two cases.
Case 1: Pull a remote file to the local host.
% sftp user@remotehost:/remote/filename localfilename
Case 2: Creating a special sftp batch file containing the set of commands one wishes to execute with out any interaction.
% sftp -b batchfile user@remotehost
Additional information on constructing a batch file is available in the sftp man page.
2.3. rsync Over SSH (preferred)
rsync is an extremely powerful program; it can synchronize entire directory trees, only sending data about files that have changed. That said, it is rather picky about the way it is used. The rsync man page has a great deal of useful information, but the basics are explained below.
Single File Synchronization
To synchronize a single file via rsync, use the following:
To send a file:
% rsync --rsh=ssh --archive --stats --progress localfile \ username@remotehost:/destination/dir/or/filename
To receive a file:
% rsync --rsh=ssh --archive --stats --progress \ username@remotehost:/remote/filename localfilename
Note that --rsh=ssh is not necessary with newer versions of rsync, but older installs will default to using rsh (which is not generally enabled on modern OSes).
To synchronize an entire directory, use the following:
To send a directory:
% rsync --rsh=ssh --archive --stats --progress localdir/ \ username@remotehost:/destination/dir/
% rsync --rsh=ssh --archive --stats --progress localdir \ username@remotehost:/destination
To receive a directory:
% rsync --rsh=ssh --archive --stats --progress \ username@remotehost:/remote/directory/ /some/localdirectory/
% rsync --rsh=ssh --archive --stats --progress \ username@remotehost:/remote/directory /some/
Note the difference with the slashes. The second command will place the files in the directory /destination/localdir; the fourth will place them in the directory /some/directory. rsync is very particular about the placement of slashes. Before running any significant rsync command, add --dry-run to the parameters. This will let rsync show you what it plans on doing without actually transferring the files.
Synchronization with Deletion
This is very dangerous; a single mistyped character may blow away all of your data. Do not synchronize with deletion if you aren't absolutely certain you know what you're doing.
To have directory synchronization delete files on the destination system that don't exist on the source system:
% rsync --rsh=ssh --archive --stats --dry-run --progress \ --delete localdir/ username@remotehost:/destination/dir/
Note that the above command will not actually delete (or transfer) anything; the --dry-run must be removed from the list of parameters to actually have it work.
Using BBCP to transfer large data files without encryption.
% bbcp [opt] user@source:/path/to/data user@destination:/path/to/store/data
Possible options include:
- -P 2
- Give a progress report every 2 seconds
- - w 2M
- TCP window size of 2MBytes
- -s 16
- Set the number of streams to 16 (default is 4)
Other options may be necessary if bbcp is not installed in a regular location on either end of the transfer. This can lead to rather complex command lines:
$ bbcp -z -T \ "ssh -x -a -oFallBackToRsh=no %I -l %U %H /home/user/Custom/bin/bbcp" \ foobar-5.4.14.tbz "firstname.lastname@example.org:foo.tbz"
2.5 Client Software
scp and sftp
The command-line scp and sftp tools come with any modern distribution of OpenSSH; this is generally installed by default on modern Linux, UNIX, and Mac OS X installs.
Windows clients include:
(puTTY-related command line utilities), and
- scp, sftp, & rsync as provided by Cygwin.
2.6. Globus (**XSEDE users only**)
Since direct ssh is not available to them, XSEDE users must use Globus to transfer data from SuperMIC to other XSEDE resources or their own local computers.
In order to use Globus, a user first needs to create a Globus account at the Globus website. Optionally, the user can link the Globus account to the XSEDE user portal account, which will enable the access to Globus services using the XSEDE credentials.
There are two options to transfer data using Globus: Graphic User Interface (either with a browser or Globus Connect client) and GridFTP (command line interface).
2.6.1 Globus GUI
Users can use a browser to transfer data between two Globus endpoints at the Globus data transfer interface. All XSEDE resources have been configured as Globus endpoints.
The snapshot below is an example of transferring data from Comet at SDSC to SuperMIC:
Alternatively, one can use the Globus Connect Personal client to transfer data.
2.6.2 GridFTP (globus-url-copy)
globus-url-copy is a command line data tranfer tool that uses the GridFTP protocol. The XSEDE documentation can found here .
The GridFTP endpoint for Super MIC is:
The example below transfers a file from Comet to SuperMIC:
$ globus-url-copy -vb gsiftp://oasis-dm.sdsc.edu:2811/home/leyan/demo_data.tgz gsiftp://smic1.hpc.lsu.edu:2811/project/lyan1/ Source: gsiftp://oasis-dm.sdsc.edu:2811/home/leyan/ Dest: gsiftp://smic1.hpc.lsu.edu:2811/project/lyan1/ demo_data.tgz 2062098422 bytes 58.88 MB/sec avg 61.64 MB/sec inst
Please refer to the GridFTP User Guide for detailed description for various options available for globus-url-copy.
Users may direct questions to email@example.com.