Introduction to rsync
#
rsync is an open-source, fast, multifunctional tool for local or remote data synchronization and backup that supports both full and incremental backups.
Introduction to rsync
#
The full English name of rsync
is Remote synchronization service software
, abbreviated as rsync
.
rsync
is an open-source, fast, multifunctional tool that can achieve full and incremental local or remote data synchronization (copy) backup, making it an excellent tool for file synchronization, often referred to as the fire blue dagger.
rsync
is a data mirroring backup tool for Linux systems. Using the fast incremental backup tool Remote Sync
, it can synchronize remotely, support local copying, or synchronize with other SSH
or rsync
hosts.
Currently, it supports cross-platform data synchronization between Windows
and Linux
.
Features of rsync
#
rsync
supports many features:
- Can mirror the entire directory tree and filesystem files.
- Can easily maintain the original file permissions, timestamps, soft and hard links, etc.
- Can be installed without special permissions.
- Fast: During the first synchronization,
rsync
will copy all content, but subsequent transfers will only transmit modified and newly created files.rsync
can perform compression and decompression during file transfer, thus using less bandwidth and reducing network pressure. - Secure: Files can be transferred using
scp
,SSH
, etc., and can also be done through directsocket
connections. - Supports anonymous transfers for easy website mirroring.
Summary: A singlersync
command integrates the functionalities ofscp
,cp
, andrm
, but is more flexible thanscp
,cp
, andrm
.
- Fast: During the first synchronization,
Three Modes of Operation for rsync
#
local
local mode --cp
Access via remote shell
access via remoteshell
--scp
daemon
daemon mode -- most commonly used
Modes of File Synchronization with rsync
#
- Full: Transfer all data to overwrite.
- Incremental: Only transfer the differences;
rsync
uses a uniquequick check
algorithm to achieve incremental data transfer.
Functionality of rsync
#
- Similar to the
cp
command — local backup transfer of data. - Similar to the
scp
command — remote backup transfer of data. - Similar to the
rm
command — achieve no-difference synchronization backup. - Similar to the
ls
command — view local file information.
Authentication Protocols for rsync
#
Before using the rsync
command to synchronize files, you need to log into the remote
host. There are two protocols used during the authentication process:
ssh
protocolrsync
protocol
In regular use, the most commonly used method isrsync-daemon
.
rsync
Authentication (rsync-daemon)#
rsync
listens on the defaultTCP
port873
under thersync-daemon
authentication method;- The
rsync-daemon
authentication method is the main authentication method forrsync
, and it is also the one we frequently use; - Only in this mode can
rsync
write the password to a file.
Note: Thersync-daemon
authentication method requires both the server and client to have thersync
service installed, and only thersync
server needs to startrsync
, while configuring thersync
configuration file. The client does not need to start thersync
service, and it does not affect the normal synchronization process.
ssh
Authentication#
- Under
ssh
authentication,rsync
can authenticate through system users, meaning that data is transferred via anssh
tunnel, similar to thescp
tool; - At this point, the synchronization operation is not limited to the synchronization folders defined in
rsync
; - There is no need to use port
873
for transmission.
Note: Thessh
authentication method does not require the server and client to configure thersync
configuration file; both parties only need to have thersync
service installed, and neither party needs to startrsync
.
#
The rsync server does not need to start the rsync daemon process; as long as you obtain the username and password of the remote host, you can directly rsync to synchronize files.
#
Since the rsync server does not start the daemon process, there is no need for the configuration file /etc/rsyncd.conf.
The ssh
authentication protocol works similarly to scp
. If you do not want to enter a password during synchronization, use ssh-keygen -t -rsa
to establish a tunnel.
//This method omits -e ssh by default and is equivalent to the following:
rsync -avz /SRC -e ssh [email protected]:/DEST
-a //File owner changes, timestamp remains the same
-V //Display detailed information during the process
-z //Compress data during transmission
//When needing to change the port, we can:
#Change the port of the ssh protocol, default is 22
rsync -avz /SRC -e "ssh -p2222" [email protected]:/DEST
rsync
Command#
Installing the rsync
Command#
Generally, RedHat
and CentOS
come with this tool. If not, you can install it directly using yum install -y rsync
.
#Check which package the rsync command requires
[root@node1 ~]# yum provides */bin/rsync
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:00:27 ago on Mon 10 May 2021 03:27:01 PM CST.
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo : @System
Matched from:
Filename : /usr/bin/rsync
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo : base
Matched from:
Filename : /usr/bin/rsync
#Install the rsync command
[root@node1 ~]# yum -y install rsync
#Installation successful
[root@node1 ~]# which rsync
/usr/bin/rsync
Format of the rsync
Command#
//Common formats of the rsync command are as follows:
rsync [OPTION]... SRC DEST
rsync [OPTION]... SRC [USER@]HOST:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
//Corresponding to the above three command formats, rsync has three different working modes:
1) Copy local files. This mode is activated when neither SRC nor DEST path information contains a single colon ":" separator.
#Command example
[root@localhost ~]# rsync -avz abc /opt/123
2) Use a remote shell program (like rsh, ssh) to copy content from the local machine to the remote machine. This mode is activated when the DST path contains a single colon ":" separator.
#Command example
[root@localhost ~]# ssh [email protected] 'ls -l /root'
3) Use a remote shell program (like rsh, ssh) to copy content from the remote machine to the local machine. This mode is activated when the SRC path contains a single colon ":" separator.
#Command example
[root@localhost ~]# rsync -avz [email protected]:/etc/yum.repos.d /root/
Details of rsync
Command Parameters#
-v, --verbose Detailed output mode
-q, --quiet Minimal output mode
-c, --checksum Enable checksum switch, forcing file transfer verification
-a, --archive Archive mode, indicating recursive file transfer while preserving all file attributes, equivalent to -rlptgoD
-r, --recursive Process subdirectories recursively
-R, --relative Use relative path information
-b, --backup Create backups, meaning that when a file with the same name already exists at the destination, the old file will be renamed to ~filename. You can use the --suffix option to specify a different backup file prefix.
--backup-dir Store backup files (like ~filename) in a specified directory.
-suffix=SUFFIX Define the backup file prefix
-u, --update Only update, meaning skip all files that already exist in DST and have a later timestamp than the file to be backed up. (Do not overwrite updated files)
-l, --links Preserve soft links
-L, --copy-links Treat soft links as regular files
--copy-unsafe-links Only copy links pointing outside the SRC path directory tree
--safe-links Ignore links pointing outside the SRC path directory tree
-H, --hard-links Preserve hard links
-p, --perms Preserve file permissions
-o, --owner Preserve file owner information
-g, --group Preserve file group information
-D, --devices Preserve device file information
-t, --times Preserve file timestamp information
-S, --sparse Special handling of sparse files to save space in DST
-n, --dry-run Show which files will be transferred
-W, --whole-file Copy files without performing incremental checks
-x, --one-file-system Do not cross filesystem boundaries
-B, --block-size=SIZE Block size used for the checksum algorithm, default is 700 bytes
-e, --rsh=COMMAND Specify using rsh or ssh for data synchronization
--rsync-path=PATH Specify the path of the rsync command on the remote server
-C, --cvs-exclude Automatically ignore files using the same method as CVS, to exclude files that you do not want to transfer
--existing Only update files that already exist in DST, without backing up newly created files
--delete Delete files in DST that are not in SRC
--delete-excluded Also delete files on the receiving end that are specified to be excluded by this option
--delete-after Delete after the transfer is complete
--ignore-errors Delete even if there are IO errors
--max-delete=NUM Delete at most NUM files
--partial Keep files that were not fully transferred for faster subsequent transfers
--force Force deletion of directories, even if not empty
--numeric-ids Do not match numeric user and group IDs to usernames and group names
--timeout=TIME IP timeout, in seconds
-I, --ignore-times Do not skip files with the same timestamp and length
--size-only When deciding whether to back up files, only check file size without considering file time
--modify-window=NUM Determine the timestamp window used when files have the same time, default is 0
-T --temp-dir=DIR Create temporary files in DIR
--compare-dest=DIR Compare files in DIR to determine whether backup is needed
-P Equivalent to --partial
--progress Display the backup process
-z, --compress Compress files during transmission
--exclude=PATTERN Specify patterns of files to exclude from transfer
--include=PATTERN Specify patterns of files to include for transfer
--exclude-from=FILE Exclude files specified by patterns in FILE
--include-from=FILE Include files matching patterns specified in FILE
--version Print version information
--address Bind to a specific address
--config=FILE Specify an alternative configuration file, not using the default rsyncd.conf file
--port=PORT Specify an alternative rsync server port
--blocking-io Use blocking IO for remote shell
-stats Provide transfer status for certain files
--progress Display transfer progress
--log-format=formAT Specify log file format
--password-file=FILE Get password from FILE
--bwlimit=KBPS Limit I/O bandwidth, KBytes per second
-h, --help Display help information
rsync
Configuration#
rsync
can be used in three ways:
- Mode One:
local
local mode - Mode Two:
Access via remote shell
access via remoteshell
- Mode Three:
daemon
daemon mode (most commonly used)
Mode One: Local Mode#
Local mode can be used directly with the command.
# Command format
rsync [OPTION...] SRC... [DEST]
Mode Two: Access via Remote shell
#
This mode generally involves pushing and pulling via remote shell
commands from the local machine.
# Command format
# Pull
rsync [OPTION...] [USER]@HOST:SRC... [DEST]
# Push
rsync [OPTION] SRC... [USER]@HOST:DEST
Note: Both the accessing and accessed ends need to have the rsync
command installed
Explanation: When transferring files, the file checksums in the source and destination directories are compared first, and only when the checksums differ will the transfer occur.
Key Point: In practical scenarios, rsync+ssh
key authentication is usually used to enable passwordless login.
Mode Three: Daemon Mode#
Daemon mode allows for real-time synchronization across different locations. Compared to the first two modes, it is more complex and powerful.
Note: This mode requires the source server to have the application: rsync
+ inotify-tools
or rsync
+ sersync
tools installed; the target server only needs to have rsync
installed.