banner
十一

十一

Stay hungry, stay foolish.

The Fire Blue Dagger of File Synchronization: rsync

Introduction to rsync#

rsync is an open-source, fast, multifunctional tool for local or remote data synchronization and backup that supports both full and incremental backups.

Introduction to rsync#

The full English name of rsync is Remote synchronization service software, abbreviated as rsync.
rsync is an open-source, fast, multifunctional tool that can achieve full and incremental local or remote data synchronization (copy) backup, making it an excellent tool for file synchronization, often referred to as the fire blue dagger.
rsync is a data mirroring backup tool for Linux systems. Using the fast incremental backup tool Remote Sync, it can synchronize remotely, support local copying, or synchronize with other SSH or rsync hosts.
Currently, it supports cross-platform data synchronization between Windows and Linux.

Features of rsync#

rsync supports many features:

  • Can mirror the entire directory tree and filesystem files.
  • Can easily maintain the original file permissions, timestamps, soft and hard links, etc.
  • Can be installed without special permissions.
    • Fast: During the first synchronization, rsync will copy all content, but subsequent transfers will only transmit modified and newly created files. rsync can perform compression and decompression during file transfer, thus using less bandwidth and reducing network pressure.
    • Secure: Files can be transferred using scp, SSH, etc., and can also be done through direct socket connections.
    • Supports anonymous transfers for easy website mirroring.
      Summary: A single rsync command integrates the functionalities of scp, cp, and rm, but is more flexible than scp, cp, and rm.

Three Modes of Operation for rsync#

  • local local mode -- cp
  • Access via remote shell access via remote shell -- scp
  • daemon daemon mode -- most commonly used

Modes of File Synchronization with rsync#

  • Full: Transfer all data to overwrite.
  • Incremental: Only transfer the differences; rsync uses a unique quick check algorithm to achieve incremental data transfer.

Functionality of rsync#

  • Similar to the cp command — local backup transfer of data.
  • Similar to the scp command — remote backup transfer of data.
  • Similar to the rm command — achieve no-difference synchronization backup.
  • Similar to the ls command — view local file information.

Authentication Protocols for rsync#

Before using the rsync command to synchronize files, you need to log into the remote host. There are two protocols used during the authentication process:

  • ssh protocol
  • rsync protocol
    In regular use, the most commonly used method is rsync-daemon.

rsync Authentication (rsync-daemon)#

  • rsync listens on the default TCP port 873 under the rsync-daemon authentication method;
  • The rsync-daemon authentication method is the main authentication method for rsync, and it is also the one we frequently use;
  • Only in this mode can rsync write the password to a file.
    Note: The rsync-daemon authentication method requires both the server and client to have the rsync service installed, and only the rsync server needs to start rsync, while configuring the rsync configuration file. The client does not need to start the rsync service, and it does not affect the normal synchronization process.

ssh Authentication#

  • Under ssh authentication, rsync can authenticate through system users, meaning that data is transferred via an ssh tunnel, similar to the scp tool;
  • At this point, the synchronization operation is not limited to the synchronization folders defined in rsync;
  • There is no need to use port 873 for transmission.
    Note: The ssh authentication method does not require the server and client to configure the rsync configuration file; both parties only need to have the rsync service installed, and neither party needs to start rsync.
#
The rsync server does not need to start the rsync daemon process; as long as you obtain the username and password of the remote host, you can directly rsync to synchronize files.
#
Since the rsync server does not start the daemon process, there is no need for the configuration file /etc/rsyncd.conf.

The ssh authentication protocol works similarly to scp. If you do not want to enter a password during synchronization, use ssh-keygen -t -rsa to establish a tunnel.

//This method omits -e ssh by default and is equivalent to the following:
rsync -avz /SRC -e ssh [email protected]:/DEST 
    -a  //File owner changes, timestamp remains the same
    -V  //Display detailed information during the process
    -z  //Compress data during transmission

//When needing to change the port, we can:
#Change the port of the ssh protocol, default is 22
rsync -avz /SRC -e "ssh -p2222" [email protected]:/DEST

rsync Command#

Installing the rsync Command#

Generally, RedHat and CentOS come with this tool. If not, you can install it directly using yum install -y rsync.

#Check which package the rsync command requires
[root@node1 ~]# yum provides */bin/rsync
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:00:27 ago on Mon 10 May 2021 03:27:01 PM CST.
rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo        : @System
Matched from:
Filename    : /usr/bin/rsync

rsync-3.1.3-9.el8.x86_64 : A program for synchronizing files over a network
Repo        : base
Matched from:
Filename    : /usr/bin/rsync

#Install the rsync command
[root@node1 ~]# yum -y install rsync

#Installation successful
[root@node1 ~]# which rsync
/usr/bin/rsync

Format of the rsync Command#

//Common formats of the rsync command are as follows:
    rsync [OPTION]... SRC DEST
    rsync [OPTION]... SRC [USER@]HOST:DEST
    rsync [OPTION]... [USER@]HOST:SRC DEST

//Corresponding to the above three command formats, rsync has three different working modes:
1) Copy local files. This mode is activated when neither SRC nor DEST path information contains a single colon ":" separator.
#Command example
[root@localhost ~]# rsync -avz abc /opt/123
2) Use a remote shell program (like rsh, ssh) to copy content from the local machine to the remote machine. This mode is activated when the DST path contains a single colon ":" separator.
#Command example
[root@localhost ~]# ssh [email protected] 'ls -l /root'
3) Use a remote shell program (like rsh, ssh) to copy content from the remote machine to the local machine. This mode is activated when the SRC path contains a single colon ":" separator.
#Command example
[root@localhost ~]# rsync -avz [email protected]:/etc/yum.repos.d /root/

Details of rsync Command Parameters#

-v, --verbose         Detailed output mode
-q, --quiet           Minimal output mode
-c, --checksum        Enable checksum switch, forcing file transfer verification
-a, --archive         Archive mode, indicating recursive file transfer while preserving all file attributes, equivalent to -rlptgoD
-r, --recursive       Process subdirectories recursively
-R, --relative        Use relative path information
-b, --backup          Create backups, meaning that when a file with the same name already exists at the destination, the old file will be renamed to ~filename. You can use the --suffix option to specify a different backup file prefix.
--backup-dir          Store backup files (like ~filename) in a specified directory.
-suffix=SUFFIX        Define the backup file prefix
-u, --update          Only update, meaning skip all files that already exist in DST and have a later timestamp than the file to be backed up. (Do not overwrite updated files)
-l, --links           Preserve soft links
-L, --copy-links      Treat soft links as regular files
--copy-unsafe-links   Only copy links pointing outside the SRC path directory tree
--safe-links          Ignore links pointing outside the SRC path directory tree
-H, --hard-links      Preserve hard links
-p, --perms           Preserve file permissions
-o, --owner           Preserve file owner information
-g, --group           Preserve file group information
-D, --devices         Preserve device file information
-t, --times           Preserve file timestamp information
-S, --sparse          Special handling of sparse files to save space in DST
-n, --dry-run         Show which files will be transferred
-W, --whole-file      Copy files without performing incremental checks
-x, --one-file-system Do not cross filesystem boundaries
-B, --block-size=SIZE Block size used for the checksum algorithm, default is 700 bytes
-e, --rsh=COMMAND     Specify using rsh or ssh for data synchronization
--rsync-path=PATH     Specify the path of the rsync command on the remote server
-C, --cvs-exclude     Automatically ignore files using the same method as CVS, to exclude files that you do not want to transfer
--existing            Only update files that already exist in DST, without backing up newly created files
--delete              Delete files in DST that are not in SRC
--delete-excluded     Also delete files on the receiving end that are specified to be excluded by this option
--delete-after        Delete after the transfer is complete
--ignore-errors       Delete even if there are IO errors
--max-delete=NUM      Delete at most NUM files
--partial             Keep files that were not fully transferred for faster subsequent transfers
--force               Force deletion of directories, even if not empty
--numeric-ids         Do not match numeric user and group IDs to usernames and group names
--timeout=TIME        IP timeout, in seconds
-I, --ignore-times    Do not skip files with the same timestamp and length
--size-only           When deciding whether to back up files, only check file size without considering file time
--modify-window=NUM   Determine the timestamp window used when files have the same time, default is 0
-T --temp-dir=DIR     Create temporary files in DIR
--compare-dest=DIR    Compare files in DIR to determine whether backup is needed
-P                    Equivalent to --partial
--progress            Display the backup process
-z, --compress        Compress files during transmission
--exclude=PATTERN     Specify patterns of files to exclude from transfer
--include=PATTERN     Specify patterns of files to include for transfer
--exclude-from=FILE   Exclude files specified by patterns in FILE
--include-from=FILE   Include files matching patterns specified in FILE
--version             Print version information
--address             Bind to a specific address
--config=FILE         Specify an alternative configuration file, not using the default rsyncd.conf file
--port=PORT           Specify an alternative rsync server port
--blocking-io         Use blocking IO for remote shell
-stats                Provide transfer status for certain files
--progress            Display transfer progress
--log-format=formAT   Specify log file format
--password-file=FILE  Get password from FILE
--bwlimit=KBPS        Limit I/O bandwidth, KBytes per second
-h, --help            Display help information

rsync Configuration#

rsync can be used in three ways:

  • Mode One: local local mode
  • Mode Two: Access via remote shell access via remote shell
  • Mode Three: daemon daemon mode (most commonly used)

Mode One: Local Mode#

Local mode can be used directly with the command.

# Command format
rsync [OPTION...] SRC... [DEST]

Mode Two: Access via Remote shell#

This mode generally involves pushing and pulling via remote shell commands from the local machine.

# Command format
# Pull
rsync [OPTION...] [USER]@HOST:SRC... [DEST]

# Push
rsync [OPTION] SRC... [USER]@HOST:DEST

Note: Both the accessing and accessed ends need to have the rsync command installed
Explanation: When transferring files, the file checksums in the source and destination directories are compared first, and only when the checksums differ will the transfer occur.
Key Point: In practical scenarios, rsync+ssh key authentication is usually used to enable passwordless login.

Mode Three: Daemon Mode#

Daemon mode allows for real-time synchronization across different locations. Compared to the first two modes, it is more complex and powerful.
Note: This mode requires the source server to have the application: rsync + inotify-tools or rsync + sersync tools installed; the target server only needs to have rsync installed.

rsync + inotify-tools Method#

rsync + sersync Method#

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.