Archive for 3rd May 2007

I JUST SPENT NINE HOURS SETTING UP RSYNC WITH SSH

WARNING: Extreme computer geek post!!

In the near future, someone will come to this page because they are trying to setup rsync using SSH authentication. The documentation out there isn’t that great, and the process is rather complex. I just spent my entire day setting this up, so I am going to share how you can do this too. Thank me.

Step By Step – Setting up rsync!

Some assumptions:

  • Assume [S] is the SOURCE that you are syncing FROM (where your files are)
    • [S-IP] is the IP of [S]
  • Assume [D] is the DESTINATION that the code is being pulled to (where your files will be)
    • [D-IP] is the IP of [D]
  • It is assumed the rsync script is run FROM [D].
  • Assume the user [USER] will be used to do the syncing. (DO NOT USE ROOT)

Now blindly follow these instructions:

  1. Install rsync. It is highly advised you install as new a version as you can find that is past 2.6.4.
  2. Create a [USER] (or whatever you want to call it) user on both servers.
  3. If it doesn’t exist already, make sure the home directory (i.e., /home/[USER]/) has a .ssh directory. This directory should have chmod 700
  4. On [D], go to login as [USER] and go to your home directory. Now make the SSH public keys:
    1. Type in ssh-keygen -t dsa -b 2048 -f rsynckey. Do not use a passphrase.
    2. Edit rsynckey.pub
      1. At the very beginning of the file, add in (no line breaks and directly before the stuff that’s already there) from=”[D-IP]“,command=”/home/utility/valid_rsync_commands.sh” (with a space between this and the old stuff)
    3. Copy rsynckey.pub to [S] in the /home/[USER]/ folder (type: scp rsynckey.pub [USER]@[S-IP]:/home/[USER]/).
    4. Type in mv rsynckey ./.ssh/rsynckey
    5. Type in mv rsynckey.pub ./.ssh/rsynckey.pub
    6. If they aren’t already there on [S], add the two following lines to /etc/services
      1. rsync 873/tcp
      2. rsync 873/udp
  5. Login as [USER] on [S]. Make sure you are in the home directory (/home/[USER]/).
  6. Run chmod 700 ./valid_rsync_commands.sh
  7. Run cat ./rsynckey.pub >> ./.ssh/authorized_keys
  8. Run chmod 600 ./.ssh/authorized_keys
  9. Run rm rsynckey.pub
  10. create 3 files:
    1. rsycd.conf (chmod 600)
      1. Example contents:

        pid file = /home/[USER]/rsyncd.pid
        log file = /home/[USER]/rsync.log
        use chroot = no
        read only = no
        list = false
        uid = nobody
        gid = nobody
        hosts deny = *
        secrets file = /home/[USER]/rsyncd.secrets
        [sometitle]
        path = /var/www/mycode
        comment = Sync my folder
        auth users = syncuser
        hosts allow = [D-IP]

    2. rsyncd.secrets (chmod 600) Add in a single line that says (no spaces): syncuser:mypassword

    3. valid_rsync_commands.sh (chmod 700)

      1. Put in the following code:

        #!/bin/sh case “$SSH_ORIGINAL_COMMAND” in
        *\&*)
        echo “Rejected”
        ;;
        *\(*)
        echo “Rejected”
        ;;
        *\{*)
        echo “Rejected”
        ;;
        *\;*)
        echo “Rejected”
        ;;
        *\<*)
        echo “Rejected”
        ;;
        *\`*)
        echo “Rejected”
        ;;
        rsync\ –server*)
        $SSH_ORIGINAL_COMMAND
        ;;
        *)
        echo “Rejected”
        ;;
        esac

  11. On [D], create the following files:

    1. rsync.exclude (chmod 600)
      1. Put in your exclusion rules in this file. Here’s my personal list of file types I exclude from my synching (the minus signs are on purpose, and should be included in the file):

        - .svn/
        - no_sync.*
        - *.pdf
        - *.log
        - *.bak
        - *.tmp
        - *.sh
        - *.zip
        - *.gz
        - *.tar
        - *.txt
        - *.cs
        - pdf/
        - *.ini
        - *~
        - .*

    2. sync.sh (chmod 700)
      1. Paste

        rsync –exclude-from=”exclude.sync” –progress –stats –compress –rsh=”/usr/bin/ssh -l [USER] -i /home/[USER]/.ssh/rsynckey” –archive –delete-after –delay-updates syncuser@[S-IP]::sometitle /var/www/destination

Explanation

Without going into too much details, the configurations I list here are the final results after comparing everything I could find on this page that talks about SSH authentication, this page on rsync daemon, this page on rsyncd.conf, and this page on the rsync client.

When you run the sync.sh script (by typing in ./sync.sh on [D]), it will:

  1. compress the data before pulling it.
  2. look in the exclude.sync folder for any exclusion rules.
  3. login to [S-IP] using the username [USER].
  4. not prompt you for the password of [USER].
  5. use the private key we created called rsynckey, found in /home/[USER]/.ssh/ on [D].
  6. know where to copy from based on the “sometitle” in the URL of the rsync command.
  7. authenticate within rsync using the username syncuser.
  8. prompt you for a password that you specified in rsyncd.secrets (I put it down as “mypassword”).
  9. copy all files on [S]:/var/www/mycode to [D]:/var/www/destination.
    1. The source is specified in the rsyncd.conf file.
    2. The destination is specified in that last block of code above.
  10. keep your [S] server relatively safe since the key can only be used to run rsync commands. Plus, no passwords are stored in the clear.
  11. Block the syncing of files that start with a period, PDFs, log files, tmp files, etc. Edit this as your please.

Now, slowly troubleshoot and fix this up. When trying to test, keep an eye on [D-IP]:/home/[USER]/rsync.log. This file will show you error messages when rsync is acting weird. If you keep getting errors and that file isn’t filling up, it means SSH authentication is failing — make sure you follow these instructions correctly. If you really wanted to run the sync process from [S] to [D], you have to switch all of the public key steps around and make sure all of the files appear in opposite locations. For example, the conf file on [D] would dictate which files get overwritten, rather than read.

REMEMBER: Word Press converts my quotes into its own slanted versions. Make sure you change them back to their original non-slanty quotation marks!