Linux: incremental backup using rsync on btrfs with snapshots
My production data are backed up by bacula. This is good for data "online" all the time, but it is not so convent for my notebook and media PC. Till now I was using only simple rsync script to "copy" everything important to USB drive or iSCSI LUN. It was working fine but hat one catch. There is no way how to retrieve older version of files.
I would like to have possibility to retrieve also older versions of files so I combined my rsync backup with btrfs and snapshot feature. Here is result:
Contents
Before first backup
I am using iSCSI lun as target for backup. Skip iSCSI part if you are using USB drive.
iSCSI configuration
It is important to password protect Target. Otherwise any device on local network can overwrite it. Also I highly recommend Header and Data digest. There is possibility of data corruption without it.
Configuration of iSCSI initiator is stored in /etc/iscsi/iscsid.conf. Following lines are mandatory:
node.session.auth.authmethod = CHAP node.session.auth.username = <user> node.session.auth.password = <pass> node.conn[0].iscsi.HeaderDigest = CRC32C node.conn[0].iscsi.DataDigest = CRC32C
All others can be left in "default" configuration.
iSCSI discovery and login
It is necessary to discover iSCSI target and log to it before system can access drive. It can be performed by following commands (replace "narnia-nas" and name of target with name used by your storage):
# iscsiadm -m discovery -t st -p narnia-nas 10.x.x.x:3260,1 iqn.2000-01.com.synology:narnia-nas.lukas # iscsiadm -m node -T iqn.2000-01.com.synology:narnia-nas.lukas --login Logging in to [iface: default, target: iqn.2000-01.com.synology:narnia-nas.lukas, portal: 10.x.x.x,3260] (multiple) Login to [iface: default, target: iqn.2000-01.com.synology:narnia-nas.lukas, portal: 10.x.x.x,3260] successful.
There should be new disk once kernel detect it (snip from dmesg command):
scsi host4: iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access SYNOLOGY iSCSI Storage 4.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 43 00 10 08 sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA sd 4:0:0:0: [sdb] Attached SCSI disk
# lsscsi | grep SYN [4:0:0:0] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdb
File system
I prefer disk labels as device file may change over time. Make sure you select correct device:
# mkfs.btrfs -L lukas-backup /dev/sdb btrfs-progs v4.0 See http://btrfs.wiki.kernel.org for more information. Performing full device TRIM (100.00GiB) ... Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 Turning ON incompat feature 'skinny-metadata': reduced-size metadata extent refs fs created label lukas-backup on /dev/sdb nodesize 16384 leafsize 16384 sectorsize 4096 size 100.00GiB
Note: TRIM on iSCSI LUN is feature of DSM 6.0 running on selected models of NAS from Synology. It may be available also on NAS devices from different vendors.
FS first time mounted
File system will be mounted by script therefore it is not necessary for system itself to mount it during boot up. Entry in /etc/fstab should looks like:
# backup LABEL=lukas-backup /mnt/backup btrfs noauto 0 0
Btrfs can be managed only while it is mounted so it is necessary to mount it first.
mkdir -p /mnt/backup chmod 0 /mnt/backup mount /mnt/backup
Subvolume & quota
Subvolume and snapshot size are internally handled as quota. In order to see which snapshot or subvolume take so much space it is necessary to enable quota on file system level:
btrfs quota enable /mnt/backup
Note: enable quota before placing data on fs otherwise it will take some time to collect (scan) it.
I prefer to use separate subvolume for backup. With this configuration it is possible to use one filesystem as target for different backups:
# btrfs sub create /mnt/backup/lukas Create subvolume '/mnt/backup/lukas'
Umount & logoff
My script is handling iSCSI initiator and mounting of fs. To have clean start I executed following commands at this point:
# umount /mnt/backup/ # iscsiadm -m node -T iqn.2000-01.com.synology:prue-nas.phoebe --logout Logging out of session [sid: 1, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260] Logout of [sid: 1, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260] successful.
Backup script
This script don't try to be smart. It simply execute commands and end immediately if failure occur.
Note: don't use this script directly. It is created to fulfill my needs and it may be necessary to modify it before you can use it in your environment.
#!/bin/bash
echo
echo "To verify using --checksum execute:"
echo "${0} verify"
echo
if [ "${1}" == "verify" ]
then
echo "verify mode selected"
opt="--checksum"
else
echo "normal mode selected"
opt=""
fi
echo
echo
echo "Discovering prue-nas"
iscsiadm -m discovery -t st -p prue-nas || exit 1
echo
echo "Login to prue-nas"
iscsiadm -m node -T iqn.2000-01.com.synology:prue-nas.phoebe --login || exit 2
echo
echo "Sleeping ... kernel need some time to detect new devices"
sleep 10
echo
echo "Mounting backup fs"
mount -v /mnt/backup || exit 3
echo
echo "Creating snapshot of backup image"
btrfs sub snap -r /mnt/backup/phoebe /mnt/backup/phoebe_$(LANG=C date +%Y-%m-%d_%s) || exit 4
echo
echo "Copying data"
( echo; echo "lsusb:"; lsusb; echo; echo "lsscsi:"; lsscsi; echo; echo "date:"; date; ) > /_lastbackup
rsync ${opt} --archive --delete --delete-excluded --human-readable --stats --progress \
--exclude=/mnt --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp \
--exclude=/data --exclude=/lib64/rc/init.d \
/ /mnt/backup/phoebe
RC=$?
echo
echo "Done with rc: ${RC}"
echo
echo "Flushing file system buffers"
time sync
echo
echo "Hit enter to continue: "
read
echo "Umounting backup filesystem"
umount -v /mnt/backup || exit 5
echo
echo "Logout from prue-nas"
iscsiadm -m node -T iqn.2000-01.com.synology:prue-nas.phoebe --logout | exit 6
echo
echo "Done"
# EOF
md5sum: b66d8a1d8d284a710185b436e3dbd059
Useful btrfs commands
"filesystem df"
BTRFS use different data layout. Regular df command is not aware about this structure so show only overall details. Use BTRFS specific commands to get more details:
# btrfs filesystem df /mnt/backup/ Data, single: total=33.01GiB, used=32.99GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00 Metadata, DUP: total=1.25GiB, used=939.14MiB Metadata, single: total=8.00MiB, used=0.00
Space is allocated in chunks. Total -> size of chunks allocated. Used -> data used by files stored on file system.
BTRFS can show size of subvolume and snapshot but it is not straightforward. It require two steps:
- First step is to get ID of subvolume:
# btrfs sub list /mnt/backup/ ID 256 gen 200 top level 5 path phoebe ID 293 gen 98 top level 5 path phoebe_2014-02-21_1392991593 ID 294 gen 171 top level 5 path phoebe_2014-02-21_1392994197 ID 295 gen 179 top level 5 path phoebe_2014-02-21_1392997683 ID 296 gen 183 top level 5 path phoebe_2014-02-21_1392998450
- Second step is list of quota groups:
# btrfs qgroup show /mnt/backup/ qgroupid rfer excl -------- ---- ---- 0/5 16384 16384 0/256 43434659840 7568146432 0/293 35893178368 446140416 0/294 35894276096 7098368 0/295 35895332864 147456 0/296 35895332864 147456
First number represent subvolume id (snapshot is also subvolume). Second one is size of it (in mean of all data accessible via subvolume). Third one show data exclusively stored in this group. This is actually difference between subvolumes (snapshots) which can be recovered by removing it.
Values in last table are in bytes so after converting you can see:
0/5 0M 0M 0/256 41422M 7217M 0/293 34230M 425M 0/294 34231M 6M 0/295 34232M 0M 0/296 34232M 0M
As a example -> By removing "0/293" I can get back 425M.
Subvolume (snapshot) remove
Snapshots and subvolumes are removed in same way:
# btrfs sub delete /mnt/backup/phoebe_2014-02-21_1392988539 Delete subvolume '/mnt/backup/phoebe_2014-02-21_1392988539'
It may take some time till BTRFS release all data used by snapshot or subvolume.
File system extend
I am using iSCSI LUN so I can extend it. Here is way how to extend also BTRFS on it:
# btrfs filesystem resize max /mnt/backup/ Resize '/mnt/backup/' of 'max'
This command will extend all attached devices in btrfs to reflect maximum HW size (for example after LUN extend)
