Linux: incremental backup using rsync on btrfs with snapshots

My production data are backed up by bacula. This is good for data "online" all the time, but it is not so convent for my notebook and media PC. Till now I was using only simple rsync script to "copy" everything important to USB drive or iSCSI LUN. This solution was working fine but hat one catch. There is no way how to retrieve older version of files.

I would like to have possibility to retrieve also older versions of files so i combined my rsync backup with btrfs and snapshot feature. Here is result:

Before first backup

I am using iSCSI lun as target for backup. Skip iSCSI part if you are using USB drive.

iSCSI configuration

It is important to password protect Target. Otherwise any device on local network can overwrite it. Also I highly recommend Header and Data digest. There is possibility of data corruption without it.

Configuration of iSCSI initiator is stored in /etc/iscsi/iscsid.conf. Following lines are mandatory:

node.session.auth.authmethod = CHAP
node.session.auth.username = <user>
node.session.auth.password = <pass>

node.conn[0].iscsi.HeaderDigest = CRC32C
node.conn[0].iscsi.DataDigest = CRC32C

All others can be left in "default" configuration.

iSCSI discovery and login

It is necessary to discover iSCSI target and log to it before system can access drive. It can be performed by following commands:

# iscsiadm -m discovery -t st -p prue-nas
10.xx.xx.5:3260,0 iqn.2000-01.com.synology:prue-nas.phoebe
10.xx.xx.5:3260,0 iqn.2000-01.com.synology:prue-nas.lukas

# iscsiadm -m node -T iqn.2000-01.com.synology:prue-nas.phoebe --login
Logging in to [iface: default, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260]
Login to [iface: default, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260] successful.

There should be new disk once kernel detect it:

scsi8 : iSCSI Initiator over TCP/IP
scsi 8:0:0:0: Direct-Access     SYNOLOGY iSCSI Storage    3.1  PQ: 0 ANSI: 5
sd 8:0:0:0: Attached scsi generic sg2 type 0
sd 8:0:0:0: [sdb] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
sd 8:0:0:0: [sdb] Write Protect is off
sd 8:0:0:0: [sdb] Mode Sense: 3b 00 00 00
sd 8:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 8:0:0:0: [sdb] Attached SCSI disk

# lsscsi | grep SYN
[6:0:0:0]    disk    SYNOLOGY iSCSI Storage    3.1   /dev/sdb 

File system

I prefer disk labels as device file may change over time. Make sure you select correct device:

# mkfs.btrfs -L phoebe-backup /dev/sdb

Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
fs created label phoebe-backup on /dev/sdb
	nodesize 16384 leafsize 16384 sectorsize 4096 size 50.00GiB
Btrfs v3.12

FS first time mounted

File system will be mounted by script therefore it is not necessary for system itself to mount it during boot up. To reflect this entry in /etc/fstab should looks like:

# data backup
LABEL=phoebe-backup		/mnt/backup		btrfs	noauto		0 0

Btrfs can be managed only while it is mounted.

mkdir -p /mnt/backup
mount /mnt/backup

Subvolume & quota

Subvolume and snapshot size are internally handled as quota. In order to see which snapshot or subvolume take so much space it is necessary to enable quota on file system level:

btrfs quota enable /mnt/backup

Note: enable quota before placing data on fs otherwise it will take some time to collect (scan) it

I prefer to use separate subvolume for backup. With this configuration it is possible to use one filesystem as target for different backups:

# btrfs sub create /mnt/backup/phoebe
Create subvolume '/mnt/backup/phoebe'

Umount & logoff

My script is handling iSCSI initiator and mounting of fs. To have clean start I executed following commands at this point:

# umount /mnt/backup/

# iscsiadm -m node -T iqn.2000-01.com.synology:prue-nas.phoebe --logout
Logging out of session [sid: 1, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260]
Logout of [sid: 1, target: iqn.2000-01.com.synology:prue-nas.phoebe, portal: 10.xx.xx.5,3260] successful.

Useful btrfs commands

"No space left on device." What ?

First of all don't panic :o) At second don't use "df" command or it's derivatives. Btrfs have specific space management so "df" command can show misleading values.

Example:

• you have btrfs on two 10GB devices with one "single" subvolume and second one using "raid1"

Simple data usage:

• If you try to write to "single" subvolume then you may write up to 20GB of data (2x10GB, not counting metadata)
• If you try to write to "raid1" subvolume then you may write up to 10GB of data (again not counting metadata)

At this point it looks simple but check this:

• You stored 5GB of data to "single" subvolume and another set of 5GB data to "raid1" subvolume. What will be space left on fs ?

That's tricky question but here is answer:

• 5GB of "raid1" mean that there is 5GB of 10GB occupied on both devices
• 5GB of "single" mean two things. If everything is balanced properly then 2.5GB of space is taken from both devices. If not then up to 5GB is on one device and rest on second one.

This mean:

• you may store up to 2.5GB of "raid1" or up to 5GB in "single" subvolume

Now you understand why df can't tell you utilization details correctly ?

Additional problems may occur if data and metadata are way to fragmented. Refer to official wiki for more details: Help! I ran out of disk space!

"filesystem df"

To explain what is going on btrfs have "btrfs filesystem df" command:

# btrfs filesystem df /mnt/backup/
Data, single: total=33.01GiB, used=32.99GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=1.25GiB, used=939.14MiB
Metadata, single: total=8.00MiB, used=0.00

Space is allocated in chunks. This mean that "total" show size of allocated chunks and "used" real usage.

How to check size of subvolume or snapshot ? It is not possible without quota. If you don't have enabled yet then enable it and make sure you ran initial scan otherwise it may show you wrong details.

First of all it is necessary to check subvolume/snapshot id:

# btrfs sub list /mnt/backup/

Second step is list of quota groups:

# btrfs qgroup show /mnt/backup/

First number represent subvolume id (yes, snapshot is also subvolume). Second one is size of it (in mean of all data accessible via subvolume). Third one show data exclusively stored in this group. This is actually difference between subvolumes (snapshots) which can be recovered by removing it.

remove

1. btrfs sub delete phoebe_2014-02-21_1392988539 phoebe_2014-02-21_1392988706

Delete subvolume '/mnt/backup/phoebe_2014-02-21_1392988539' Delete subvolume '/mnt/backup/phoebe_2014-02-21_1392988706'