Replicating ZFS to FreeNAS

There are plenty of articles that discuss how to handle FreeNAS to FreeNAS replication, its even covered in the FreeNAS documentation (thank you iX Systems for both automatic and manual setups.

Whats not covered replicating something that isn’t FreeNAS, to a FreeNAS. I wanted something simple, that would manage the snapshots by cleaning up after itself. I found Zap in the ports tree. I liked that it was a shell script. This makes it really simple to run on FreeNAS. I don’t need to do any hacking.

Zap

Start by installing the port

pkg install sysutils/zap

The setup is pretty straight forward. Set the zap:snap property to ‘on’ for the filesystems you want to manage with Zap. Set it to off for those you want it to ignore. I don’t need to backup crashdumps, /var/tmp, or /var/mail on this system, but I do want to backup /usr, /var, and /iocage.

# zfs set zap:snap=on zroot/usr zroot/var/ zroot/iocage
# zfs set zap:snap=off zroot/var/crash zroot/var/tmp zroot/var/mail
# zap snap -v 2d

This last line will actually take the snapshots. You an view them with

[louisk@zproxy1 louisk 103 ]$ zfs list -t snapshot 
NAME                                                                          USED  AVAIL  REFER  MOUNTPOINT
zroot@ZAP_ns_2020-07-18T00:00:00p0000--2d                                        0      -    88K  -
zroot@ZAP_ns_2020-07-19T00:00:00p0000--2d                                        0      -    88K  -
zroot/ROOT@ZAP_ns_2020-07-18T00:00:00p0000--2d                                   0      -    88K  -
zroot/ROOT@ZAP_ns_2020-07-19T00:00:00p0000--2d                                   0      -    88K  -
zroot/ROOT/default@ZAP_ns_2020-07-18T00:00:00p0000--2d                       1.20M      -  1.62G  -
zroot/ROOT/default@ZAP_ns_2020-07-19T00:00:00p0000--2d                        424K      -  1.62G  -
zroot/tmp@ZAP_ns_2020-07-18T00:00:00p0000--2d                                  64K      -   136K  -
zroot/tmp@ZAP_ns_2020-07-19T00:00:00p0000--2d                                  64K      -   136K  -
zroot/usr@ZAP_ns_2020-07-18T00:00:00p0000--2d                                    0      -    88K  -
zroot/usr@ZAP_ns_2020-07-19T00:00:00p0000--2d                                    0      -    88K  -
zroot/usr/home@ZAP_ns_2020-07-18T00:00:00p0000--2d                               0      -  1.17G  -
zroot/usr/home@ZAP_ns_2020-07-19T00:00:00p0000--2d                               0      -  1.17G  -
zroot/usr/ports@ZAP_ns_2020-07-18T00:00:00p0000--2d                              0      -   724M  -
zroot/usr/ports@ZAP_ns_2020-07-19T00:00:00p0000--2d                              0      -   724M  -
zroot/usr/src@ZAP_ns_2020-07-18T00:00:00p0000--2d                                0      -   704M  -
zroot/usr/src@ZAP_ns_2020-07-19T00:00:00p0000--2d                                0      -   704M  -
zroot/var@ZAP_ns_2020-07-18T00:00:00p0000--2d                                    0      -    88K  -
zroot/var@ZAP_ns_2020-07-19T00:00:00p0000--2d                                    0      -    88K  -
zroot/var/audit@ZAP_ns_2020-07-18T00:00:00p0000--2d                              0      -    88K  -
zroot/var/audit@ZAP_ns_2020-07-19T00:00:00p0000--2d                              0      -    88K  -
zroot/var/crash@ZAP_ns_2020-07-18T00:00:00p0000--2d                              0      -    88K  -
zroot/var/crash@ZAP_ns_2020-07-19T00:00:00p0000--2d                              0      -    88K  -
zroot/var/log@ZAP_ns_2020-07-18T00:00:00p0000--2d                             292K      -  1020K  -
zroot/var/log@ZAP_ns_2020-07-19T00:00:00p0000--2d                             240K      -  1.05M  -
zroot/var/mail@ZAP_ns_2020-07-18T00:00:00p0000--2d                             84K      -   288K  -
zroot/var/mail@ZAP_ns_2020-07-19T00:00:00p0000--2d                             84K      -   304K  -
zroot/var/tmp@ZAP_ns_2020-07-18T00:00:00p0000--2d                                0      -    88K  -
zroot/var/tmp@ZAP_ns_2020-07-19T00:00:00p0000--2d                                0      -    88K  -
[louisk@zproxy1 louisk 104 ]$

You can see the date/timestamp that the snapshot was taken, and also on the end, how long it should be kept for. You should see snapshots for each of the filesystems you set the zap:snap=on property. I picked 2 days because its relatively short duration, but long enough that I can leave it overnight and see that its still doing the right thing the next day. I don’t want to get too far into this with out verifying that its working as expected.

The crontab entry looks like this

#minute	hour	mday	month	wday	command

# make snapshots, replicate, cleanup remote end
0	0	*	*	*	root	/usr/local/bin/zap snap 2d | logger
30	0	*	*	*	root	/usr/local/bin/zap rep -v | logger
0	6	*	*	*	root	/usr/local/bin/zap destroy | logger

The ‘| logger’ sends the output to syslog, in case I need to see whats happened. Now I check it the next day and make sure its creating and deleting snapshots properly.

Replication

If you’ve gotten this far, now its time to setup replication.

Create a passwordless ssh key for the root user. You will need to put the public key on your FreeNAS box.

Create a user that will accept the snapshots I’m using the zap user, for consistency. Make sure you define a home directory, as you will need to add an ssh public key and it needs to be stored there.

Create a dataset in your pool where you will store your snapshots

Set the ownership to the user you created above

root@freenas1[~]# zfs allow zap create,destroy,diff,mount,readonly,receive,release,send,userprop cess/zproxy1
root@freenas1[~]# zfs allow cess/zproxy1
---- Permissions on cess/zproxy1 -------------------------------------
Local+Descendent permissions:
	user zap create,destroy,diff,mount,readonly,receive,release,send,userprop
root@freenas1[~]#

In the web interface under System > Sysctls > Add sysctl:

Variable: vfs.usermount
Value: 1
Enabled: yes

Back on our FreeBSD machine, we can add the crontab entry to replicate things to our NAS.

#minute	hour	mday	month	wday	command

# replicate
30	0	*	*	*	root	/usr/local/bin/zap rep -v | logger

Looking at the NAS, we should be able to see snapshots showing up in the correct place:

root@freenas1[~]# zfs list -t snapshot | grep ZAP
NAME                                                                                  USED  AVAIL  REFER  MOUNTPOINT
cess/zproxy1@ZAP_ns_2020-07-19T00:00:00p0000--2d                                         0      -   188K  -
cess/zproxy1/ROOT@ZAP_ns_2020-07-19T00:00:00p0000--2d                                    0      -   188K  -
cess/zproxy1/ROOT/default@ZAP_ns_2020-07-19T00:00:00p0000--2d                            0      -  1.86G  -
cess/zproxy1/tmp@ZAP_ns_2020-07-19T00:00:00p0000--2d                                     0      -   324K  -
cess/zproxy1/usr@ZAP_ns_2020-07-19T00:00:00p0000--2d                                     0      -   188K  -
cess/zproxy1/usr/home@ZAP_ns_2020-07-19T00:00:00p0000--2d                                0      -  1.17G  -
cess/zproxy1/usr/ports@ZAP_ns_2020-07-19T00:00:00p0000--2d                               0      -  1.46G  -
cess/zproxy1/usr/src@ZAP_ns_2020-07-19T00:00:00p0000--2d                                 0      -  1.12G  -
cess/zproxy1/var@ZAP_ns_2020-07-19T00:00:00p0000--2d                                     0      -   188K  -
cess/zproxy1/var/audit@ZAP_ns_2020-07-19T00:00:00p0000--2d                               0      -   188K  -
cess/zproxy1/var/crash@ZAP_ns_2020-07-19T00:00:00p0000--2d                               0      -   188K  -
cess/zproxy1/var/log@ZAP_ns_2020-07-19T00:00:00p0000--2d                                 0      -  1.46M  -
cess/zproxy1/var/mail@ZAP_ns_2020-07-19T00:00:00p0000--2d                                0      -   469K  -
cess/zproxy1/var/tmp@ZAP_ns_2020-07-19T00:00:00p0000--2d                                 0      -   188K  -
root@freenas1[~]# 

Footnotes and References