Re: Backup using ZFS Snapshots

On Tue, Dec 13, 2016 at 4:47 PM, Andrew Martin <amartin_at_xes-inc.com> wrote:

>
>
> ----- Original Message -----
> > From: "Mark Phippard" <markphip_at_gmail.com>
> > To: "amartin" <amartin_at_xes-inc.com>
> > Cc: "users" <users_at_subversion.apache.org>
> > Sent: Tuesday, December 13, 2016 3:35:37 PM
> > Subject: Re: Backup using ZFS Snapshots
>
> > On Tue, Dec 13, 2016 at 4:17 PM, Andrew Martin <amartin_at_xes-inc.com>
> wrote:
> >
> >> Hello,
> >>
> >> I am running a Subversion 1.9.3 server on Ubuntu 16.04. I currently use
> >> svnadmin hotcopy to safely backup the SVN repositories, but since the
> >> repositories are now hosted on a ZFS dataset, I would like to utilize
> ZFS's
> >> snapshot capabilities to create atomic, point-in-time backups of the
> >> repositories. My plan to do this is as follows:
> >>
> >> 1. create zfs snapshot
> >> 2. clone zfs snapshot and mount at a temporary location
> >> 3. run svnadmin hotcopy from the mounted clone to safely create a backup
> >> 4. umount and destroy clone
> >>
> >> My only concern is if a commit is in-progress when the zfs snapshot
> occurs,
> >> would svnadmin hotcopy still be able to safely handle creating the
> backup?
> >>
> >
> > svnadmin hotcopy will not have a problem with an in-process commit. That
> > is kind of one of the points of that command.
> >
> >
> >
> >> Is this a safe procedure for creating backups?
> >>
> >
> > I know very little about ZFS but I do not understand why you would do
> > this. If it can take snapshots, then why wouldn't you just take a
> snapshot
> > of the actual live repository, why would you want to copy it?
> >
> > Setting that aside, I think hotcopy is a very good way to do backups ...
> > but it is not clear what value would come from taking a snapshot of the
> > backup given that version control history is immutable etc. Also, once
> you
> > create the initial hotcopy, you can use the hotcopy --incremental option
> to
> > just copy the newer revisions. Do that from a post-commit hook and you
> can
> > always have a backup up to the latest commit.
> >
> > To me snapshots would seem like a way to do backups without using
> hotcopy.
> > I am not sure the value of combining the two ... but it should work
> > technically if you have "reasons".
> >
> > Mark
>
> The reason I want to do it this way is that I want to "pull" the backups to
> the backup server rather than "pushing" them from the Subversion server. In
> order to do that, I plan on mounting the clone over NFS on the backup
> server
> and then using "svnadmin hotcopy" to pull any new data onto the backup
> server.
> Yes, I could just mount the live repository, but I have a number of
> repositories
> to backup and using the ZFS snapshot ensures that they are all from a
> single
> point-in-time.
>
> It sounds like "svnadmin hotcopy" will just ignore any in-progress commit,
> so
> even if the ZFS snapshot happened in the middle of a commit, it would still
> result in a consistent backup.
>
>
Yes, hotcopy will leave you with a consistent repository at the state it
was in when the backup started. Any commits that are in process will not
be copied/included in the backup. The problem with hotcopy, is that unless
you craft a method where you can use --incremental, you are copying the
entire repository every time you do a backup even though only a small
number of new files may have been created. The SVN repository format is
write-once. Only new files get added as new revisions get created. So the
repository can be backed up very efficiently if you just keep capturing the
new revisions being created.

FWIW, I am not sure how concerning it really is if you took a snapshot in
the middle of a commit. From what I recall, there is a very tiny race
condition on the file that says what the HEAD revision is -- db/current.
But in the worst case and there is a problem, you can just run svnadmin
recover to reset it. It is not like you will have to deal with total file
corruption or something. There is also a SQLite DB cache now as well which
might add some other edge case problems on a live repository, but there
would be solutions for any problems there as well.

We have used NetApp snapshots to backup repositories for over a decade and
I am not aware of any problems we have ever had when recovering a
repository. I assume ZFS snapshots could work similarly where the entire
FS could be snapshotted and then presumably it has features like NetApp to
synch those snapshots to a DR location.

-- 
Thanks
Mark Phippard
http://markphip.blogspot.com/