[Users] ZCS OSE Hot DR Solutions

[Andrew Martin]

----- Original Message -----
> From: "Truong Anh Tuan" <tuanta at iwayvietnam.com>
> To: "amartin" <amartin at xes-inc.com>, "users" <users at lists.zetalliance.org>
> Sent: Monday, September 26, 2016 11:26:38 PM
> Subject: Re: [Users] ZCS OSE Hot DR Solutions

> On 09/27/2016 02:42 AM, Andrew Martin wrote:
>> Hello,
>> 
>> I am running ZCS OSE 8.7 on Ubuntu 14.04 in a single-node
>> configuration. My current backup/DR strategy is to perform these
>> steps nightly: * zmcontrol stop * rsync -avHKS --delete /opt/zimbra
>> backupserver:/opt/zimbra * zmcontrol start * mysqldump all tables
>> (just as a precaution) * zmslapcat (just as a precaution)
>> 
>> This works pretty well, but it has a couple disadvantages: *
>> requires zimbra downtime every night * RPO of 24 hours
>> 
>> It would be nice to have an OSE hot backup solution as well,
>> nothing as fancy as ZExtras Backup (which does per-account
>> restore), but just a basic DR solution. It seems like this would be
>> possible with a combination of the following: * rsync /opt/zimbra
>> to catch index, misc files, and store * zmslapcat to get the LDAP
>> database * Percona XtraBackup for consistent MySQL backups
>> 
>> The challenge would be making sure that the database backup and
>> /opt/zimbra/store were in sync.
>> 
>> Has anyone developed a solution for OSE hot DR backups, or
>> something similar to what I've described?
> 
> We have done this well by using Snapshot. All you need is choose a
> file system that supports snapshot for your Zimbra partition /opt/zimbra.
> 
> Here we chose LVM snapshot. Please see a sample at:
> https://github.com/EmailConsortium/zimbra-lvm-backup
> 
> For DB + LDAP in sync with files, you should do DB + LDAP dumping
> before snapshot + rsync files. Your process should be changed to:
> * mysqldump all tables (just as a precaution)
> * zmslapcat (just as a precaution)
> * Start snapshot
> * rsync -avHKS --delete /opt/zimbra backupserver:/opt/zimbra
> * End snapshot
> (no downtime needed)
> 
Tuan,

Thanks for the good information! I really wanted to put /opt/zimbra on
a ZFS dataset, but unfortunately ZFS is only officially supported on
Ubuntu 16.04 and ZCS 8.7 is only considered "beta" status on 16.04, so I
stayed with 14.04 and ext4 for now. Even if I were using LVM or ZFS for
/opt/zimbra and could snapshot, there's still the problem of the database
tables changing between the time that mysqldump is initiated and the time
that the snapshot is taken. 

My understanding is that mysqldump locks each database as it dumps it, but
since there are over 100 databases to dump, this takes several minutes to
complete on my system. During that time, new mail is being delivered and 
thus by the time mysqldump finishes, the databases that were dumped first
will now be out-of-date compared to the ones that were dumped at the end.

Therefore even with a snapshot, you don't have a self-consistent, point-in-time
backup. Sure you could lock all the databases, run mysqldump, snapshot, and
then unlock, but then new writes will be blocked during that time. This is
the problem I'm struggling to solve - how do you ensure fully self-consistent
point-in-time database backups without stopping the zimbra services?

Thanks,

Andrew