[Users] ZCS OSE Hot DR Solutions

[gebhardt at openfactory.ch]

Personally, I only know of one solution that does actual hot backups of
mysql properly and that's r1soft - that's what we use.

Works pretty good, I have used it also from time to time to do a full
restore if I change the hardware, my zimbra grows from time to time

contact me offlist if you want to try it out.

another option (opensource) is to use replication of the database, have
a binlog in mysql, replicate it to a second DB Server on another machine
and do the dump there. Too much hassle and work to monitor for me.


BR
Silvan


Am 27.09.2016 um 15:34 schrieb Andrew Martin:
> ----- Original Message -----
>> From: "Truong Anh Tuan" <tuanta at iwayvietnam.com>
>> To: "amartin" <amartin at xes-inc.com>, "users" <users at lists.zetalliance.org>
>> Sent: Monday, September 26, 2016 11:26:38 PM
>> Subject: Re: [Users] ZCS OSE Hot DR Solutions
> 
>> On 09/27/2016 02:42 AM, Andrew Martin wrote:
>>> Hello,
>>>
>>> I am running ZCS OSE 8.7 on Ubuntu 14.04 in a single-node
>>> configuration. My current backup/DR strategy is to perform these
>>> steps nightly: * zmcontrol stop * rsync -avHKS --delete /opt/zimbra
>>> backupserver:/opt/zimbra * zmcontrol start * mysqldump all tables
>>> (just as a precaution) * zmslapcat (just as a precaution)
>>>
>>> This works pretty well, but it has a couple disadvantages: *
>>> requires zimbra downtime every night * RPO of 24 hours
>>>
>>> It would be nice to have an OSE hot backup solution as well,
>>> nothing as fancy as ZExtras Backup (which does per-account
>>> restore), but just a basic DR solution. It seems like this would be
>>> possible with a combination of the following: * rsync /opt/zimbra
>>> to catch index, misc files, and store * zmslapcat to get the LDAP
>>> database * Percona XtraBackup for consistent MySQL backups
>>>
>>> The challenge would be making sure that the database backup and
>>> /opt/zimbra/store were in sync.
>>>
>>> Has anyone developed a solution for OSE hot DR backups, or
>>> something similar to what I've described?
>>
>> We have done this well by using Snapshot. All you need is choose a
>> file system that supports snapshot for your Zimbra partition /opt/zimbra.
>>
>> Here we chose LVM snapshot. Please see a sample at:
>> https://github.com/EmailConsortium/zimbra-lvm-backup
>>
>> For DB + LDAP in sync with files, you should do DB + LDAP dumping
>> before snapshot + rsync files. Your process should be changed to:
>> * mysqldump all tables (just as a precaution)
>> * zmslapcat (just as a precaution)
>> * Start snapshot
>> * rsync -avHKS --delete /opt/zimbra backupserver:/opt/zimbra
>> * End snapshot
>> (no downtime needed)
>>
> Tuan,
> 
> Thanks for the good information! I really wanted to put /opt/zimbra on
> a ZFS dataset, but unfortunately ZFS is only officially supported on
> Ubuntu 16.04 and ZCS 8.7 is only considered "beta" status on 16.04, so I
> stayed with 14.04 and ext4 for now. Even if I were using LVM or ZFS for
> /opt/zimbra and could snapshot, there's still the problem of the database
> tables changing between the time that mysqldump is initiated and the time
> that the snapshot is taken. 
> 
> My understanding is that mysqldump locks each database as it dumps it, but
> since there are over 100 databases to dump, this takes several minutes to
> complete on my system. During that time, new mail is being delivered and 
> thus by the time mysqldump finishes, the databases that were dumped first
> will now be out-of-date compared to the ones that were dumped at the end.
> 
> Therefore even with a snapshot, you don't have a self-consistent, point-in-time
> backup. Sure you could lock all the databases, run mysqldump, snapshot, and
> then unlock, but then new writes will be blocked during that time. This is
> the problem I'm struggling to solve - how do you ensure fully self-consistent
> point-in-time database backups without stopping the zimbra services?
> 
> Thanks,
> 
> Andrew
>