[Users] Zimbra mailbox project

[David Touitou]

Hello, 

I'm not sure I get the point of Kubernetes/Docker about standard VM so I'll follow this thread with curiosity. 

About the export/import/migration of mailboxes, where do you intend to store the emails? 
I mean, are they supposed to be in the containers (or VM) or stored as objects through an object-storage-solution (HSM+S3, OpenIO, Ceph+Beezim, Scality)? 

David 

> De: "Randy Leiker" <randy at skywaynetworks.com>
> À: users at lists.zetalliance.org
> Envoyé: Mercredi 17 Janvier 2018 18:59:23
> Objet: Re: [Users] Zimbra mailbox project

> Hi Jonathan,

> You have several good questions. As the project is in the early phases of
> getting started, I don't have all of the answers, but can share what I know so
> far. A key premise of the project would be to transition from having a small
> number of very large mailbox nodes to instead having a large number of small
> mailbox nodes. In other words, you would see your average mailbox node size of
> 16 TB drop considerably, perhaps into the 100s of gigabytes or less, while the
> number of mailbox nodes you have would in turn increase greatly from 50 to
> perhaps something in the low hundreds of mailbox nodes. Since each mailbox node
> would have far fewer mailboxes on it than your current configuration, it
> becomes much more feasible to evacuate all mailboxes from a given node, should
> that node require maintenance or replacement. In effect, it makes the mailbox
> nodes something of a disposable commodity, since mailbox nodes are no longer a
> unique, difficult to replace snowflake, and become more like a herd of cattle
> (pun intended), which is easy to replicate.

> Since the project is currently in the information gathering stage, to be
> followed by a proof of concept stage, I wanted to publicly announce it within
> the Zeta Alliance early on to solicit feedback & ideas to improve on the
> original concept. In the email announcing the project yesterday, I didn't
> elaborate much on the methods being evaluated for evacuating end user mailboxes
> from a mailbox node, but two that I'm looking at now, for automation with
> scripting, are both zmmboxmove & zmmailbox. They both use differing methods,
> where the former seems to use a rsync-like method, and the latter uses an
> export/import method. Since the intent of the project is to use existing admin
> functionality within Zimbra, along with additional helper open source utilities
> where appropriate, I don't yet know which of these offers the best match & to
> what degree these utilities remain functional when a mailbox node goes bad. I
> think this question can only be answered through additional testing.

> So far, I've successfully spun up some Docker containers running Zimbra mailbox
> nodes in some preliminary testing within a lab environment. It's too early in
> the process yet, so performance stats aren't available, but the idea of having
> mailbox nodes operate within containers, is that they share a common host
> operating system (Linux, Ubuntu, etc.), so the containers themselves have just
> the Zimbra components. Since the containers are sharing a host OS, they should
> use the host's resources more efficiently, as opposed to running the full stack
> (OS + Zimbra) within a given VM. It's similar in many ways to the efficiencies
> gained with running many VMs on a single physical host. This should then allow
> for running larger numbers of mailbox nodes (in containers) across multiple
> VMs/physical hosts. This is where Kubernetes comes in, as a means to
> orchestrate all of the individual containers running Zimbra mailbox nodes.

> It's possible that Zimbra & Zextras's backup utilities will come into play, but
> for now, I'm trying to keep the project focused on doing online mailbox moves,
> or at least nearly online moves (accounting for mailboxes being placed
> temporarily into maintenance mode) as is possible with the current Zimbra admin
> utilities.

> Randy Leiker ( randy at skywaynetworks.com )
> Skyway Networks, LLC
> 1.800.538.5334 / 913.663.3900 Ext. 100
> https://www.skywaynetworks.com

> From: "Jonathan Labbé" <jlabbe at neonova.net>
> To: "Randy Leiker" <randy at skywaynetworks.com>
> Cc: users at lists.zetalliance.org
> Sent: Wednesday, January 17, 2018 10:31:19 AM
> Subject: Re: [Users] Zimbra mailbox project

> Wish, I did not have to leave in the middle of the meeting. So we take huge
> advantage of this already. We are currently running 4x LDAP (2 MMR servers and
> 2 Replica only servers), 8 x MTAs, 8x Proxies, and ~50x Mailbox stores, spread
> across two virtual data centers. We have F5's in the mix to help with proper
> load balancing and handling our SSL termination, and our own mailproxy system
> in front of Zimbra handling our client authentication.
> We have been looking at ways to try and maintain balanced numbers on our mailbox
> stores, with size of mailbox, imap usage (which hopefully the new IMAP servers
> will be good), etc. The feasibility of just evacuating a mail store for us, as
> you put it, just doesn't seem feasible. For example, we have a good chunk of
> our mail stores with 16TB of mail on them. That's a lot to just suddenly move,
> especially if you're trying to ensure as little or no loss of mail while this
> happens.
> A few questions I have;

> How are you evacuating mailboxes off of a "bad" mailstore? How does that user
> data and their mail get transferred to the other mail stores? I am not aware of
> any zmmailbox command that just does this, except for zmboxmove, this can be a
> slow process.
> Have you ran a dockerized mailbox store before? How does it perform? How many
> users were you able to run concurrently on this mail store?
> Is this process also taking advantage of Zimbra's backup processes to ensure
> fast mailstore recovery?

> 	Jonathan Labbé

> 919-460-3330 • MailScanner has detected a possible fraud attempt from
> "jlabbe at neonova.net" claiming to be jlabbe at neonova.net
> www.neonova.net

> On Tue, Jan 16, 2018 at 6:54 PM, Randy Leiker < randy at skywaynetworks.com >
> wrote:

>> Hi Everyone,

>> Earlier, on today's weekly Zeta Alliance call, I proposed a project to introduce
>> new resiliency capabilities to Zimbra mailbox servers. Here's an outline of
>> what was discussed & a further expansion on that topic:

>> Project Background
>> Dating back to early Zimbra versions, it's always been possible to create a
>> cluster of Zimbra server nodes where individual components of Zimbra can be
>> broken out into separate nodes for both load balancing & reliability benefits.
>> As an example, consider a minimal Zimbra cluster with:

>>     * 2 x LDAP nodes
>>     * 2 x Proxy nodes
>>     * 2 x MTA nodes
>>     * 2 x Mailbox nodes

>> Within this example cluster, you could afford to lose 1 of each type of node
>> (either due to maintenance, human error, or a disaster event), and the Zimbra
>> cluster would remain mostly operational. The exception, is the mailbox nodes,
>> in that, if you lose a mailbox node, any user mailboxes on that node become
>> immediately unavailable. This limitation exists because, by design, a user's
>> mailbox storage is tightly bound to a given mailbox node.

>> Shortly after Zimbra was acquired from Yahoo by VMware, there were mentions in
>> various webinars & presentations regarding expanding on Zimbra's high
>> availability (HA) capabilities. Most of those discussions seemed to focus on
>> leveraging HA capabilities that were part of VMware products, but not
>> necessarily in adding native HA capabilities to the Zimbra Suite itself. The
>> VMware approach to HA for Zimbra works, as long as the VM hosting a Zimbra node
>> remains bootable and all of the Zimbra services can start successfully
>> following recovery by VMware's HA feature. However, it's of no help when you
>> need to take a Zimbra mailbox node down for maintenance, perform a mailbox node
>> upgrade, troubleshoot a fault on a mailbox node, or just deprecate a mailbox
>> node for migration to newer hardware, since the end result is down time for
>> mailbox end users.

>> In more recent years, Zimbra mailbox node HA was on the road map for Telligent,
>> and most recently is on the road map for Synacor, with some suggestions
>> indicating that it might make an appearance in a Zimbra 9.0 release. I think
>> many Zimbra partners understand the significant undertaking that will be needed
>> to decouple the storage of end user mailboxes from the mailbox nodes, and given
>> that effort, the arrival of a true mailbox node HA feature is probably some
>> ways off yet into the future.

>> Project Proposal
>> I propose building new flexibility around Zimbra mailbox nodes, while leaving
>> the standard Zimbra distribution as-is. This would allow for addressing the
>> current shortcomings outlined above. Above all, the intent of the project is to
>> avoid changing the standard Zimbra distribution, so as not to create future
>> support or upgrade problems, and rather to leverage a combination of freely
>> available, open source tools & built-in Zimbra admin utilities to make it easy
>> enough that Zimbra admins (of all skill levels) have the ability to take Zimbra
>> mailbox nodes on & offline at-will with no disruption to mailbox end users.
>> This doesn't eliminate the need for best practices, such as regular backups of
>> your Zimbra infrastructure, but rather seeks to solve issues that backups don't
>> address.

>> The project involves placing all Zimbra mailbox nodes within Docker containers,
>> so it's the same mailbox node install you're used to doing, but just within a
>> container instead. Other components of Zimbra (LDAP, MTAs, Proxies, etc.) could
>> continue to run as VMs, physical machines, or perhaps within containers as
>> well. With at least several containers, each running a Zimbra mailbox node,
>> user mailboxes would presumably be evenly distributed over those mailbox nodes.
>> This distribution could be done arbitrarily by a Zimbra admin, or through the
>> course of normal day-to-day provisioning of mailboxes, by allowing Zimbra to
>> choose which mailbox node to provision mailboxes on, from the pool of available
>> mailbox nodes, which is functionality that exists today within Zimbra. With the
>> introduction of the Zextras tools in Zimbra 8.8, using the Zextras
>> backup/restore functionality would be yet another means to migrate customer
>> mailboxes into mailbox nodes, housed within containers.

>> A script would then be created to allow for a given mailbox node (container) to
>> have all of its mailboxes evacuated automatically. There are many cases where
>> you may need to place a mailbox node into maintenance mode. To name a few:

>>    * A service impacting configuration change is needed. For example, a Zimbra
>>    service restart for a new Let's Encrypt SSL certificate, or a zmprov local
>>     configuration value change that you want to test.
>>     * A hardware or software (operating system/Zimbra package) fault exists.
>>    * The mailbox node has insufficient hardware resources & is being deprecated for
>>     a newer mailbox node with more resources.
>>    * The mailbox node has developed an unknown, difficult to troubleshoot problem,
>>     so rather than troubleshoot it, the node is simply replaced.
>>    * A prior configuration error (aka human error) has led to an unstable mailbox
>>     node, so rather than fixing it, the node is replaced.

>> The script would take a few simple inputs, such as the target mailbox node, and
>> the desired action. Available actions might include:

>>     * Evacuating all mailboxes from a node
>>     * Restoring mailboxes to a node
>>     * Evacuating all mailboxes & removing the node from the cluster
>>    * Adding a node to the Zimbra cluster & re-distributing mailboxes to the newly
>>     added node

>> For options requiring evacuation of mailboxes, the script would query Zimbra
>> LDAP to determine which mailboxes are on that server, then using Zimbra's
>> built-in zmmailbox utility, evacuate those mailboxes evenly across the
>> remaining mailbox nodes (containers). For options requiring adding/removal of
>> mailbox nodes, this would have a tie-in with both Docker & perhaps Kubernetes
>> to allow for automating the provisioning & de-provisioning of containers for
>> hosting the mailbox nodes. I think there's room to greatly expand on the
>> available actions for this script, but this could be the first few steps. Care
>> would be needed to ensure that the script handles error conditions well, most
>> likely by alerting a human to a problem encountered carrying out a given
>> action, with a recommendation of what's needed next to resolve it.

>> I know that's a long email, but wanted to offer some explanation to give you a
>> scope of the project. The project would be free & open source for all in the
>> Zimbra community to use. Several people expressed interest in this project on
>> the weekly call earlier today. I'm curious to hear if others in the Zeta
>> Alliance feel that this would be a worthwhile project that your organization
>> could use and/or contribute to in the form of ideas, development, or testing.
>> Your thoughts & feedback please?

>> Randy Leiker ( randy at skywaynetworks.com )
>> Skyway Networks, LLC
>> 1.800.538.5334 / 913.663.3900 Ext. 100
>> https://www.skywaynetworks.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zetalliance.org/pipermail/users_lists.zetalliance.org/attachments/20180118/492ea3a4/attachment.html>