<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;">Hi David,</div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div><font face="arial, helvetica, sans-serif"><span style="font-size: 10pt;">I'm looking at using Docker containers because it allows for a</span></font><span style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 13.3333px;">bstracting the Zimbra services of a mailbox node as much as possible from the operating system (short of changing the standard Zimbra distro), which I think will make it easier to script the process of spinning up or tearing down new mailbox node instances. One of the goals of the project is to identify ways to automate as much of the Zimbra mailbox node provisioning & de-provisioning process as practical, to allow for easier online moves of end user mailboxes from one node to another, and I think the key to doing this is to minimize the amount of infrastructure that needs to be cloned for each new instance of a Zimbra mailbox node. </span><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;">It's definitely possible to script the process of cloning an entire VM, but it seems like that adds extra complexity as compared to deploying a cloned Docker container with a pre-configured instance of a Zimbra mailbox node, since a scripted cloning of a VM also needs to consider customizations for the underlying cloned OS too. If the Zimbra mailbox node is housed solely within a container, it should minimize the amount of deployment steps for new mailbox nodes, which I'm hoping will translate to faster deployment times & greater simplicity. Of course, there's also the possibility t</span></font><span style="font-size: 13.3333px; font-family: arial, helvetica, sans-serif;">his approach may prove to have some unforeseen</span><span style="font-size: 13.3333px; font-family: arial, helvetica, sans-serif;"> pitfalls, and that's why I welcome everyone's input, as I'm exploring all options to refine the original idea into something that's beneficial for all.</span></div><div><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;"><br></span></font></div><div><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;">Storage in Docker containers works differently than in VMs. Container storage is transient by design, but there are options for making container storage persistent. I'm just starting to look into which storage method for containers would allow for minimizing time consuming copies of large numbers of end user mailboxes across the network, while trying to steer clear of expensive, proprietary storage options. So, I don't know the answer to that question yet.</span></font></div><br><br><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span name="x"></span><div><div><div><span style="color: rgb(255, 102, 0); font-weight: bold;"><br>Randy Leiker (</span><span style="font-weight: bold;"> <span style="color: rgb(51, 51, 255); background-color: rgb(255, 255, 255);">randy@skywaynetworks.com</span> <span style="color: rgb(255, 102, 0);">)</span></span><br><span style="color: rgb(0, 0, 153);">Skyway Networks, LLC</span><br><span style="color: rgb(0, 0, 153);">1.800.538.5334</span> <span style="color: rgb(255, 102, 0);">/</span> <span style="color: rgb(0, 0, 153);">913.663.3900 Ext. 100</span><br><span style="color: rgb(0, 0, 153);"></span><a href="http://www.skywaynetworks.com">https://www.skywaynetworks.com</a><br></div></div></div><span name="x"></span><br></div><hr id="zwchr" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><div style="color: rgb(0, 0, 0); font-family: Helvetica, Arial, sans-serif; font-size: 12pt; font-weight: normal; font-style: normal; text-decoration: none;"><b>From: </b>"David Touitou" <david@network-studio.com><br><b>To: </b>"Randy Leiker" <randy@skywaynetworks.com><br><b>Cc: </b>users@lists.zetalliance.org<br><b>Sent: </b>Thursday, January 18, 2018 4:02:05 AM<br><b>Subject: </b>Re: [Users] Zimbra mailbox project<br><br><div style="font-family: lucida console,sans-serif; font-size: 10pt; color: #000000"><div>Hello,</div><div><br></div><div>I'm not sure I get the point of Kubernetes/Docker about standard VM so I'll follow this thread with curiosity.<br></div><div><br></div><div>About the export/import/migration of mailboxes, where do you intend to store the emails?</div><div>I mean, are they supposed to be in the containers (or VM) or stored as objects through an object-storage-solution (HSM+S3, OpenIO, Ceph+Beezim, Scality)?</div><div><br></div><div>David</div><div><br></div><hr id="zwchr"><div><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>De: </b>"Randy Leiker" <randy@skywaynetworks.com><br><b>À: </b>users@lists.zetalliance.org<br><b>Envoyé: </b>Mercredi 17 Janvier 2018 18:59:23<br><b>Objet: </b>Re: [Users] Zimbra mailbox project<br></blockquote></div><div><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><style>p { margin: 0; }</style><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000;"><div>Hi Jonathan,</div><br><div>You have several good questions. As the project is in the early phases of getting started, I don't have all of the answers, but can share what I know so far. A key premise of the project would be to transition from having a small number of very large mailbox nodes to instead having a large number of small mailbox nodes. In other words, you would see your average mailbox node size of 16 TB drop considerably, perhaps into the 100s of gigabytes or less, while the number of mailbox nodes you have would in turn increase greatly from 50 to perhaps something in the low hundreds of mailbox nodes. Since each mailbox node would have far fewer mailboxes on it than your current configuration, it becomes much more feasible to evacuate all mailboxes from a given node, should that node require maintenance or replacement. In effect, it makes the mailbox nodes something of a disposable commodity, since mailbox nodes are no longer a unique, difficult to replace snowflake, and become more like a herd of cattle (pun intended), which is easy to replicate.</div><br><div>Since the project is currently in the information gathering stage, to be followed by a proof of concept stage, I wanted to publicly announce it within the Zeta Alliance early on to solicit feedback & ideas to improve on the original concept. In the email announcing the project yesterday, I didn't elaborate much on the methods being evaluated for evacuating end user mailboxes from a mailbox node, but two that I'm looking at now, for automation with scripting, are both zmmboxmove & zmmailbox. They both use differing methods, where the former seems to use a rsync-like method, and the latter uses an export/import method. Since the intent of the project is to use existing admin functionality within Zimbra, along with additional helper open source utilities where appropriate, I don't yet know which of these offers the best match & to what degree these utilities remain functional when a mailbox node goes bad. I think this question can only be answered through additional testing.</div><br><div>So far, I've successfully spun up some Docker containers running Zimbra mailbox nodes in some preliminary testing within a lab environment. It's too early in the process yet, so performance stats aren't available, but the idea of having mailbox nodes operate within containers, is that they share a common host operating system (Linux, Ubuntu, etc.), so the containers themselves have just the Zimbra components. Since the containers are sharing a host OS, they should use the host's resources more efficiently, as opposed to running the full stack (OS + Zimbra) within a given VM. It's similar in many ways to the efficiencies gained with running many VMs on a single physical host. This should then allow for running larger numbers of mailbox nodes (in containers) across multiple VMs/physical hosts. This is where Kubernetes comes in, as a means to orchestrate all of the individual containers running Zimbra mailbox nodes.</div><br><div>It's possible that Zimbra & Zextras's backup utilities will come into play, but for now, I'm trying to keep the project focused on doing online mailbox moves, or at least nearly online moves (accounting for mailboxes being placed temporarily into maintenance mode) as is possible with the current Zimbra admin utilities.</div><br><br><div><span></span><div><div><div><span style="color: #ff6600; font-weight: bold;"><br>Randy Leiker (</span><span style="font-weight: bold;"> <span style="color: #3333ff; background-color: #ffffff;">randy@skywaynetworks.com</span> <span style="color: #ff6600;">)</span></span><br><span style="color: #000099;">Skyway Networks, LLC</span><br><span style="color: #000099;">1.800.538.5334</span> <span style="color: #ff6600;">/</span> <span style="color: #000099;">913.663.3900 Ext. 100</span><br><span style="color: #000099;"></span><a href="http://www.skywaynetworks.com" target="_blank">https://www.skywaynetworks.com</a><br></div></div></div><span></span><br></div><hr id="zwchr"><div style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Jonathan Labbé" <jlabbe@neonova.net><br><b>To: </b>"Randy Leiker" <randy@skywaynetworks.com><br><b>Cc: </b>users@lists.zetalliance.org<br><b>Sent: </b>Wednesday, January 17, 2018 10:31:19 AM<br><b>Subject: </b>Re: [Users] Zimbra mailbox project<br><br><div dir="ltr">Wish, I did not have to leave in the middle of the meeting. So we take huge advantage of this already. We are currently running 4x LDAP (2 MMR servers and 2 Replica only servers), 8 x MTAs, 8x Proxies, and ~50x Mailbox stores, spread across two virtual data centers. We have F5's in the mix to help with proper load balancing and handling our SSL termination, and our own mailproxy system in front of Zimbra handling our client authentication. <br><div>We have been looking at ways to try and maintain balanced numbers on our mailbox stores, with size of mailbox, imap usage (which hopefully the new IMAP servers will be good), etc. The feasibility of just evacuating a mail store for us, as you put it, just doesn't seem feasible. For example, we have a good chunk of our mail stores with 16TB of mail on them. That's a lot to just suddenly move, especially if you're trying to ensure as little or no loss of mail while this happens.<br><div>A few questions I have;</div><br><div>How are you evacuating mailboxes off of a "bad" mailstore? How does that user data and their mail get transferred to the other mail stores? I am not aware of any zmmailbox command that just does this, except for zmboxmove, this can be a slow process.</div><div>Have you ran a dockerized mailbox store before? How does it perform? How many users were you able to run concurrently on this mail store? </div><div>Is this process also taking advantage of Zimbra's backup processes to ensure fast mailstore recovery?<br><br><div> <br><div class="gmail_extra"><br clear="all"><div><div class="m_5597709289716434337gmail_signature"><br><table width="100%" style="color: #222222; font-family: arial,sans-serif; font-size: small; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; width: 389.6px;" border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="font-family: arial,sans-serif; margin: 0px;"><table width="100%" style="max-width: 100%;" border="0" cellspacing="0" cellpadding="0"><tbody><tr><td valign="top" style="font-family: arial,sans-serif; margin: 0px; vertical-align: top; padding-right: 15px; padding-top: 5px; max-width: 75px;" rowspan="3"><img width="50" height="37" style="vertical-align: initial; border-radius: 4px; max-width: 75px;" alt="photo" src="http://neonova.net/wp-content/uploads/2016/12/N_Logo_NeoNova.png"></td><td valign="top" style="font-family: arial,sans-serif; margin: 0px; vertical-align: top;"><table border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="font-family: arial,sans-serif; margin: 0px; padding-bottom: 5px;"><table width="100%" style="line-height: 1.6; font-family: Verdana,Arial,sans-serif; font-size: 11px; color: #4e4b4c; padding-left: 2px; width: 322.8px;" border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="font-family: arial,sans-serif; margin: 0px;"><span style="margin: 0px; padding: 0px; color: #175083; font-size: 15px;">Jonathan Labbé</span><br></td></tr></tbody></table></td></tr><tr><td style="font-family: arial,sans-serif; margin: 0px; padding: 5px 0px 8px; border-top: 1px solid gray;"><table width="100%" style="line-height: 1.6; font-family: Verdana,Arial,sans-serif; font-size: 11px; color: #4e4b4c; padding-left: 2px; width: 325px;" border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="font-family: arial,sans-serif; margin: 0px; line-height: 18px;"><span style="display: inline-block; white-space: nowrap; color: #4e4b4c; text-decoration: none;">
<a href="tel:(919)%20460-3330" target="_blank">919-460-3330</a>
• <a href="http://jlabbe@neonova.net" style="color: #4e4b4c; text-decoration: none;" target="_blank"><font color="red"><b>MailScanner has detected a possible fraud attempt from "jlabbe@neonova.net" claiming to be</b></font> <span style="color: red;"><b>MailScanner has detected a possible fraud attempt from "jlabbe@neonova.net" claiming to be</b></span> jlabbe@neonova.net</a></span><b></b><br><span style="display: inline-block; white-space: nowrap;"><a style="color: #4e4b4c; text-decoration: none;" href="https://neonova.net" target="_blank">www.neonova.net</a></span> <a href="https://www.facebook.com/NeoNovaNNS/" target="_blank"><img width="10" height="10" src="http://neonova.net/wp-content/uploads/2016/12/facebookIcon.png"></a> <a href="https://twitter.com/NeoNova_NNS" target="_blank"><img width="10" height="10" src="http://neonova.net/wp-content/uploads/2016/12/twitterIcon.png"></a> <a href="http://www.linkedin.com/company/neonova-network-services" target="_blank"><img width="10" height="10" src="http://neonova.net/wp-content/uploads/2016/12/linkedinIcon.png"></a></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><br></div></div>
<br><div class="gmail_quote">On Tue, Jan 16, 2018 at 6:54 PM, Randy Leiker <span dir="ltr"><<a href="mailto:randy@skywaynetworks.com" target="_blank">randy@skywaynetworks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000;"><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Hi Everyone,</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Earlier, on today's weekly Zeta Alliance call, I proposed a project to introduce new resiliency capabilities to Zimbra mailbox servers. Here's an outline of what was discussed & a further expansion on that topic:</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><b><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Project Background</span></b></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Dating back to early Zimbra versions, it's always been possible to create a cluster of Zimbra server nodes where individual components of Zimbra can be broken out into separate nodes for both load balancing & reliability benefits. As an example, consider a minimal Zimbra cluster with:</span></div><div style="color: #000000;"><ul><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">2 x LDAP nodes</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">2 x Proxy nodes</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">2 x MTA nodes</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">2 x Mailbox nodes</span></li></ul><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Within this example cluster, you could afford to lose 1 of each type of node (either due to maintenance, human error, or a disaster event), and the Zimbra cluster would remain mostly operational. The exception, is the mailbox nodes, in that, if you lose a mailbox node, any user mailboxes on that node become immediately unavailable. This limitation exists because, by design, a user's mailbox storage is tightly bound to a given mailbox node.</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Shortly after Zimbra was acquired from Yahoo by VMware, there were mentions in various webinars & presentations regarding expanding on Zimbra's high availability (HA) capabilities. Most of those discussions seemed to focus on leveraging HA capabilities that were part of VMware products, but not necessarily in adding native HA capabilities to the Zimbra Suite itself. The VMware approach to HA for Zimbra works, as long as the VM hosting a Zimbra node remains bootable and all of the Zimbra services can start successfully following recovery by VMware's HA feature. However, it's of no help when you need to take a Zimbra mailbox node down for maintenance, perform a mailbox node upgrade, troubleshoot a fault on a mailbox node, or just deprecate a mailbox node for migration to newer hardware, since the end result is down time for mailbox end users.</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">In more recent years, Zimbra mailbox node HA was on the road map for Telligent, and most recently is on the road map for Synacor, with some suggestions indicating that it might make an appearance in a Zimbra 9.0 release. I think many Zimbra partners understand the significant undertaking that will be needed to decouple the storage of end user mailboxes from the mailbox nodes, and given that effort, the arrival of a true mailbox node HA feature is probably some ways off yet into the future.</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><b><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Project Proposal</span></b></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">I propose building new flexibility around Zimbra mailbox nodes, while leaving the standard Zimbra distribution as-is. This would allow for addressing the current shortcomings outlined above. Above all, the intent of the project is to avoid changing the standard Zimbra distribution, so as not to create future support or upgrade problems, and rather to leverage a combination of freely available, open source tools & built-in Zimbra admin utilities to make it easy enough that Zimbra admins (of all skill levels) have the ability to take Zimbra mailbox nodes on & offline at-will with no disruption to mailbox end users. This doesn't eliminate the need for best practices, such as regular backups of your Zimbra infrastructure, but rather seeks to solve issues that backups don't address.</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">The project involves placing all Zimbra mailbox nodes within Docker containers, so it's the same mailbox node install you're used to doing, but just within a container instead. Other components of Zimbra (LDAP, MTAs, Proxies, etc.) could continue to run as VMs, physical machines, or perhaps within containers as well. With at least several containers, each running a Zimbra mailbox node, user mailboxes would presumably be evenly distributed over those mailbox nodes. This distribution could be done arbitrarily by a Zimbra admin, or through the course of normal day-to-day provisioning of mailboxes, by allowing Zimbra to choose which mailbox node to provision mailboxes on, from the pool of available mailbox nodes, which is functionality that exists today within Zimbra. With the introduction of the Zextras tools in Zimbra 8.8, using the Zextras backup/restore functionality would be yet another means to migrate customer mailboxes into mailbox nodes, housed within containers.</span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div style="color: #000000;"><span style="font-family: arial, helvetica, sans-serif; font-size: small;">A script would then be created to allow for a given mailbox node (container) to have all of its mailboxes evacuated automatically. There are many cases where you may need to place a mailbox node into maintenance mode. To name a few:</span></div><div><ul><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">A service impacting configuration change is needed. For example, a Zimbra service restart for a new Let's Encrypt SSL certificate, or a zmprov local configuration value change that you want to test.</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">A hardware or software (operating system/Zimbra package) fault exists.</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">The mailbox node has insufficient hardware resources & is being deprecated for a newer mailbox node with more resources.</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">The mailbox node has developed an unknown, difficult to troubleshoot problem, so rather than troubleshoot it, the node is simply replaced.</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">A prior configuration error (aka human error) has led to an unstable mailbox node, so rather than fixing it, the node is replaced.</span></li></ul><div><span style="font-family: arial, helvetica, sans-serif; font-size: small;">The script would take a few simple inputs, such as the target mailbox node, and the desired action. Available actions might include:</span></div><div><ul><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Evacuating all mailboxes from a node</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Restoring mailboxes to a node</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Evacuating all mailboxes & removing the node from the cluster</span></li><li><span style="font-family: arial, helvetica, sans-serif; font-size: small;">Adding a node to the Zimbra cluster & re-distributing mailboxes to the newly added node</span></li></ul><span style="font-family: arial, helvetica, sans-serif; font-size: small;">For options requiring evacuation of mailboxes, the script would query Zimbra LDAP to determine which mailboxes are on that server, then using Zimbra's built-in zmmailbox utility, evacuate those mailboxes evenly across the remaining mailbox nodes (containers). For options requiring adding/removal of mailbox nodes, this would have a tie-in with both Docker & perhaps Kubernetes to allow for automating the provisioning & de-provisioning of containers for hosting the mailbox nodes. I think there's room to greatly expand on the available actions for this script, but this could be the first few steps. Care would be needed to ensure that the script handles error conditions well, most likely by alerting a human to a problem encountered carrying out a given action, with a recommendation of what's needed next to resolve it.</span></div></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: small;"><br></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: small;">I know that's a long email, but wanted to offer some explanation to give you a scope of the project. The project would be free & open source for all in the Zimbra community to use. Several people expressed interest in this project on the weekly call earlier today. I'm curious to hear if others in the Zeta Alliance feel that this would be a worthwhile project that your organization could use and/or contribute to in the form of ideas, development, or testing. Your thoughts & feedback please?</span></div><div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 10pt;"></div><div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 10pt;"><br></div><br><div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 10pt;"><span></span><div><div><div><span style="color: #ff6600; font-weight: bold;"><br>Randy Leiker (</span><span style="font-weight: bold;"> <span style="color: #3333ff; background-color: #ffffff;"><a href="mailto:randy@skywaynetworks.com" target="_blank">randy@skywaynetworks.com</a></span> <span style="color: #ff6600;">)</span></span><br><span style="color: #000099;">Skyway Networks, LLC</span><br><span style="color: #000099;"><a href="tel:(800)%20538-5334" target="_blank">1.800.538.5334</a></span> <span style="color: #ff6600;">/</span> <span style="color: #000099;"><a href="tel:(913)%20663-3900" target="_blank">913.663.3900 Ext. 100</a></span><br><span style="color: #000099;"></span><a href="http://www.skywaynetworks.com" target="_blank">https://www.skywaynetworks.com</a><br></div></div></div><span></span><br></div></div></div></blockquote></div><br></div></div></div></div></div>
</div></div><br></blockquote></div></div></div><br></div></body></html>