[Users] November 3, 2020 Zeta Alliance Conference Call Summary

Mon Nov 9 19:07:56 CET 2020

Hello Zeta Alliance Community, 

Here is a summary of last week’s conference call. A few brief reminders: 

    * Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: [ https://www.freeconferencecall.com/wall/zetalliance | https://www.freeconferencecall.com/wall/zetalliance ] 
    * Each week’s call agenda can be found at: [ https://docs.google.com/document/d/1uUUDJpwp2CAylU6lxtbEdVcUX_qSbciyes6gLTWw2fY/edit | https://docs.google.com/document/d/1uUUDJpwp2CAylU6lxtbEdVcUX_qSbciyes6gLTWw2fY/edit ] 
    * A copy of each week’s summary is also posted to the Zimbra Forums: 
        * All Prior Months: [ https://forums.zimbra.org/viewforum.php?f=9 | https://forums.zimbra.org/viewforum.php?f=9 ] 
        * October 2020 : [ https://forums.zimbra.org/viewtopic.php?f=9&t=68823 | https://forums.zimbra.org/viewtopic.php?f=9&t=68823 ] 
        * November 2020: [ https://forums.zimbra.org/viewtopic.php?f=9&t=68942 | https://forums.zimbra.org/viewtopic.php?f=9&t=68942 ] 
    * Constructive feedback on these call summaries is always welcome. 

November 3, 2020 

New Zextras Release 
Cine reported that Zextras released version 3.1.4, on November 2nd and highlighted two new features: 

    1. Zimbra data can now be backed up to external volumes such as an Amazon Web Services (AWS) S3 Bucket or a Network File System (NFS). 
    2. Mobile Password management has been moved to the Zimbra Web Client, allowing end-users to manage the mobile password, rather than only Zimbra administrators. Additionally, an unlimited number of mobile passwords can now be set. 

Marc G. asked if Zextras 3.1.4 will be released as part of the next Zimbra patch. Cine said he was not sure, but expected it may be part of 8.8.15 Patch 16 and 9.0 Patch 9. All of the changes included in the Zextras 3.1.4 release can be found at: [ https://docs.zextras.com/zextras-suite-documentation/latest/home.html | https://docs.zextras.com/zextras-suite-documentation/latest/home.html ] 

Saving Zimbra Backups To AWS S3 Buckets Or NFS 
Cine said that this new feature in Zextras 3.1.4 relies on a local caching feature provided by Zextras HSM (Hierarchical Storage Management) since an external volume (S3 or NFS) may not be able to immediately write data. He explained that this will be especially helpful in cases where NFS is used as an external backup volume, since NFS does not always tell Zimbra when a write fails, leading to the possibility of an incomplete backup. This allows for S3 to be used as a local storage device on a Zimbra server, since Zimbra sees it as a local mount point. Cine said that the HSM cache ensures consistent writes to an external NFS volume, by checking up to several times with the NFS volume to confirm that a write succeeded. Marc G. asked how this new cache feature confirms that a write succeeds when using S3 storage? Cine said that it is tougher to verify with S3, but usually is not required since the writes to S3 tend to be more reliable, as compared to NFS. Matthew F. commented that S3 provides a hash value for each write, which should provide a means for Zimbra to confirm a write. 

Mark S. asked how much data might be recoverable from the HSM cache, should a Zimbra server unexpectedly fail before the cache can be fully flushed to an S3 bucket. Cine said that the RPO (recovery point objective) should be near zero, since the Zimbra SmartScan feature can check what data was written to an S3 bucket, once the Zimbra server is recovered. 

Matthew F. asked if the new Zextras 3.1.4 version will reduce the number of files that are written to S3, since earlier implementations required a very large number of files to be written within a short period of time. Cine said that the new version will watch for contention errors from S3, and if observed, the HSM cache will slow down the writes sent to an S3 bucket, allowing it additional time to catch up. This in affect throttles the speed at which data transfer takes place increasing the likelihood it can succeed writing all data for a Zimbra backup over a longer period of time. 

Cine suggested that if using an S3 bucket as an external volume for Zimbra backups, it is a best practice to have a separate bucket for each Zimbra mailbox server. He said that after initially setting up external backups to S3, by default, Zimbra will use the same bucket for all mailbox servers in a cluster. However, he described a command line process whereby each mailbox server can be configured to use a different S3 bucket. 

Cine commented that revisions to the Zimbra Administrator’s guide, in the Backup/Recovery section will be forthcoming that detail using external volumes such as the S3 and NFS options. 

Managing Mobile Passwords 
Cine said that in addition to this feature being moved from the Zimbra Administration Console to the Zimbra Web Client in Zextra 3.1.4, where end-users can self-manage their own mobile passwords, if the Zextras mobile app is installed on a user’s phone/tablet, they can optionally scan a QR code generated by the Zimbra Web Client, instead of entering a conventional user name and password, to login to their email account. The login session on the Zextras app then remains active for as long as the QR code is valid in the Zimbra Web Client. 

Managing IMAP Mailboxes With Large Numbers Of Folders 
Randy L. asked if anyone on the call had suggestions about the threshold at which it makes sense to turn on the new “zimbra_imap_folder_pagination_enabled” local configuration setting, introduced in the 8.8.15 Patch 15 and 9.0 Patch 8 patches, since the release notes ( [ https://wiki.zimbra.com/wiki/Zimbra_Releases/9.0.0/P8 | https://wiki.zimbra.com/wiki/Zimbra_Releases/9.0.0/P8 ] ) indicate this setting is disabled by default. Mark S. said he enabled this feature earlier to try and lower the CPU load on his Zimbra servers, and has not observed any apparent differences in the operation of email apps on end-user devices, nor any performance related issues for the end-users. He added that on the Zimbra server-side, he has observed the Java garbage collector running less often. Enabling this local configuration setting requires a restart of the mailboxd service. 

John E. said that when a user has about 1,500 or more folders they are synchronizing with IMAP, that this seems to be the threshold at which a performance bug appears, based on internal test cases run by Synacor. Mark S. asked why is the default value for “zimbra_imap_folder_pagination_size”, discussed in the release notes, set to 2,000 rather than 1,500? John E. looked-up additional metrics from the internal Synacor testing on the issue. He said that performance timing was done from a Mac running Thunderbird with 42,000 email folders, with each containing 4 sub-folders with the “zimbra_imap_folder_pagination_enabled” setting set to false. This resulted in 388% CPU usage with a load average of 6-7 per CPU core. No sub-folders in the email folder tree synchronized in Thunderbird until two hours passed. Next, the “zimbra_imap_folder_pagination_enabled” setting was changed to true. The CPU usage dropped to less than 100% with load averages of 1-2. Sub-folders synchronized in Thunderbird after 20 minutes. These testing results have been documented internally at Synacor as ZBUG-1694, and resolved in the most recent patches for 8.8.15 and 9.0. John E. said that the internal testing at Synacor showed that a default setting of 2000 for “zimbra_imap_folder_pagination_size” was selected as it was found to help with optimizing IMAP usage from Outlook clients. 

Randy L. said he did not understand why the “zimbra_imap_folder_pagination_enabled” setting is disabled by default in the latest patches, but felt that it should be enabled by default, at least as a defensive measure to avoid a performance issue with end-user mailboxes containing large numbers of folders. Mark S. commented that he noticed in Office 365 they are limiting customers to a total of 1,000 email folders and felt that Zimbra’s support of very large numbers of folders is a competitive advantage since there are many use cases where a large folder tree may be needed. Marc G. commented that this is an example of where it is probably costing Zimbra BSPs (Business Service Providers) more money in the form of added server resources to provide service for IMAP-only users at a lower per mailbox licensing cost, than it would be if Synacor were to restructure the BSP program so that the ActiveSync feature could be used at some of the lower Zimbra mailbox licensing tiers, since ActiveSync provides a more efficient folder syncing process, as compared to IMAP. 

Industry-Related Meetup Discussing Searching Encrypted Data 
Mark S. shared an upcoming free meet-up on November 17th: [ https://www.meetup.com/Cloud-Security-Alliance-CSA-Boston/events/274055654/ | https://www.meetup.com/Cloud-Security-Alliance-CSA-Boston/events/274055654/ ] with a focus on how to perform searches on encrypted data, such as encrypted mailboxes. He said that this meet-up is a discussion on what is going on in the world of encrypted mailbox searching for those interested in the privacy implications of encrypting mailbox data at rest, while maintaining traditional mailbox searching features that normally require access to unencrypted mailbox data. 

Disappearing Zimbra Logs In CentOS 8 
Mark S. said that there is a previously reported issue with Zimbra on CentOS 8 where the default log file rotation in CentOS 8 causes Zimbra logs to disappear after 24 hours. A possible fix is documented at: [ https://forums.zimbra.org/viewtopic.php?f=15&t=68919&sid=103c05f7ddab9a324a1c1c7e6f70c340 | https://forums.zimbra.org/viewtopic.php?f=15&t=68919&sid=103c05f7ddab9a324a1c1c7e6f70c340 ] . He also said he has an open support case with Zimbra on this issue too. 

Randy Leiker ( randy at skywaynetworks.com ) 
Skyway Networks, LLC 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zetalliance.org/pipermail/users_lists.zetalliance.org/attachments/20201109/9224686c/attachment.html>