[Users] February 9, 2021 Zeta Alliance Conference Call Summary

Randy Leiker randy at skywaynetworks.com
Tue Apr 13 17:58:45 CEST 2021

Hello Zeta Alliance Community, 

Here is a summary of this week’s conference call. A few brief reminders: 

    * Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: [ https://www.freeconferencecall.com/wall/zetalliance | https://www.freeconferencecall.com/wall/zetalliance ] 
    * Each week’s call agenda can be found at: [ https://drive.google.com/drive/folders/1xDyBJFjnfZYxuXJHiDzsXjjMuGGtIl7J | https://drive.google.com/drive/folders/1xDyBJFjnfZYxuXJHiDzsXjjMuGGtIl7J ] 
    * A copy of each week’s summary is also posted to the Zimbra Forums: 
        * All Prior Months: [ https://forums.zimbra.org/viewforum.php?f=9 | https://forums.zimbra.org/viewforum.php?f=9 ] 
        * January 2021 : [ https://forums.zimbra.org/viewtopic.php?f=9&t=69121 | https://forums.zimbra.org/viewtopic.php?f=9&t=69121 ] 
        * February 2021 : [ https://forums.zimbra.org/viewtopic.php?f=9&t=69470 | https://forums.zimbra.org/viewtopic.php?f=9&t=69470 ] 
    * Constructive feedback on these call summaries is always welcome. 

February 9, 2021 

Experiences With Zimbra Centralized Storage On AWS 
To follow-up on the February 2, 2021 Zeta Alliance call, Mark S. shared that he has been continuing to explore using Zimbra’s Centralized Storage feature for his Zimbra servers hosted at AWS for the purposes of speeding up mailbox moves and improving recovery time objectives in the event of a disaster recovery scenario. He said that it seems to be working well, but that he found the Zextras documentation for Centralized Storage ( [ https://docs.zextras.com/zextras-suite-documentation/latest/powerstore.html#_centralized_storage | https://docs.zextras.com/zextras-suite-documentation/latest/powerstore.html#_centralized_storage ] ) was missing a discussion of options for setting up different storage classes for use with an AWS S3 volume. Cine commented that it can be a challenge for the documentation to remain vendor neutral and avoid becoming focused on providing AWS S3 support. Mark S. suggested perhaps the documentation could be updated to note that each vendor has varying options available for tiered storage so that Zimbra administrators are aware of the potential to save money on their storage hosting costs. 

Mark S. said he wanted to use intelligent storage tiering, but found that his storage objects initially went into the standard storage class in AWS S3 instead. To correct this, he manually changed the volume settings on each Zimbra mailbox server so that future blobs will be correctly placed in to the intelligent storage tiering. He said he plans to test the Zimbra Raw Restore feature in Zimbra with this new setup in a lab environment and will share his findings in a future Zeta Alliance call. Mark added that he is not using Centralized Storage in Zimbra for his primary mailbox server volumes yet, but instead continues to use HSM, as he is unsure of how Centralized Storage could affect mailbox performance, particularly those mailboxes that have intensive IMAP users. He plans to ask his customers how they are using their mailbox data to set the most aggressive HSM policy possible, so as to keep the minimal amount of recent mailbox data outside of S3 storage. Marc G. asked if Mark S. is using blobless backups. Mark S. said he has not reached this point yet in his testing. Cine commented that blobless backups are designed to be used in conjunction with Centralized Storage. 

New Lifesize Zimlet 
Barry D. shared that a new Zimlet integration is now available for Lifesize ( [ https://github.com/Zimbra-Community/zimbra-zimlet-lifesize | https://github.com/Zimbra-Community/zimbra-zimlet-lifesize ] ), for creating virtual meeting rooms for virtual conferences. Marc G. asked if Barry has had experience with using Jitsi as he finds Lifesize tends to be very expensive. Barry said that there is already a beta Zimlet integration available for Jitsi in the Zimbra 9 Modern UI ( [ https://zimbra.github.io/zimbra-9/adminguide.html#_setting_up_jitsi_videoconferencing_solution_beta_early_access | https://zimbra.github.io/zimbra-9/adminguide.html#_setting_up_jitsi_videoconferencing_solution_beta_early_access ] ). John E. added that a packaged Jitsi Zimlet is anticipated to be included in a future Zimbra version release. 

Performance Testing With Zimbra Centralized Storage 
Matthew F. shared that he is exploring options for using Centralized Storage in Zimbra, combined with HSM, and small AWS EBS storage volumes for his frequently accessed data on his Zimbra servers. He currently runs his Zimbra mailbox servers in AWS. He said that he has been conducting performance testing and has observed big performance hits when mailbox data is stored within AWS S3 volumes, when using Zimbra HSM. He shared his preliminary testing results so far in the section titled “S3 Testing Data” ( [ https://docs.google.com/document/d/1zujlwsimMf-JhvDrLxWsd5mKFl-yK0SEwgpkg3oQkc8/edit | https://docs.google.com/document/d/1zujlwsimMf-JhvDrLxWsd5mKFl-yK0SEwgpkg3oQkc8/edit ] ). He said his biggest concerns are how these performance tests will reflect real-world usage should a complete mailbox server restore be needed in the event of a disaster recovery scenario, in addition to mailbox users conducting full Zimbra Connector for Outlook synchronizations. His calculation of these performance test results suggest it could require between 1-2 weeks to fully restore a mailbox server from S3 storage, in the event of the loss of a complete mailbox server. 

Cine said that performance testing of blob checks on S3 storage is the worst case scenario since it requires interacting with all components of a mailbox. He suggested using 4 parallel threads for a mailbox move or a disaster recovery, as past experience suggests this tends to be the right amount of threads, but said that it may require some trial and error to boost performance. Cine said that Matthew may want to take a look at this command where the number of threads can be specified: “zxsuite backup doExternalRestore /path/to/data/ domains domain1.com domain2.com concurrent_accounts 5”. Mark S. suggested going from an AWS T3.small server to a T5.large server and re-running the performance testing. 

Mark S. commented that S3 has throttling limits that are supposedly account-wide, and Matthew might be reaching those upper limits while running the performance tests and suggested opening a support case with AWS to determine if this is the cause. He added that S3 will always be slower than block storage in AWS. Marc G. said that delays are expected with S3 storage, but the current challenge is that he and Matthew do not feel they can count on it for disaster recovery if restoring a whole mailbox server, given the calculated 1-2 weeks of estimated time to restore a complete mailbox server from S3 storage. Mark S. said that if the HSM policy is set appropriately, a disaster recovery scenario would not involve restoring all mailbox blobs, but rather just the most recent few days worth of mailbox data to the mailbox servers’ primary storage volumes, while leaving all older blobs in object (S3) storage. 

Marc G. asked if anyone on the call had experience using Scality? John E. said that when using the S3 Connector or the EMC Connector for Zimbra, they utilize blobless backups, and the only potential risk is orphaning of blobs, which are defined as blobs (mailbox items) on disk, but no longer referenced within Zimbra’s MariaDB database. He explained the Zimbra NG modules provide compatibility with both the NG backups and HSM while the Scality S3 & EMC Connectors do not. John said that the user case for the Scality and EMC Connector is usually very large scale mailbox storage for some of the largest Zimbra service providers. The EMC Connector is available in the Zimbra GitHub repo if searching for “EMC”. 

Different Zimbra Patch Versions When Moving Mailboxes 
Matthew F. said while doing a mailbox move between two Zimbra mailbox servers with the same major and minor version number, but different Zimbra patch levels installed (Patch 14 versus Patch 18), he encountered a problem where the moved mailbox reported one unexpected blob, along with the Zimbra Connector for Outlook producing an error about a sync token being too old, thereby preventing Outlook syncing from operating. Matthew tried deleting the Outlook profile and re-creating it, but this did not resolve the issue. Mark S. said that he encountered a similar problem with mailbox moves, and ended up doing a restore of the mailbox to the destination server to fix the issue. Cine said he thinks mailbox moves between servers with different patch levels should work as there are some cross-safety checks that take place during a mailbox move operation. He added that the mailbox servers should negotiate the API level they can both support. 

Randy Leiker ( randy at skywaynetworks.com ) 
Skyway Networks, LLC 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zetalliance.org/pipermail/users_lists.zetalliance.org/attachments/20210413/a758f32b/attachment.html>

More information about the Users mailing list