[Users] June 8, 2021 Zeta Alliance Conference Call Summary

[Randy Leiker]

Hello Zeta Alliance Community, 

Here is a summary of this week’s conference call. A few brief reminders: 


    * Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: [ https://www.freeconferencecall.com/wall/zetalliance | https://www.freeconferencecall.com/wall/zetalliance ] 
    * Each week’s call agenda can be found at: [ https://drive.google.com/drive/folders/1xDyBJFjnfZYxuXJHiDzsXjjMuGGtIl7J | https://drive.google.com/drive/folders/1xDyBJFjnfZYxuXJHiDzsXjjMuGGtIl7J ] 
    * A copy of each week’s summary is also posted to the Zimbra Forums: 
        * All Prior Months: [ https://forums.zimbra.org/viewforum.php?f=9 | https://forums.zimbra.org/viewforum.php?f=9 ] 
        * May 2021 : [ https://forums.zimbra.org/viewtopic.php?f=9&t=69570 | https://forums.zimbra.org/viewtopic.php?f=9&t=69570 ] 
        * June 2021 : [ https://forums.zimbra.org/viewtopic.php?f=9&t=69677 | https://forums.zimbra.org/viewtopic.php?f=9&t=69677 ] 
    * Constructive feedback on these call summaries is always welcome. 

June 8 , 2021 

Proposal To Change The Zeta Alliance Conferencing Platform 
Barry D. proposed that the Zeta Alliance consider moving away from freeconferencecall.com as the conferencing platform for the weekly Tuesday calls and instead use the Jitsi integration in either Zimbra 9 or Zimbra Cloud. He said one advantage of using freeconferencecall.com over the years has been a dial-in number for those not using the freeconferencecall.com desktop app, and he was not sure if this would be possible to replicate within Jitsi. He suggested sending a note to the Zeta Alliance mailing list to hold a vote on this platform change. 

Setting Up Centralized Logging For Zimbra With Elastic Stack 
Barry D. said he has published a new guide ( https://github.com/Zimbra/elastic-stack ) describing how to setup centralized logging in Zimbra with Elastic Stack and Rsyslog for visualization of log events and gathering new insights. This also makes it easy to search historic logs, in addition to monitoring service uptime and SSL certificate status. He explained that his guide shows that you do not need to have Elastic Stack components installed on all of the Zimbra servers, but rather just on a dedicated server instead. He has also published a guide introducing how to use a Zimlet to capture events: ( https://github.com/Zimbra/zimlet-cli/wiki/Capture-Zimbra-events-inside-a-Zimlet ). Barry shared the following links to other helpful development guides: 


    * [ https://wiki.zimbra.com/wiki/DevelopersGuide | https://wiki.zimbra.com/wiki/DevelopersGuide ] 
    * [ https://wiki.zimbra.com/wiki/Other_admin_guides_and_articles | https://wiki.zimbra.com/wiki/Other_admin_guides_and_articles ] 

Integrating Prometheus, Fluentd, Elastic Stack, and Logstash With Zimbra 
Marc G. asked if Synacor has plans to create an integration for Prometheus ( https://prometheus.io/ ) for Zimbra log data, as he is currently trying to feed log data to the logz.io platform ( https://logz.io/ ) and they require logs to be formatted with Prometheus. Barry D. said he is not aware of any current plans to do so, but said that if enough people ask for this feature, it would increase the probability of this integration in the future. John E. said that this should be possible, as Synacor is using an integration with Prometheus internally and much depends on how Marc opts to get logs out of Zimbra. He explained that he has found Fluentd ( https://www.fluentd.org/ ) and Fluent Bit ( https://fluentbit.io/ ) helpful. Fluent Bit can be deployed on individual Zimbra servers to avoid deploying the complete Fluentd stack on each Zimbra server. Fluent Bit then communicates with a dedicated server running Fluentd and it can push log data to just about any destination, since many log data transformers are available. John E. added that using Logstash ( https://www.elastic.co/logstash ) is also useful for parsing the logs, thereby taking the load of parsing off of the Zimbra servers. 

Marc G. said that he is seeking to export both Zimbra logs and deep monitoring of metrics to detect known patterns of system usage known to create problems. John E. said that both log and metric data can co-exist when using Elastic Stack with Zimbra. Barry D. asked what is the benefit of using Prometheus instead of Elastic Stack? John E. said that Prometheus is good for collecting metrics and monitoring tasks, while Elastic Stack excels at collecting log data. He also said that Prometheus’ alert manager feature works well for monitoring metrics. 

Marc G. said that he thinks auditing Zimbra logs for SIEM-like ( https://en.wikipedia.org/wiki/Security_information_and_event_management ) things, such as security events, in addition to understanding mail flow would be very helpful. Another example includes showing the lifecycle of an email as it travels from entry to final destination, or to unexpected places too, like a user’s misconfigured Zimbra filter. John E. recommended using Elastic Stack with Logstash for this purpose. He said that Marc’s initial focus should be on getting his Zimbra log data in to Elastic Stack where he will discover many different ways to use this data. Barry D. said that this is the goal of the Elastic Stack guide that he recently published, to help get everyone started, and suggested that Marc begin with getting RSyslog configured first, as it is helpful for setting up security logging. Then, he can proceed to working on getting Elastic Stack configured. Marc G. asked how RSyslog handles re-sending log data if there is a communication fault? Barry D. said that RSyslog does not handle this scenario by default, but thinks it can be configured to re-try sending logs. 

Barry D. asked how much does Prometheus costs? Marc G. said that it is free open source software. John E. said that a Prometheus integration is the kind of thing you may not want delivered as part of Zimbra, especially with the additional external infrastructure required to handle the log data, requiring a non-standardized setup for each Zimbra installation. Marc G. suggested that this would be a great open source project to make it easier for the Zimbra community to implement log visualization and metric monitoring features. He said that many log analysis vendors are charging eye watering amounts of money for storing log data like this, so he often stores his log data in S3 storage at AWS on limited redundancy disks. John E. said Marc might also consider storing his logs in AWS’ Glacier service, or somewhere similar, as cheaply as possible, then use Logstash to filter out what he is looking for, and import the remaining data in to Elastic Stack. John E. said that starting with log data for low volume email customers first, for a limited time frame is best, just to play around with it, and refine the filters to get the most interesting bits of log data out. He said you will then start to discover things out of the logs that you did not know you wanted. Barry D. added that it is very easy to break Elastic Stack, and can often be easier to restore a backup than trying to fix it, so using a test environment is important. He said that Elastic Stack auto updates too, and the same version can probably be run reasonably safely for about a year before needing to upgrade. Noah P. said that he has been looking in to this same need too, but there does not seem to be a cheap way around it with the log analysis vendors, so taking the free open source software route makes the most sense. Marc G. said that log analysis can feel like an endless rabbit hole, so it is important to identify the specific goals that you want to accomplish first from the Zimbra log data. Noah P. said he would love to see something for this purpose more turnkey ready from Synacor, as he finds he does not have the time in the day to get all of the components tuned and working smoothly. John E. said that Barry D’s recently published Elastic Stack guide comes the closest to Noah’s request, but this topic gets so far outside of what Synacor offers, he does not see it likely that this would be packaged and distributed within Zimbra. Barry D. said that he found in his work with Elastic Stack that storage is consumed quickly, where he had 100 GB initially allocated to his setup, that was fully consumed in just 2.5 months. 


Randy Leiker ( randy at skywaynetworks.com ) 
Skyway Networks, LLC 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zetalliance.org/pipermail/users_lists.zetalliance.org/attachments/20210614/c1898fab/attachment-0001.html>